Scaling laws hold that a language model grows more capable with more parameters and training data. Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous stored parameter bank per token.

That success is built on static pretraining data. A deployed model faces a different world, where much of the data that would make it more useful is not in its training set but in the live interaction it is currently handling, such as facts a user supplies or corrections they give. A conventional model cannot learn from this data, because its weights are frozen after training. Instead, knowledge and behaviour supplied at run time are placed in the prompt and discarded once the request ends.

We ask how an architecture could learn from live interaction by writing it into its weights. Taking inspiration from MoE, we propose the Infinite-Parameter LLM. A compact hypernetwork turns run-time data into a low-rank modulation of a shared base network, so feed-forward weights are generated from live data rather than stored in a fixed bank. Where prior weight generators read context once and freeze, we carry a Bayesian belief over the generator's latent code and update it online.

The stored footprint stays fixed, yet the weights the model can compile are effectively infinite. Carrying run-time knowledge in weights rather than the prompt is amortized in compute, frees the context window, persists across turns, and can generalise better. We specify an evaluation protocol testing this against in-context learning and retrieval.