GitHub user volotat has released mini-AGI, a continual learning byte-level language model designed to train from scratch on consumer hardware with just 8 GB VRAM. The project addresses the limitation that training models at 1B+ parameter scale typically requires enterprise-grade infrastructure. By implementing a Mixture of Experts (MoE) architecture that dynamically adds and prunes experts during training, and utilizing batch-1 training on a single continuous data stream, the model bounds parameter count by available disk space rather than VRAM capacity.

The system loads and unloads experts to GPU memory only when needed, eliminating the need to store large randomized batches and gradients. This allows the model to read continuous interleaved passages—each 32K characters long—mimicking human sequential learning. The current training run processes a 7.8B character corpus, expected to take several weeks to complete.

The author acknowledges extensive collaboration with Claude during development, stating the project would be impossible without AI assistance. While labeled a "toy-level model" not matching frontier capabilities, it demonstrates that continual learning without catastrophic forgetting is achievable on modest hardware. The repository is publicly available for cloning and observation, enabling anyone to train or continue training their own aligned models.