Modern LLMs struggle with static knowledge and catastrophic forgetting during updates. Inspired by human sleep, Google AI introduces a sleep framework for continual learning. This paradigm alternates between an Active (wake) phase—where the model interacts and stores short-term context—and a Sleep phase—where it consolidates memory offline. During sleep, the model unlocks Mixture-of-Experts (MoE) memory blocks and uses Generalized Knowledge Distillation (GKD) to transfer fragile context into stable long-term parameters. This process prevents forgetting, retains knowledge up to 10M tokens, and improves complex reasoning without changing the core architecture.
The framework addresses catastrophic forgetting through memory consolidation and dreaming. Consolidation involves parameter expansion via MoE and Hope architecture, integrating low-rank experts into stable blocks. Knowledge seeding uses On-policy distillation and Learning to Imitate (LTI) to transfer high-frequency context into long-term parameters. Synaptic pruning further refines memory. By mimicking NREM slow-wave sleep, the model abstracts and distills new facts, enabling continuous adaptation and self-improvement in deployed LLMs.
Source: Google AI Blog · Summarized by HeadlinesBriefing