On-policy self-distillation (OPSD) has rapidly become a major topic in machine learning research. Although its frameworks build on older, established approaches, the first papers on the topic appeared only in late January 2026, and several dozen have followed in a brief span. This article explores the literature on OPSD, mainly through the lens of the Self-Distillation Fine-Tuning (SDFT) framework introduced by Shenfeld et al. in "Self-distillation enables continual learning."
The term "continual learning" in that paper's title refers to a model's ability to effectively and dynamically learn from a datastream. The article begins by examining the problem OPSD frameworks set out to solve: catastrophic forgetting, where models lose the ability to handle older tasks as they learn new ones. As foundation models grow more powerful, post-training them for complex or customized tasks becomes essential, since pre-training from scratch is prohibitively expensive.
The article then looks at how SDFT works and the theoretical explanations for its success. Experiments reported in the paper show promising results, lending credence to the prospect of OPSD being widely applicable. Finally, it touches on two central concerns that have surfaced regarding the method.
Source: Towards Data Science · Summarized by HeadlinesBriefing