A flurry of recent activity in continuous diffusion models for language suggests this approach is making a comeback after years of dormancy. Fully discrete diffusion methods had largely supplanted earlier continuous diffusion attempts, but the tide is starting to turn. Modern language models are predominantly autoregressive, generating sequences one token at a time, which has proven extremely scalable through parallel training via teacher forcing.

However, autoregression isn't the only iterative generative approach—researchers explored diffusion for language generation by reversing a corruption process that gradually destroys information. Early discrete diffusion models emerged in 2021, replacing continuous corruption with discrete methods to handle categorical data. In 2022, continuous diffusion for discrete data gained traction with approaches like Diffusion-LM, which represented discrete categories with continuous embedding vectors amenable to Gaussian noise corruption, enabling application of well-established Gaussian diffusion mechanisms without modification to the core algorithm.