Two families of generative AI algorithms dominate today: diffusion models for continuous data like images, and autoregressive models for discrete data like text. Diffusion models generate entire sequences at once, starting from a rough draft and refining every position in parallel over several rounds. This approach enables trade-offs between speed and quality, built-in error correction, and bidirectional context. Autoregressive LLMs generate one token at a time, left to right, taking as many steps as there are tokens. Diffusion LLMs instead start from a full-length draft and rewrite the whole sequence at each step. Applying diffusion to language was long an open problem, but by 2024 diffusion models became competitive with autoregressive models on quality. By 2026, diffusion LLMs are a reality, with releases from leading industry labs including Mercury 2 (Inception Labs), Gemma Diffusion (Google), and Nemotron Diffusion (NVIDIA). The central concept underlying diffusion models is denoising: producing images step by step, starting from pure random noise and removing a little at every step until a coherent image emerges. The forward process takes a clean training image and produces increasingly noisy images. The reverse process learns to invert this transformation, turning noise back into an image.

Diffusion models learn to denoise by training on examples of noise being gradually transformed into images. The forward process adds noise to a clean image one step at a time, while the reverse process learns to invert this transformation. This two-step approach makes diffusion models highly effective for generation.

Modern diffusion language models trace their foundations to these core principles, adapted for discrete data like text.