Diffusion models excel at generative tasks like image generation and molecular discovery, exhibiting "creativity" by generalizing beyond training data. In "On the Interpolation Effect of Score Smoothing in Diffusion Models", presented at ICLR 2026, Google AI researchers investigate this capability's mathematical origins.

Training involves corrupting data with noise and learning to reverse it via denoising. A perfect score function would drive particles to replicate training points exactly — memorization. However, neural networks learn approximate score functions. Due to regularization like weight decay, networks produce score smoothing, softening sharp transitions in the score function. In a 1-D example with points at +1 and -1, smoothing creates a gentler slope between them, causing particles to settle in the interpolation zone rather than collapsing to training points. Experiments with two-layer ReLU networks trained via AdamW confirm stronger weight decay increases smoothing.

In high dimensions, data resides on a data manifold. Score smoothing acts directionally: it slows flow along tangential directions (preventing collapse to training data) but not toward the manifold (preserving image quality). This balances realism and novelty.

The findings suggest diffusion model "creativity" is a predictable mathematical consequence of imperfect neural network learning, enabling interpolation between known data for novel outputs in image generation and drug discovery.