Diffusion models, powerful tools for generative tasks like image creation, exhibit "creativity" by generalizing beyond their training data. A new paper, "On the Interpolation Effect of Score Smoothing in Diffusion Models," presented at ICLR 2026, explores the mathematical basis for this capability. The research reveals that a model's "creativity" stems not from a random fluke, but from how neural network training naturally "smooths" the transformation from noise back to data during generation.

Training a diffusion model involves corrupting data with noise and then training the model to reverse this process, reconstructing data from noise. While perfect denoising would lead to memorization, diffusion models typically generalize. This generalization is attributed to "score smoothing," a consequence of imperfect neural network training, often due to regularization techniques like weight decay. This smoothing effect results in the denoising process interpolating between training points, thus generating novel data.

In complex, high-dimensional data spaces, this score smoothing is crucial for "manifold recovery." It slows particle flow tangential to the data manifold while allowing movement towards it, balancing image quality and novelty. The paper suggests that the "creativity" observed in diffusion models is a predictable mathematical outcome of neural networks' inherent imprecision, enabling them to bridge and interpolate between known data points to suggest new, plausible outputs.