Explorative Modeling (XMs) introduces a new paradigm for generative AI by adding a third pretraining axis that enhances exploration during training. This approach improves existing models across modalities like images, video, and language, with gains scaling significantly with data and parameters (7%→36% with data, 13%→23% with parameters). XMs achieve 6.2× sample efficiency, 4.1× FLOP efficiency, and 47% better parameter efficiency compared to traditional methods.

The core issue XMs address is the 'average problem' in generative modeling. Direct regression predicts the average of all valid outputs, resulting in generic, non-representative results. For example, training a model to generate 'dog' images produces a blurred average rather than specific dogs. XMs solve this by allowing models to explore multiple valid answers during training. In a dartboard analogy, traditional models guess the center (average), while XMs spread guesses across actual dart locations, reducing error. This exploration extends to end-to-end generation, enabling models to match diffusion performance with 256× less inference compute.

XMs differ from autoregressive or diffusion models by not factoring generation into small steps. Instead, they explore during training, which prevents errors from compounding during inference (a problem called exposure bias). This allows XMs to run end-to-end, aligning training and inference processes. The method scales well, with more exploration (e.g., 20 guesses vs. 2) spreading predictions across data distributions. XMs thus combine the benefits of factoring generation with end-to-end learning, offering a scalable solution to generative modeling challenges.