Image and video models have gotten a lot better over the last few years, even though the internals of these models haven't changed much since Stable Diffusion 3. Of course, there have been small variants like the auto-regressive diffusion that GPT-Image popularized. But at a high level, it's pretty much all flow matching with a transformer backbone and a v-prediction objective.

In our experience, most of the gains are directly attributable to 3 flavors of data improvements: Data Filtering & Rebalancing, Data Annotation, and Synthetic Data Generation. RL has driven some improvement, but it's only started truly working for image and video in the past few months.

A couple of years ago, the prevailing wisdom across all generative models was to aggregate as much data as humanly possible for pre-training. Luckily, the field has gotten a lot smarter about this. If you throw a bunch of low-quality data into pre-training, your model will waste capacity learning to mimic that slice of data. If you filter your dataset well, your model will have an easier time learning what you want it to learn.

We walk you through how our approach to data filtering has evolved since 2024, and hopefully save you from a couple of headaches if you end up training your own generative models down the line.