The rapid advancement of text-to-image AI models, such as Nano Banana, Stable Diffusion and Flux, has transformed creative design, but steering these models to meet precise user intent remains a balancing act. For example, prompting for "a lizard wearing sunglasses" might yield a realistic lizard without sunglasses, or distort the lizard's face if forced. Existing methodologies are disconnected: developers use inference-time techniques like classifier-free guidance, or heavy fine-tuning with parameter-efficient adapters like LoRA, reward-weighted regressions, or policy gradients. Because these tools are treated as distinct fixes, the field lacks a single mathematical language to unify control.

The Diffusion Controller framework solves this by reframing the denoising process as a smooth control problem. Its lightweight add-on network outperformed the industry standard for matching human preferences, and its fully unlocked version achieved a 90% win rate over the baseline. The framework treats generation as a controlled journey, with the base model frozen and a steering damper adjusting the trajectory.

This approach dynamically recalibrates the model's behavior, giving more weight to user-defined targets while preserving image quality. In the lizard example, the damper ensures sunglasses are included, but a penalty guardrail prevents distortion. The framework bridges theory and practice with two fine-tuning methods based on a final reward score, including policy gradient and PPO, which iteratively steers the model with a built-in clipping rule.