How good are query optimizers, really? Leis et al. asked this in 2015, then again 10 years later. Despite a decade of research, they found query optimizers continue to leave much to be desired. A Postgres database should know everything about its tables, no? How hard can it be? As it turns out: enormously hard. Join ordering, a key task, is NP-hard. But verifying a query plan's quality is easier: good optimizers produce fast plans, bad ones slow plans.

Language models excel at tasks with easily verifiable outputs. With a single axis—execution time—the problem reduces to reinforcing behaviors that yield faster query plans. I ran an experiment: can a small, open-weights model be post-trained via supervised fine-tuning (SFT) and agentic reinforcement learning (RL) to produce Postgres query plans that beat Postgres's default plans?

The answer is a resounding yes. Highlights: Attaining a 44.7% latency reduction across 113 join-heavy queries from a 4B model initially unable to produce a query plan for 99 of them. Constructing a Postgres measurement rig that minimizes Linux page cache contention noise. Designing a custom GRPO variant for scoring RL rollouts in a noisy environment. Splitting RL across two machines: vLLM and the trainer on a rented 2x H100 node and four Postgres containers on my desk. Running off-policy distillation across half a thousand GPT-6 Astra agent trajectories.

Let's start from the beginning. Inside a query optimizer, consider the IMDb dataset. A query asks: "Which Japanese companies put out the most titles in the 2000s?" Postgres's path depends on selective predicates. Without filters, join ordering becomes critical.