Large language models like Claude and Codex can produce impressive code at superhuman speeds, but they frequently generate buggy or nonsensical results when given too much freedom. John Regehr proposes using executable oracles - automated tools that constrain the LLM's choices - to eliminate degrees of freedom where models tend to fail. The approach builds on existing testing practices but applies them systematically to LLM-assisted development.

Regehr demonstrates this with several examples. When building Claude's C Compiler, incorporating fuzzers like Csmith and YARPGen into the testing loop would have prevented 34 miscompilation bugs that slipped through standard test suites. For dataflow transfer function synthesis, constraining Codex between soundness and precision oracles produced results better than both manual and randomized approaches. The JustHTML project showed how combining fuzzing for correctness with performance metrics led to impressive outcomes, though architecture decisions still required human intervention.

The key insight is that we already have abundant correctness oracles - test suites, fuzzers, sanitizers, static analyzers - and performance oracles including profilers and benchmarks. By making these tools available to LLMs during development rather than after, we can prevent many classes of bugs before they occur. However, some aspects like software architecture remain difficult to constrain programmatically, suggesting a hybrid approach where humans guide high-level design while oracles handle implementation details.