A reference implementation is a simplified but complete version of a system that trades performance for clarity. At first glance, PyTorch—the lingua franca of modern deep learning—seems to be a reference language. But upon a closer look, confusion arises: reference implementations aren’t usually deployed to production, yet I do my training jobs with it.

I’ve found a clarifying perspective: PyTorch plays a dual role—as both the reference language and the implementation language. When the scale is modest or the compiler works well, the reference implementation can ship to production. But increasingly, it serves as an artifact that verifies the iteration.

The modern use of kernel DSLs illustrates this. Traditional compiler‑maximalist views posited that users write high‑level APIs that a compiler then optimizes. But for key ops like matrix multiplies and attention, compilers struggle to guarantee peak performance; kernel DSLs let people spell out tiling and data movement explicitly.

The new recipe keeps the traditional PyTorch autograd‑friendly code as the reference. LLMs generate an explicit forward‑backward version that can be optimized separately. A verifier—bitwise tests or translation validation—ensures the reference and the real implementation remain equivalent.