Neural networks excel at generating code in dependently-typed languages like Idris, Lean, and Agda, yet most large language models train to predict tokens without understanding types. Current approaches either retry after compilation failures or use constrained decoding to ensure type safety, but neither fundamentally solves the problem of learning structure during training.

These workarounds show promise - FrontierMath scores jumped from under 2% to nearly 50% in two years, and AlphaZero achieves superhuman chess performance with 30x fewer parameters by integrating game rules into training. The contrast is stark: while LLMs reach intermediate chess ratings as a byproduct of general training, AlphaZero never makes illegal moves. This suggests that encoding language rules during training could dramatically improve performance in typed programming domains.

The fundamental challenge remains: how do you differentiate through discrete type systems? Current research like CHAD and Higher Order AD shows promise by partitioning input spaces, but requires programmers to define these partitions rather than letting networks learn them automatically. The question isn't whether typed generation works - it's whether we can rebuild LLMs from the ground up to produce structured, typed output during training rather than as an afterthought.