Constrained decoding solved a real problem. Before grammar-based methods like Outlines and SGLang, getting valid JSON from a language model was a retry loop. Teams adopted it fast, assuming schema compliance meant data correctness. However, BAML's benchmarks say otherwise. On function-calling tasks, unconstrained generation with post-hoc parsing reached 93.63% accuracy; constrained decoding on the same model scored 91.37%. The always-valid JSON was less accurate than the sometimes-broken JSON.

I started tracking this after a classification pipeline began returning plausible but fabricated values on roughly one in twelve runs. The JSON always parsed. Pydantic never complained. It took weeks to notice, because every downstream check was structural. Five failure modes keep showing up. They all produce schema-valid output that breaks your pipeline silently: Enum hallucination: valid enum, wrong meaning. Confident fabrication: plausible values in free-text fields. Cross-field contradiction: fields valid individually, impossible together. Distributional collapse: convergence on safe defaults. Array hallucination: fabricated entries instead of empty arrays.

Structured output used to mean hoping the model behaved. Constrained decoding was built to fix that, and it did, just not the whole problem. The progression was real: prompt-and-pray JSON gave way to regex-guided generation, then to grammar-based constrained decoding. XGrammar, now the default backend for vLLM and TensorRT-LLM, adds near-zero overhead per token. The syntax problem is solved. But solving syntax created a blind spot. Schema validation checks whether a field is typed correctly: a string is a string, a number is a number. It says nothing about whether that string or number is correct. Forcing a model into a strict output format costs it something. It has to spend part of its attention on staying inside the format, instead of spending all of it on getting the actual answer right. Lee et al. measured that cost directly. Across open-weight models, forcing structured output formats produced a 3-to-9 percentage point accuracy drop. On math reasoning tasks specifically, where getting the reasoning right matters more than the format, the gap exceeded 15 percentage points. The format is not free, and most teams aren't accounting for that cost.