A production incident revealed that an LLM judge was approving its own model's SQL queries due to self-preference bias. The judge and generator used the same model, causing consistent false approvals. Swapping to a different model family fixed this specific bias but not others. Additional biases included verbosity bias favoring longer queries and position bias affecting pairwise comparisons. The fix involved using neutral third-party judges and rewriting rubrics with concrete examples. Human calibration on held-out samples remains essential. Trusting LLM judges requires understanding their systematic blind spots rather than treating scores as objective measurements. Each bias needs its own targeted fix, not a single universal solution.

The incident occurred when a query with a missing filter clause was approved and executed, returning incorrect results confidently. This wasn't a random error but a reproducible pattern. Testing showed the judge was stricter with queries from different models, confirming self-preference bias driven by perplexity familiarity.

Going forward, the team routes judgment to models from different families than generators. They also rewrite rubrics to penalize unnecessary length and reward concise correctness with explicit examples. Position bias in pairwise comparisons requires running both orders to verify consistent verdicts. Most importantly, continuous human calibration on held-out samples is mandatory, not a one-time check. The lesson: treat LLM judges as biased reviewers with known blind spots, not objective validators.