Databricks built an internal benchmark evaluating coding agents on its multi-million line codebase spanning Python, Go, TypeScript, Scala, Rust, Java, Bazel, and Protobuf. Using Unity AI Gateway to capture thousands of real engineering interactions, they constructed tasks from merged PRs — filtering for recency, human authorship, quality test suites, and self-contained changes across the full stack. About 25% of tasks were low complexity, 60% medium, revealing that expensive models were default despite most work not requiring them.

Three capability tiers emerged. GLM 5.2 landed in the top tier, statistically tied with Opus 4.8 on quality (87% task completion) but costing $1.28/task versus Opus's $1.94. Token pricing proved misleading: Sonnet 5 was 1.7x cheaper per token yet cost $2.09/task — 1.9x more tokens consumed due to reasoning inefficiency. The Pareto frontier requires mixing OpenAI, Anthropic, and open models.

Harness choice dramatically shifted economics. Running the same model through Pi versus Claude Code/Codex changed cost per task by over 2x with identical quality. Pi sent 3x less context per turn, maintaining a tighter working set and finishing in fewer runs. This validated Databricks' investment in Omnigent for seamless model-and-harness swapping.

The team is now pushing daily work to Haiku and GPT-5.4 Mini class models, reserving top-tier models for design exploration. Public benchmarks like SWE-Bench failed them — solutions leak into training data, and task distributions don't match their polyglot, service-oriented architecture. Custom benchmarking on private PRs gives confidence that efficiency gains won't hamper developers.