Current text-to-SQL benchmarks, like Spider and Bird-SQL, show high accuracy but fail to reflect the complexities of real-world data stores. These benchmarks often use "in-the-pile" data that LLMs can train on, unlike proprietary, access-controlled enterprise data warehouses.

Real data warehouses suffer from "schema rot," where table and column names become non-intuitive due to frequent updates, leading to ambiguous semantics (e.g., multiple "salary" columns with different meanings). They also contain idiosyncratic data specific to the organization, which LLMs struggle with. Furthermore, real-world queries are often more complex, involving multiple joins.

To address these issues, the Beaver Benchmark was created, incorporating schema rot, idiosyncratic data, and complex queries. On Beaver, pure LLMs achieved zero accuracy, with RAG and agentic AI raising it to only 10+%. Providing LLMs with explicit table and join clauses improved accuracy to 30+%. This contrasts sharply with the 80-90+% accuracy seen on simpler benchmarks, highlighting the gap between academic interest and practical application. The creators welcome contributions to improve LLM performance on real-world business applications.