A new benchmark, ORCA-bench, evaluates the readiness of large language model (LLM) agents for production oncall tasks. While LLMs can write and patch code, oncall requires reasoning over complex, noisy data like metrics, logs, and traces, starting from ambiguous user reports. ORCA-bench simulates a production-fidelity oncall setting using a live microservice system with real telemetry interfaces and source code access. It includes 1,079 RCA tasks that vary report specificity, detection time, and fault scenarios, with ground truth validated by expert SREs.

Across five leading agents, the best RCA Accuracy achieved was only 25.3% on medium-difficulty tasks and 10.0% on hard tasks. The poorest performing model hallucinated root causes in 40% of reports. Removing source code access further degraded performance across all metrics. These results, from a controlled 50 GB / six-day testbed, represent a lower bound on the engineering required to safely deploy LLM agents for production reliability, given the significantly larger and more dynamic nature of real-world systems. The public dataset is now available.