A new benchmark called OTelBench reveals significant challenges for AI models in performing basic Site Reliability Engineering (SRE) tasks. The benchmark tested 14 AI models on OpenTelemetry (OTel) instrumentation tasks across 11 programming languages. The best-performing model, Claude 4.5 Opus, only succeeded 29% of the time, highlighting limitations in understanding complex systems.

OTelBench focuses on assessing AI's ability to add distributed traces to codebases. Distributed tracing is essential for monitoring microservices. It allows engineers to track requests as they move across services. The study's results suggest that current AI models struggle with the nuanced requirements of OpenTelemetry, often failing to correctly propagate context and understand the business logic.

The benchmark authors found that even cutting-edge models have difficulty with relatively simple tasks. They struggled to differentiate between independent user actions, leading to the merging of separate traces. The open-source nature of OTelBench allows developers to evaluate different models and contribute to the project. This will reveal the capabilities of AI in complex software engineering.

Looking ahead, the development of more sophisticated AI models is crucial. As systems become more complex, the ability to automate SRE tasks becomes increasingly important. The QuesmaOrg/otel-bench project provides a valuable tool for tracking progress in this area. It will be interesting to see how future iterations of AI models perform on these tasks.