A new tracker monitors the daily performance of Claude Code with Opus 4.5, specifically focusing on detecting performance degradations on SWE tasks. The goal is to identify statistically significant drops in performance, providing an early warning system for users. The tool uses a curated subset of SWE-Bench-Pro, running benchmarks directly in the Claude Code CLI.

This initiative stems from the need for independent oversight, following past instances of model performance regressions. The tracker provides daily, weekly, and monthly views, aggregating results for more reliable estimates. It also offers email alerts for statistically significant performance drops. The methodology involves statistical testing to detect changes.

The tracker's value lies in its ability to provide objective and timely assessments of model performance. By identifying regressions quickly, developers can adapt their workflows to mitigate negative impacts. This is especially important for those relying on AI-powered coding assistants. It can also help Anthropic improve their models.

Looking ahead, the ongoing monitoring will provide valuable insights into the stability and reliability of large language models in coding applications. As models evolve, the ability to rapidly identify and address performance changes becomes increasingly important. This tool is a step toward greater transparency and accountability in the AI development process.