OpenAI has identified significant reliability issues within SWE-Bench Pro, a prominent benchmark for evaluating AI coding capabilities. An audit found approximately 30% of its tasks are broken, undermining its utility for accurately assessing model performance.

Issues stem from overly strict tests, underspecified prompts, low-coverage tests, and misleading instructions. These flaws can invalidate correct solutions or allow incomplete ones to pass, creating a false impression of AI progress. OpenAI's analysis pipeline flagged 200 tasks (27.4%), with human reviewers identifying 249 (34.1%) as problematic.

These findings necessitate a careful re-examination of benchmark results by model developers. OpenAI has retracted its previous recommendation to adopt SWE-Bench Pro, underscoring the difficulty in curating robust coding evaluations and the increasing need for agents to assist in data quality checks.