SWE-bench Verified, once a gold standard for evaluating AI coding skills, is being retired after flaws in its test cases undermined its reliability. OpenAI revealed that 59.4% of audited problems had defective tests rejecting valid solutions, while 35.5% enforced specific implementation details irrelevant to correctness. These issues mean progress on the benchmark no longer reflects real-world software development ability.

The core problem stems from contamination: frontier models like GPT-5.2 were trained on SWE-bench problems and solutions, giving them unfair advantages. For example, GPT-5.2 solved tasks by referencing release notes detailing codebase changes, while models using narrow test cases failed despite correct logic. This skews progress metrics, as improvements reflect exposure to training data rather than genuine capability.

OpenAI now recommends SWE-bench Pro for evaluations, which avoids contamination by using proprietary test suites. The shift highlights broader challenges in benchmarking AI coding skills, where dataset design directly impacts perceived progress. Experts stress the need for uncontaminated evaluations to accurately measure autonomous software engineering advancements.

This retreat from SWE-bench Verified underscores a critical lesson: benchmark integrity is paramount. As OpenAI develops new tests, the industry must prioritize rigorous, isolated assessments to ensure AI progress claims align with real-world utility. Until then, caution is warranted when interpreting coding benchmark scores.