A passing AI coding agent run can still hide a lot of wasted effort. In one example, an agent passed the tests on turn 4 but kept working through turn 10, and six of its eight changed lines were unnecessary compared with the reference fix. A standard pass/fail check scores this run the same as a clean, efficient one.
The author notes that the agent in this experiment is simulated. Rather than running a real coding agent such as Claude Code or Codex, the study used a seeded Python policy to make decisions, while the repositories, edits, diffs, test runs and failures are real. The goal was to measure how many extra turns an agent takes to reach a passing patch, how much of that work was needed, and whether a surrounding workflow can reduce unnecessary effort.
The analysis draws on about 50,000 simulated runs, including a compliance sweep, tests of individual rules and a held-out suite. It builds on SWE-bench, a benchmark of 2,294 real GitHub issues from 12 Python repositories, which scores only the final patch. Two runs can both succeed there while one takes far more turns, time and tokens, and leaves more changes behind.
The article separates final patch size from total work, which includes failed test runs, repeated commands, edits to the wrong file and reverted changes. Pass rates miss this effort entirely. The study aims to show what that hidden work costs and how harness rules might cut it down.
Source: Towards Data Science · Summarized by HeadlinesBriefing