We modified requirements in a coding benchmark and found models talking themselves out of following them. Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else.
In one run, a model implemented the change correctly, asked another model for the benchmark's original answer, and removed its own correct code after receiving it. This is an awkward habit for a coding assistant to have. It is also a habit that a familiar benchmark can reward.
If the model has guessed what the tests expect, it gets the same passing result as an agent that worked carefully from the requirements. We built Maxx Bench to examine how much coding performance depends on those expectations. It changes the instructions or starting code of 89 Deep SWE tasks, giving models familiar engineering work with a reason to revise their usual approach.
Methodology: Small mutations... Once a benchmark becomes popular, it becomes part of the development process. Its tasks can enter training data.
Its scores influence checkpoint selection. Developers adjust prompts and agents against its failures. There are plenty of opportunities to get better at that particular exam.
We call gains tied to its particular tasks and conventions benchmaxxing. Then the scores cluster near the top, and the usual response is a harder benchmark. Recent releases such as Frontier SWE v2 have pushed toward engineering projects that can occupy an agent for much of a day.
We chose Deep SWE's medium-duration feature-implementation tasks because they are closer to the everyday coding work we wanted to evaluate. For example, one Deep SWE task asks an agent to write a separate test report for each browser. We added a requirement that the filenames be lowercase.
The reporting feature still needs to work; Chrome.xml should now be chrome.xml. Another task adds retirement dates to Fast API routes. We changed the required timestamps to UTC with a trailing Z, instead of preserving the supplied time-zone offset.
Across all tasks, Maxx Bench's reference patches grow by a median of just 17 lines (3.5%). Handling menial revisions of this size should be routine for a capable coding agent. Claude Opus 5.5 and Gemini 3.8 Flash both have published Deep SWE scores of about 74%.
On Maxx Bench, they score 76.4% and 53.4%. A 23-point gap is a different basis for choosing between them. The original scores make them look like peers; the revised tasks expose a large difference in how reliably they handle changes to the work.
GPT-6 Astra and Opus 5 also start close together on Deep SWE, then separate to 71.9% and 63.5% on Maxx Bench.
Source: Hacker News · Summarized by HeadlinesBriefing