Existing AI coding benchmarks measure solve rate and token cost, but they do not penalize excess code, which makes codebases harder to maintain over the long term. Parsimony was built to measure how much code AI models add, a factor that matters most for long-term projects.
The tool takes the patches that established benchmarks already publish, including DeepSWE, SWE-bench Verified and SWE-bench Live, and counts coding units. These are syntax elements such as statements, calls, comparisons, names and literals. Lines of code are not counted, so cramming code is not rewarded, and deleted code is subtracted, giving a net figure of added minus removed units.
Models are ranked by mean net units added on the tasks they solved, lowest first. The mean also includes failed attempts. Solved status comes from SWE-bench Verified, and tests are not rerun. Each patch is applied to the task's code and both versions of every changed file are parsed. Nothing is executed, and tests, docs and generated files are excluded. Only Python is currently measured.
The author notes limits. Smaller is not always better, since units measure size rather than readability or design. Models that write more code also tend to solve more tasks, with a Spearman correlation of 0.33 to 0.60 on boards with 26 or more models. All models use the same agent, mini-SWE-agent, which still influences how much code is written. Users can add their own benchmark data through a downloadable example file that stays in the browser.
Source: Hacker News · Summarized by HeadlinesBriefing