Parsimony ranks AI coding agents by how much code they add

Most benchmarks for AI coding agents answer one question: did the model fix the bug. A new open-source benchmark adds a second one, how much code did it add to fix it, and ranks models by the smaller number. Parsimony, written by an independent researcher who signs posts as lasu, parses the patches that public benchmarks already publish and counts coding units: syntax elements such as statements, calls, comparisons, names and literals.

Lines are not the unit, so cramming code into fewer lines does not help the ranking. Deleted code is subtracted. Each model's score is the mean net units it adds on the tasks it solves, with the overall mean, the benchmark's own solve rate and the number of measurable attempts shown next to it. The site opens with a plain sentence: "Existing benchmarks measure solve rate and cost, but don't punish excess code, which makes a codebase harder to work with in the long term."

The boards

The site publishes six boards. DeepSWE in Python ranks 26 models across 34 tasks with four attempts each. DeepSWE in JavaScript covers 26 models over five tasks and four attempts. DeepSWE TypeScript ranks 26 models on 35 tasks, DeepSWE Go on 34 tasks. SWE-bench Verified ranks 35 models across 500 tasks with one attempt each. There is also an SWE-bench Live Lite board for Python and a SWEeper-Bench board for PHP with 15 models over 128 tasks.

The method

Each model's patch is applied to the task source, both file versions are parsed and nothing is run. Test, documentation and generated files are left out. Pass or fail comes from the benchmark's own published results. Every attempt names the artifact hash it came from and the frozen task population it was measured against. Anybody can recompute every number by cloning the repository and running the shipped scripts.

A worked example on the site counts a loop replaced by sum: 13 units removed and 3 added, net minus ten. The analyzer code and the full counting rules are in docs/method.md.

The stated limits

The site puts the caveats next to the numbers. Smaller code is not automatically better: units measure size, not readability or design. Models that solve more tasks also tend to write more code, with a Spearman correlation of 0.33 to 0.60 on the larger boards. Every model uses the same agent, mini-SWE-agent, so the harness still shapes how much code each one writes. Some attempts cannot be measured because of missing files or parse errors, and the Measured column shows how many did.

The main DeepSWE Python page also shows faded self-run entries where the author tested newer models on three to ten tasks. GPT-5.6 Sol was run both ways and added 0.88 times the code of its published run, which the site presents as a limit rather than a claim about the model.

Reveneau's take: for a codebase a team keeps for more than a quarter, review cost is linear in the size of the patch. A model that passes the same test with less code is cheaper to live with. The site's own correlation caveat is why it is a second ranking rather than the only one.

Source

Parsimony by lasu, with code on GitHub under MIT.