Earendil measures agent-written code and finds about twice the duplication and function complexity of human code

Earendil published a post on 10 September by researcher Sebastian on how to measure the "sloppiness" of code that AI agents write. Sebastian says LLMs now produce code that is "almost perfectly correct" against hidden tests, but that correctness misses a separate problem: agents add unneeded abstractions, duplicate work, and grow files fast, and there is no accepted way to score any of it.

The two metrics he used

Sebastian took two measures from the SlopCodeBench paper on arXiv. The first, verbosity, counts lines that AST-Grep flags plus lines inside detected code clones, divided by total lines of code. The second, erosion, is the share of a codebase's "mass" that sits inside functions with a cyclomatic complexity above 10, where a function's mass is defined as its complexity times the square root of its source lines. High verbosity means duplication. High erosion means a codebase's logic is packed into a small number of big, tangled functions.

What the measurements show

On the SlopCodeBench evaluation, established repositories score an average verbosity of 0.15 with a standard deviation of 0.06, and an average erosion of 0.31 with a standard deviation of 0.17. Agent-written code scores 0.33 for verbosity and 0.68 for erosion, roughly double on both. Sebastian says he then ran the same measures against his own vibe-coded projects and saw verbosity as high as 0.4 and erosion as high as 0.75, so the gap is not an artefact of the benchmark.

He also flags a second result from the same paper. SlopCodeBench evaluates agents over multiple rounds of instruction and hidden tests, and erases the model's context between checkpoints, so mistakes and shortcuts accumulate the way they do in a real project. Under the strict rule that every test must pass at every checkpoint, "even state of the art models achieve 0% pass rate". Sebastian notes that this was not tested on Anthropic's Claude Fable 5.1 or OpenAI's GPT-6 Astra, only on earlier frontier models.

Where the shortcuts break

The post spends a page on the two shortcuts a team might use to score AI-written code. Asking an LLM to score code from 1 to 10 comes back close to random. Giving one LLM two solutions labelled A and B and asking it to pick the better one is unstable: renaming the solutions flips the preference, an effect Sebastian traces to an arXiv paper on judge-model bias, and the flip is weaker on bigger models but the direction of the effect stays. Human review works, but a team cannot scale it across model providers or training runs. That leaves the two structural measures, imperfect as they are, as the honest option.

For a team shipping software that agents wrote, the practical read is that lines-of-code throughput and unit-test pass rate are the two numbers that look best in a dashboard and the two that are least honest about slop. If agent code is on average carrying twice the duplication and twice the complexity concentration, the review load per merged line goes up, and the accumulated erosion after a few iterations is what the 0% strict solve rate is measuring. Sebastian is open to talking to teams working on evals and gives his email at the end of the post.

Source

Earendil: If coding is solved, what now?: Measuring the sloppiness of code, by Sebastian, 10 September 2026. Cites SlopCodeBench on arXiv.