Harvard study: AI coding agents raise code 30 percent, software output zero

An AI coding agent writes a lot of code. The question has always been whether any extra software reaches a user because of it. Ars Technica reports on a Harvard study that says the answer, so far, is no.

Writing for Ars Technica on 9 October, Kyle Orland summarises a working paper by Harvard researchers Fiona Chen and James Stratton that uses engineering analytics data from Jellyfish, covering 300 million work events and issue tracking at more than 700 software firms employing over 700,000 people from 2021 through March 2026. The researchers compare firms before and after they adopted either AI coding assistants, which complete code a human is writing, or AI coding agents, which write and submit code on their own, using a difference-of-differences regression.

What the paper found

Ars Technica reports that introducing AI coding agents at a firm raised total lines of code by 30 per cent on average, total commits by 20 per cent, and pull requests by 23 per cent. Despite that, the researchers report no statistically significant change in the rate at which Jira issues and epics were resolved after the agents arrived, and no change in the size or complexity of the Jira items either. Ars Technica writes that the extra code was "absorbed by downstream constraints in the production process", with code review as the named constraint.

The review number is what pins the story down. Ars Technica reports that the time from a pull request opening to being merged went up 49 per cent on average after AI agents landed, the share of pull requests needing changes nearly doubled, and the number of comments per pull request rose 35 per cent. The share of staff doing code review went up 14 per cent, and Chen and Stratton say they could not attribute any employment change to AI after cross-referencing Jellyfish data with LinkedIn.

What the sample covered

Ars Technica says that by March 2026, 95 per cent of firms in the sample had introduced AI coding agents, and 80 per cent used some form of AI code review. The AI reviewers, however, wrote only 23.3 per cent of all review comments and were responsible for only 10.8 per cent of pull requests. Humans were still doing most of the reviewing. The data cutoff is March 2026, and agent quality has moved since then, which Ars Technica notes.

Why this matters for a team using agents

The reader of this item is a team already running Claude Code, Copilot or Codex and watching its pull request queue grow faster than before. The paper's result, if it holds, puts review at the step that decides how much software ships. A team that doubles the number of pull requests without changing anything about review is spending more and shipping the same amount. Three plain actions follow: measure the median pull request's time in review before and after an agent lands, assign a reviewer per agent rather than per author, and treat an AI reviewer's comment as a draft that still needs a human on consequential changes, since the paper's measured AI-review share is low enough that the human is still the bottleneck in practice.

Source

AI coding agents generate more code, but not more software, Kyle Orland, Ars Technica, citing a working paper by Fiona Chen and James Stratton at Harvard University, using data from Jellyfish.