Nitin Garg's study of AI-generated code review reports the same bug-catch rate with or without a written spec, and 81 percent finding-to-requirement attribution with the spec against 0 without

Nitin Garg published a study on InfoQ on 2026-09-10, accepted at GAISS 2026, on whether writing a specification before generating code with an AI model changes what a human reviewer finds. Five reviewers with 3 to 10 years of experience each reviewed two AI-generated banking services under two conditions: code with a specification baseline, and code alone. The paper is labelled preliminary and directional with small samples.

What the numbers actually say

Garg says the two services carried 11 and 10 adjudicated defects, seeded and unintentional. Recall did not move: 0.525 with the specification baseline against 0.518 with code only, p equal to 0.69. Attribution did move: 81 percent of findings were tied to a named requirement when the baseline was present, against 0 percent without it, p equal to 0.043. Review time went from 27 minutes on code alone to 48 minutes with the baseline, an extra 21.7 minutes per review at the same p value.

Garg also ran 90 machine reviews across Anthropic Claude Opus 4.8, OpenAI GPT-5.2, DeepSeek V4 and Google Gemini 3.1 to check that the pattern was not a quirk of the human sample. He reports that the same shape held: no recall gain, a real attribution gain.

The generation half of the study

The paper covers generation as well as review. Garg reports that a staged approach that produces a specification first and then code from it reached a 45 percent pass rate on his banking tasks, against 23.8 percent for direct generation from the same prompt. That is a bigger swing than anything in the review numbers, and it is where Garg locates the payoff of a written specification: hard, multi-constraint work built by a capable-but-imperfect model.

Where the payoff actually sits

Adding a specification to the reviewer's screen did not help them catch more bugs. Garg says the sample is small, and the p value on recall is 0.69. Every finding could be pinned to a written requirement rather than to a reviewer's private judgement. In regulated software that is the point of the exercise: the auditor asks which requirement a defect trips, and the answer stops being "the reviewer thought so".

The 21 extra minutes per review is the price. On a service with 10 or 11 defects that works out to 2.07 minutes of extra reviewer time per defect that can now be cited to a requirement rather than to intuition. In finance, health and other domains where an audit will land, that is a trade a team will take. On a script that nobody will audit, the same 21 minutes buys nothing.

The other trap in the numbers is that the recall figure of 0.525 is already low: reviewers caught half the defects, with or without a specification. Garg reads this as evidence that the specification changes the vocabulary of the review while leaving its depth alone. The finding fits the shape of the recall gap he measured, and it leaves the question of how to raise recall open.

Source

• Nitin Garg, "When Spec-Driven Development Pays Off," InfoQ, 2026-09-10