How many tests does AI-generated code need?

AI-generated code needs one check for every promise your product makes where a failure would cause harm, and a coverage percentage is the wrong way to count those checks. Reveneau writes all of its code with AI, and every change has to pass an eval suite written from the specification before it is released.
Coverage counts which lines of code executed during a test run. It does not count whether anything was proven about them. You can execute a line and assert nothing, and a large share of the tests in most real codebases do exactly that.
So here is the unit we use instead, and how to know when you have enough.
Count protected promises, not covered lines
Your product makes promises. A user's data is only visible to their own organisation. A payment is either taken and recorded or neither. An export contains every row the filter selected and no others. A deleted account's tokens stop working immediately.
For each promise, ask one question: is there a check that would fail if this stopped being true?
That is the unit. It is harder to measure than a percentage and it is the only one that tells you anything. A codebase with fifteen checks covering fifteen real promises is safer than one with two thousand tests covering none of them.
The exercise takes an afternoon. List your promises, honestly, including the ones nobody wrote down. Mark the ones with a real check behind them. The gaps are your work queue, in order of what a failure would cost. We wrote the full version of the method in how to write your first eval suite.
Why coverage targets cause harm
A coverage target does harm because of the work it rewards.
The cheapest way to raise coverage is to test the code that is easiest to test. That means mappers, formatters, plain data objects, small pure functions with no dependencies. Those are also the parts of a codebase least likely to cause an incident. Meanwhile the expensive risks are in the code that is hardest to test: permission logic mixed into a service, a migration with side effects, an integration that needs a real database to exercise properly.
A team managing to a coverage number will produce a beautiful report and leave the dangerous parts untouched, because each hour of work on the dangerous parts adds less coverage. A coverage target produces this result every time, unless somebody works against it on purpose.
Use coverage as a guide to where tests exist, if you like. Zero percent on a module tells you nobody has looked. As a target it sends effort to exactly the wrong code.
Where generated code needs more than usual
Some classes of check matter more when a machine is writing the implementation, and the evidence for this is specific rather than a feeling.
Veracode's spring 2026 update, published on 24 March 2026 and read on 1 November 2026, reports on 80 coding tasks and more than 150 language models tested to date. Only 55 percent of generation tasks produced secure code. Java had a 29 percent security pass rate, the worst of the languages tested, and cross-site scripting had a 15 percent pass rate. Syntax correctness exceeded 95 percent. Veracode says syntax pass rates have climbed steadily while security pass rates are virtually identical to where they stood two years ago. Its summary: "Models have become excellent at writing code that compiles. They've failed at writing code that's safe."
So security needs checks of its own. Give the common flaw classes explicit tests on the paths that matter: injection anywhere a string becomes a query, escaping wherever user text is rendered, authentication asserted by testing that the unauthenticated call is refused.
The second area is error paths. Generated code is strongest in normal use, because that is what most examples in any training corpus demonstrate. In Stack Overflow's 2025 survey, the leading frustration with AI tools, cited by 66 percent of respondents, was output that is almost right but not quite. Almost right nearly always means correct in the common case and wrong in an unusual one: an empty list, a duplicate submission, an expired token, a dependency that timed out.
A useful ratio, if you want one number: at least a third of your checks should assert that something is refused, rejected, or handled as an error.
The full list of what is worth checking, in priority order, is in what to check in an eval suite.
The two tests of whether you have enough
Forget the count. Two questions answer this better.
Would the suite have caught your last three incidents? Go and look. Pull the three most recent things that reached a customer, and ask what check would have stopped each one. If none of them were preventable by an automated check, your risk is elsewhere and you should stop adding tests. If all three were, you know exactly what to write next.
Do your engineers believe a passing build means the change is safe? Ask them, plainly, and listen for uncertain answers. A team that says yes will use the pipeline as a real approval step and let an agent work unattended. A team that says no will re-run builds, review defensively, and quietly rely on manual checking, which means the whole suite is producing a routine with no real value instead of confidence.
If the answer is no, more checks will not help. The problem is trust, and trust is broken by specific things: assertions that cannot fail, flakiness, and tests that mirror the implementation instead of the requirement. Those failure modes and their warning signs are in when evals give false confidence.
Deleting tests is part of the job
A suite gets better by removing tests as often as by adding them. Three categories are worth removing as soon as you find them.
Anything flaky. A check that fails one run in twenty costs more than it saves, because of what the team learns from it: a failed check might mean nothing, so re-run it. That lesson then applies to every other check in the suite, including the honest ones.
Anything asserting internal structure. Private call counts, class existence, the internal order of operations. These break on every refactor, and a team that spends its time updating tests to agree with new code has a copy of the code rather than a check.
Anything cosmetic. Exact copy, exact spacing, exact colour. Assert that the element is present and reachable. Do not make ordinary design work pay that cost.
What good looks like
We are wary of numbers in this piece for a reason, so here is what it contains rather than how big it is. A healthy suite on a product we would be comfortable running has: every promise that would hurt if broken covered by at least one check that has actually failed in real use, at least a third of its checks on refusals and error paths, explicit assertions on permissions and on the security classes generated code handles badly, migrations checked against data that looks like production data, nothing flaky in the blocking tier, and a real failure caught somewhere in the last month.
Count the checks if you want. It will be a smaller number than you expect, and it will do more work than the two thousand tests it replaced.
Coverage tells you what your tests ran. Only your incident history tells you what they were worth.
Related guide: Eval-driven development: how to prove AI-written code works.
Sources
- Veracode, Spring 2026 GenAI Code Security update, 24 March 2026, read 1 November 2026: 80 coding tasks, more than 150 models tested to date, 55 percent of generation tasks secure, Java at 29 percent, cross-site scripting at 15 percent, syntax correctness above 95 percent, and security pass rates virtually identical to two years earlier.
- Stack Overflow 2025 Developer Survey, AI section, read 1 November 2026: 66 percent cite "AI solutions that are almost right, but not quite" as their leading frustration.


