What is a good definition of done for an AI-generated feature?

A definition of done for an AI build should cover three things: the feature passes acceptance tests written from the specification before any code was generated, those tests run automatically on every change, and a human has reviewed the output for correctness in the cases the tests cannot reach. A test suite written after the generation, to match what the code does, is a description of what got shipped, and nothing more.

Why is defining done harder when AI writes the code?

Because generated code can look finished before it is. A human writing code usually stops when they cannot proceed, which makes the state of incomplete work visible. Generated code fills in confidently regardless, so a file that compiles and passes surface checks can still be missing a business rule or mishandling an edge case.

Should acceptance criteria be written before or after generation?

Before. Criteria written after generation are shaped by what the code already does, which means they cannot test what it was supposed to do. Write the acceptance tests from the specification, run them against the generated output, and treat any test that fails as a gap to fix rather than a test to revise.

What belongs in the done criteria for an AI build that handles user data?

At minimum: the feature passes tests covering the happy path, the main failure cases, and every case where user data is written, deleted, or shared. Those tests must pass with a real database, because generated code that handles data correctly in a mock can still write to the wrong table or leak a row in production.

How many test cases is enough for a definition of done?

Enough to cover every distinct outcome the specification names. A checkout flow with three payment states needs three passing tests at a minimum. The question is whether every outcome the specification says should happen has a test that proves it does.

Can an eval suite replace a human review in a done check?

An eval suite covers what it was written to cover, which is the part of a definition of done that can be specified in advance. What it cannot cover is a reviewer reading the code and noticing that a function does something the specification never mentioned: logging a password in plain text, or silently ignoring an error that should surface to the user.

What is the difference between a passing build and done?

A build passes when the code compiles and the automated tests run without failing. Done means the feature does what was specified, the tests prove it does, and a person has checked it in context. Many teams treat a passing build as done and discover the gap when a feature that compiled and tested clean turns out to mishandle a case the tests did not cover.

How should done criteria change as a project grows?

Add outcome categories as the product adds them. A new payment method adds a new set of done criteria. A new user role adds permission boundary tests. Done criteria written at the start of a project and never updated guarantee that new features ship without a check for the cases those features introduce.

What is the right person to write done criteria on an AI build?

The person who understands the business rule the feature is supposed to enforce, usually a product manager or technical lead. The job of done criteria is to specify what correct looks like from the outside, which requires understanding the problem the feature solves.

Does the done definition change when the whole product is AI-generated?

The form stays the same and the strictness increases. A product where every change is generated has a higher chance of a subtle regression on each change, because the generator has no memory of the invariants the product depends on. The done criteria for each change should include a check that the invariants it touches still hold, not just that the new feature works in isolation.

How to define done on an AI build: Reveneau

The question "when is a feature done?" used to have a simple answer. The code worked, the tests passed, someone reviewed it, and you shipped it. When a person wrote the code, the state of incompleteness was usually visible: a half-written function, a missing branch, a TODO in the diff. You could see where the work stopped.

Generated code removes that signal. A model fills in every gap with something that compiles. The result looks complete. The tests pass. The diff is clean. And somewhere in the middle of it, a business rule the specification required is absent, quietly replaced by something plausible that the model inferred from the surrounding code.

This is why defining done on an AI build requires more precision than it did before.

The gap between plausible and correct

When AI generates code, the output is shaped by patterns the model has seen. It produces the most likely code for the context it has. For standard operations, writing a record, sending a request, handling a response, the most likely code is usually correct enough. For the specific rules your product enforces, which users can see which records, what happens when two operations race, what a failure here is supposed to surface to a user there, the model's best guess is frequently off in ways that do not cause an error.

This is the specific failure mode that a definition of done has to catch. A passing test tells you the code ran. It does not tell you the code ran the right business rule.

Teams that discover this problem do so after a feature ships and a user finds the wrong behavior. A checkout that lets a discount code apply twice. A role-based screen that shows fields it was supposed to hide. A deletion that works but does not clean up the related records. Each of these is a case the specification covered and the generated code missed, and each passed a build check.

Write the criteria before the generation

The most important procedural change for an AI build is the same one it has always been for good engineering: write the acceptance criteria before the code exists.

When criteria are written after the generation, they are shaped by what the code already does. The reviewer reads the code, sees that it handles three cases, and writes three tests that pass. What they cannot see is the fourth case the specification required that the code never handled.

Criteria written before generation describe what the feature must do from the outside. They are written from the specification, by the person who understands the business rule. When those criteria are handed to the generation step as a check to pass, the generator has explicit targets. When they are run after generation and one fails, that is a gap to close, not a test to revise.

This is the same principle behind test-driven development, restated for a world where most of the code is generated: the check shapes the code, not the other way around.

What done criteria must cover

For an AI build, the minimum definition of done has three layers.

The happy path, in full. Every outcome the specification says should happen has a test that proves it does, end to end, with a real database if the feature touches data. A test that works against a mock but fails against the real storage layer is not a passing test.

The main failure cases. A missing required field. A network timeout. A duplicate record. A user trying to do something they are not allowed to do. These are the cases where generated code most often produces something that compiles but does the wrong thing, usually because the generator picks the most common handling pattern rather than the specific one your product needs.

The permission boundaries. For anything that touches user data, the done criteria must name the specific rows or operations each user role can and cannot reach, and must include a test for the boundary. Generated code handles permissions correctly in the obvious case, the same user, the same role, the same resource, and incorrectly in the edge cases, like a user viewing a resource they own alongside a resource that belongs to another account.

The two-stage gate

A definition of done that works for an AI build runs two distinct gates before a feature closes.

The first gate is automated. The acceptance tests, written from the specification, pass on every run. This catches the gaps that can be specified: missing outcomes, incorrect permission behavior, broken failure handling.

The second gate is a person reading the code in context. This catches what tests cannot reach: logic that is technically correct but misses a subtle constraint, error handling that swallows a failure that should surface to the user, a function that does something the specification never mentioned. The reviewer is not checking whether the code compiles. They are checking whether the code does exactly what was required and nothing else.

Both gates are necessary. An automated suite with no human review ships the things tests cannot catch. A human review with no automated suite relies on the reviewer to catch everything, which is exactly the job you asked the generator to help with.

What done criteria are not

Done criteria are not a description of what the code does. They are a description of what the feature must do. That distinction matters because it determines who writes them and when.

If the person running the generation writes the criteria after reviewing the output, the criteria describe the output. If the person who owns the business rule writes the criteria before the generation runs, the criteria describe the requirement. Only the second version is a check.

Done criteria are also not test code. They are a list of outcomes, written in plain language, that the test code then proves. A team that skips the plain-language list and goes straight to writing tests ends up with tests shaped by the code rather than by the requirement, for the same reason criteria written after generation are shaped by the output.

Keeping criteria current

Done criteria written at the start of a project and never updated stop working as a project grows. A new user role adds permission boundary cases that the original criteria do not cover. A new payment method adds state combinations that did not exist before.

The practical fix is simple: treat the done criteria file the same way you treat the specification. When a new piece of the system is added, add the done criteria for it at the same time. The team reads the criteria before closing any ticket. Any ticket that closes without meeting all of them is not done.

The harder habit to build is treating a failed done criterion as a gap to close rather than a criterion to revise. Generated code that cannot pass a criterion written from the specification is code that does not meet the specification. The right response is to fix the code, not to relax the bar.

Sources

Beck, Kent. Test Driven Development: By Example. Addison-Wesley, 2002.
OWASP. OWASP Testing Guide. Open Web Application Security Project.

How to define done on an AI build