What makes an acceptance criterion machine-runnable?

It names four things: the actor performing the action, the action itself, the observable result a caller can actually see, and the limit condition being tested. A criterion missing any of the four requires a human to supply the missing piece, and two humans will supply different pieces.

Why is "users should only see their own data" not testable?

Because it does not say which users, what counts as seeing, or where the limit is. A test written from it might check the list endpoint and pass while the search endpoint, the export, and the not-found error message all expose data. The vague version cannot fail in the places that matter.

Should acceptance criteria be written before or after the code?

Before, and this matters more with generated code than it did with hand-written code. A criterion written after the fact tends to describe what the code does rather than what the product requires, which produces a check that can never fail and gives false confidence.

Who should write acceptance criteria: the product owner or the engineer?

Both, in one sitting. The product owner supplies the rule that must hold and the engineer supplies the observable that proves it. Written separately, you get criteria that are either untestable or that test implementation details nobody outside the code cares about.

How specific is too specific?

A criterion becomes too specific when it names an internal function, a database table, or a code path, because then it fails whenever the implementation changes even though the product behaviour did not. Keep the sentence at the level a caller can observe and the check keeps working after refactoring.

What about criteria that are genuinely subjective, like "the page should feel fast"?

Convert them to a measurable substitute and accept that the substitute is imperfect. "Largest Contentful Paint under 2.5 seconds at the 75th percentile on a mid-range Android device" is testable, whereas "feels fast" is only an opinion. If no honest substitute exists, keep it as a design review item rather than pretending it is a check.

How many acceptance criteria should one requirement have?

Usually three to six, and if a requirement needs fifteen it is really several separate requirements that were given one name. The count matters less than the coverage: write one criterion for normal use, and one for each way a caller could go past a limit.

Does this approach work for AI features whose output is not deterministic?

Yes, with a change from exact-match assertions to property assertions. You cannot assert the exact wording of a generated answer, but you can assert that it cites a source present in the retrieved set, that it never returns a field the user lacks permission for, and that a refusal happens on out-of-scope questions.

What is the fastest way to improve an existing set of criteria?

Take the ten criteria attached to the riskiest part of the product and rewrite each one to name an actor and an observable. It usually takes an afternoon, and the rewriting exercise itself reveals two or three rules the team believed were enforced and were not.

How does this relate to writing specs for an AI coding agent?

It is the same practice applied earlier. An agent will implement what a sentence literally says, so an ambiguous criterion produces confidently wrong code rather than a clarifying question. Precision in the criterion is what makes the generated implementation checkable rather than merely plausible.

How to write an acceptance test a machine can run: Reveneau

Over the past year we have read a lot of other people's acceptance criteria. Some arrive as Jira tickets, some as a paragraph in a product brief, some as a Slack message that became a requirement by accident. Almost all of them share one property: they were written for a human reader who would add the missing details.

A machine adds nothing. That difference is the entire subject of this piece.

Here is the sentence we see most often, in some form, on nearly every project:

Users should only see their own data.

Everybody agrees with that. Nobody can test it. Which users? Their own by what definition, ownership or membership or delegation? Seen where: in the list, in the search results, in the CSV export, in the error message that helpfully names the record that was not found?

The version a machine can run says this instead:

A request authenticated as a member of organisation A that requests a record belonging to organisation B returns 404, and the response body contains no field from that record.

That is one sentence and an automated check can enforce it. It also catches the data exposed through the error message, which the original wording would have allowed without question.

Why this matters more now

When a person wrote every line, ambiguity in a requirement got resolved by a person who had context. They read "users should only see their own data", pictured the actual product, remembered the delegation feature added last spring, and wrote something reasonable. The requirement was vague and the outcome was often fine, because the missing details were supplied by judgment that never appeared in writing.

Generated code handles this differently. A model implements what the sentence literally says, quickly and confidently, and it does not stop to ask which of the four readings you meant. What comes back compiles, looks right, handles normal use, and is wrong at exactly the limit the vague sentence failed to name.

The measurements of this pattern are clear. Veracode's Spring 2026 GenAI code security research tested more than 150 models and found that generated code compiled and ran correctly over 95 percent of the time while being secure in only about 55 percent of generations. Syntax is solved. The limits are not. And the limits are precisely where an imprecise requirement leaves the decision to the model.

So the sentence now has to do the job that human judgment used to do.

The four rules

Every criterion we rewrite goes through the same four questions. They are simple and repetitive, and that is intended.

1. Name the actor, not "the user"

"The user" is a placeholder for whichever user makes the sentence true. Replace it with a role that has specific permissions and a specific relationship to the data.

Before: An admin can delete a project. After: An org admin can delete a project in their own org. An org admin who requests deletion of a project in another org receives 403 and the project still exists after the call.

The second version has two actors, one of whom is the one you actually worried about.

2. Name the observable result

A check can only assert what a caller can see. Status codes, response fields, database state after the call, the contents of an emitted event, a row in an audit log. If your criterion asserts something that cannot be seen from outside the code, it will end up asserting an implementation detail and it will break on the next refactor while catching nothing.

Before: Deleted records are soft-deleted. After: After DELETE /projects/:id returns 204, GET /projects/:id returns 404, the project does not appear in GET /projects, and an audit event of type project.deleted exists with the actor id and a timestamp.

Soft deletion is a mechanism. The three assertions are the behaviour anyone actually depends on.

3. State the limits, not normal use

Normal use is the case that already works. It works because it is the case everybody had in mind while building. Writing a check for it is not free, and it is not where the defects are.

Limit cases worth a sentence each: empty input, an input at the maximum size, a second concurrent request, an expired token, a user whose permission was revoked between authentication and the call, a record that was deleted between the read and the write, a numeric field at zero and at negative.

We wrote about where to focus this attention first in what to check in an eval suite, and the underlying reason generated code fails at the limits rather than in normal use is covered in is AI-generated code secure.

4. Write the failure you fear, then negate it

This is the one that helps people who do not know how to start an empty criteria field.

Do not start from "what should this do". Start from "what news headline do I not want". Then write the sentence that makes that headline impossible, and that sentence is your criterion.

Feared headline: Customer's data appeared in another customer's export. Criterion: An export requested by a member of org A contains zero rows whose org_id is not A, verified on a fixture where both orgs have records with adjacent ids.

The "adjacent ids" clause is there because that fixture is how you catch an off-by-one in a range query, and a person who has debugged one of those knows to ask for it.

A worked example, start to finish

Take a realistic requirement: a customer can invite a teammate.

Written for a human, that is a sentence and everyone starts work. Written for a machine, it breaks down into criteria that name actors and observables:

An org admin who posts a valid, previously-unseen email address to the invite endpoint gets 201, and an invitation row exists with status pending and a random token that is at least 32 bytes long.
A member without the admin role who posts to the same endpoint gets 403 and no invitation row is created.
Inviting an email that already belongs to an active member of the same org returns 409 and creates no second invitation.
An invitation token accepted after its expiry timestamp returns 410, and the account is not created.
An invitation token is single-use: a second acceptance of the same token returns 410 and creates no second membership.
The invitation email body contains the token exactly once and contains no other org's name or address.
An org admin can list pending invitations for their own org and receives 403 for another org's.

Seven sentences, each one runnable. Item 5 is the one teams forget, and it is the one that turns into an incident about duplicate memberships nine months later.

Notice that none of the seven mention a framework, a table, or a function. You could rewrite the entire implementation and all seven should still pass. That property is what separates a criterion from a test that only copies the code, a failure mode we describe in when evals give false confidence.

Non-deterministic features need property assertions

The objection we hear most is that this works for CRUD and does not work for AI features, where the output is different every time. It changes the assertion, not the practice.

You cannot assert the exact wording of a generated summary. You can assert:

Every citation in the answer points to a document that was actually in the retrieved set.
No field the requesting user lacks permission for appears in the output, checked against a fixture where the corpus contains such a field.
A question outside the configured scope produces a refusal rather than an answer.
The answer is under the length limit that the interface can display.
Two runs on the same input do not disagree on a factual field that came from structured data.

Those are properties rather than exact matches, and they are enforceable. The full explanation is in how to evaluate AI quality, and the specific trap of asking a model to certify its own output is in never let the model grade its own work.

What good looks like on a real backlog

A criterion is finished when you can hand it to someone who has never seen the product and they can write the check without asking a question. That is the whole standard. It sounds high and it takes about four extra minutes per criterion once the habit forms.

Two practical notes from doing this on real projects.

First, the rewriting exercise finds bugs before any code is written. Roughly a third of the time, writing out the boundary case for an existing feature reveals that the rule everyone believed was enforced is not. That finding is free and it arrives during a planning conversation instead of during an incident.

Second, do not do this to your entire backlog. Take the requirements attached to the parts of the product where a failure that nobody notices would be expensive: permissions, money transfers, data deletion, anything a customer makes a decision from. Twenty criteria rewritten properly are worth more than two hundred rewritten carelessly, and the twenty are the ones that would have caused damage.

Thanks to the product managers who let us rewrite their criteria in a working session and argued about every clause. The arguments were the useful part, because each one was an unclear point that a model would otherwise have decided without telling anyone.

A requirement a machine cannot check is not a requirement. It is only a hope, written in a ticket.

Related guide: Writing specs an agent can verify.

Sources

Veracode, Spring 2026 GenAI Code Security update: across more than 150 models, generated code compiled and ran correctly in over 95 percent of cases while being secure in roughly 55 percent.

How to write an acceptance test a machine can run