How do you test something that has no single correct answer?

Stop asserting the exact output and assert properties of it instead: things that must always hold and things that must never happen. A summary has no one right form, and it can still be required to avoid naming people absent from the source, to stay within a length limit, and to refuse empty input rather than invent content.

What is a property-based check in this context?

A rule about the form or content of any acceptable answer rather than about one specific answer. It converts an unanswerable question, whether this output is the best one, into an answerable question, whether this output belongs to the set of acceptable ones.

Why is asking a model to grade its own output unreliable?

Because the grader shares the gaps in understanding of the thing being graded, so it tends to approve exactly the outputs that needed a second opinion. It can be useful for sorting large volumes for human attention, and it should not be the check that decides whether something is released.

How many real examples do you need to start?

Around twenty to thirty, chosen to cover the awkward cases rather than the typical ones. A small set of well-selected examples with genuine human judgments attached is more useful than a large generated set, because the generated set has the same assumptions as whatever produced it.

Which cases should the example set cover?

The empty input, the very long input, the input in an unexpected language, the input containing something that looks like an instruction, and the cases where the correct behaviour is refusing to answer. Those unusual inputs are where real failures gather, and an example set of normal inputs will never include them.

What should never be left to a probabilistic check?

Anything with a definite right answer, including permissions, money arithmetic, and whether a record was written or deleted. Those get ordinary deterministic checks, and mixing them into the uncertain category is how a hard requirement quietly becomes a preference, which is most likely to happen to the checks that sit closest to the model.

How do you handle the case where the feature should refuse?

Treat refusal as a required behaviour with its own checks, rather than as an unusual case. Features that will answer anything are the ones that invent content when given nothing useful, so the ability to decline needs testing as deliberately as the ability to respond.

What does a regression look like for this kind of feature?

A change in quality across many outputs rather than a single broken case, which is why you compare a fixed example set before and after every model or prompt change. Without that fixed set there is no way to distinguish a real decline from normal variation.

How does permission testing work for a generated answer?

By running the same question as two accounts entitled to different data and comparing what comes back. The failure here produces a fluent and helpful answer rather than an error, so only a comparison between accounts reveals that content reached a user who should not see it.

When should you accept that something cannot be tested automatically?

When the judgment genuinely requires a person, such as whether a tone is appropriate for a particular audience. The correct response is scheduling human review on a sample and recording that decision, rather than pretending an automated check covers it.

How to test a feature you cannot fully specify: Reveneau

Q: How do you check for stability rather than correctness?

Run the same input more than once and compare the outputs for meaning rather than for exact characters. A feature that returns two materially different answers to the same question is unreliable regardless of whether either answer was defensible on its own.

"We cannot really test this, there is no right answer."

We hear this on almost every AI feature, and it is partly true, and it leads teams to a bad conclusion. The true part: there is no single correct summary of a document, no single correct ranking of search results, no single correct suggested reply. Checking for an exact match with a stored expected output will fail constantly for reasons that have nothing to do with quality.

The false part, and it is the important part: plenty of outputs are definitely wrong, and those are testable today.

Stop asking whether it is right

The question "is this the best possible summary" has no answer you can write into a test. The question "does this summary name a person who never appears in the source document" has a clear answer, and a wrong answer there is a defect a user would report.

That is the whole method. Check properties rather than outputs: check that every clearly unacceptable answer is stopped.

For most features, the property list is shorter than people expect and can be written in an afternoon. A summarising feature might require:

No named person, company, date, or figure that is absent from the source.
Output within the length the interface can actually display.
A refusal when the input is empty or unusable, rather than invented content.
Nothing drawn from material the asking user has no permission to see.
Stable meaning when the same input is submitted twice.
The original language preserved unless translation was requested.

None of those requires knowing the ideal output. Every one of them is a real failure someone would complain about.

Separate the uncertain part from the predictable parts

The mistake that follows from "there is no right answer" is treating the whole feature as unmeasurable. Most AI features behave predictably in most parts, with one uncertain part in the middle.

Whether the right records were retrieved is a fact. Whether the user was allowed to see them is a fact. Whether the request was logged, whether the rate limit applied, whether a failure returned a sensible error, whether the total on the screen matches the total in the database: all facts, all testable in the ordinary way.

Decide clearly which parts are which and test them differently. The common failure is treating a firm requirement as uncertain because it happens to be close to the model, and permissions are where this happens most often. There is no probabilistic version of "this user may not see this document". We have written about why that specific rule is hard in why permissions are the hardest part of an AI feature.

Build a small set of real examples

Collect twenty to thirty real inputs with human judgments attached. Not the typical cases: the awkward ones.

The empty input. The very long input. The input in a language nobody planned for. The input containing something that looks like an instruction. The input where the correct behaviour is to refuse. The input where two reasonable people would disagree about the right answer, which is valuable precisely because it shows the limit of what you can check.

Thirty well-chosen cases beat a thousand generated ones, because a generated set has the same assumptions as whatever produced it and will systematically miss the same things the feature misses. Getting real examples takes an afternoon of asking people for the requests that went badly, and that afternoon is the most valuable work in the whole exercise. The method is in how to write your first eval suite.

Keep this set fixed. Its main job is comparison: run it before and after every prompt change, model version, and retrieval adjustment. Without a fixed set there is no way to tell a genuine decline from normal variation, and you end up arguing about whether the feature feels worse.

The problem with a model as grader

The obvious shortcut is to ask a model whether the output was good. It is fast, it handles large volumes, and it produces a number.

It also shares its failure modes with the thing it is grading. A grader misunderstanding the domain in the same way as the generator will approve confidently at exactly the moments you needed an independent opinion. The number looks like measurement and carries much less information than it appears to.

There is a reasonable use for it: sorting large volumes of output so a person looks at the most suspicious cases first. That is sorting by priority, and it is genuinely useful. It should not be the check that decides whether something is released. Our fuller argument is in never let the model grade its own work and LLM as judge for code review.

Test refusal as a feature

Anything that will always produce an answer will invent one when given nothing useful. That is the source of a large share of the worst outputs these features produce.

So specify what the feature should decline to do, and check it. Empty input, contradictory input, a question outside its scope, a request for information it has no source for. Refusing well is a behaviour with a correct answer, which makes it one of the easier things on this list to verify.

Teams often discover here that nobody ever decided what the feature should do when it does not know. That conversation is worth having before launch rather than after.

It also contains a product decision. A feature that declines is admitting a limit, and somebody has to agree that admitting the limit is better than guessing. In most cases it clearly is, because a confident wrong answer costs more trust than an honest "I do not have that information" ever does. But it is a decision, and it should be made by the person who owns the product rather than left to whoever wrote the prompt.

Stability is a property you can measure

Run the same input several times and compare the outputs for meaning rather than exact characters.

A feature that returns two materially different answers to the same question is unreliable, whatever the quality of either answer. Users notice this quickly, and they lose trust in it faster than they do for a feature that is consistently mediocre. Consistency is a real quality attribute and it happens to be one of the easiest to check.

Where automated checks cannot help

Some judgments need a person. Whether the tone suits this audience. Whether the summary emphasised the thing that actually mattered. Whether the answer, though accurate, was unhelpful.

The right response is to schedule human review on a sample and write down that this is how that part of quality is covered. What goes wrong is that nobody notices this part of quality has no check, because the dashboard shows every check passing, and those checks cover only the things that were easy to measure.

This is the same reason the DORA research is worth reading alongside any quality programme: the 2025 report, drawn from nearly 5,000 professionals, found AI adoption relating positively to delivery throughput and negatively to delivery stability, and summarised the pattern as "AI doesn't fix a team; it amplifies what's already there". Faster output through a process that measures only the measurable parts produces more of exactly what nobody was checking.

Thanks to the teams who sat down and listed everything their feature must never do. That list was always shorter than they expected and always contained two or three things nothing was currently preventing.

You do not need to know the right answer to recognise a wrong one, and most of the value is in the second.

Related guide: How to evaluate AI quality.

Sources

2025 DORA report announcement, Google Cloud: nearly 5,000 respondents; AI adoption relates positively to delivery throughput and negatively to delivery stability, and amplifies a team's existing strengths and dysfunctions.

How to test a feature you cannot fully specify