Writing Noul checks for acceptance criteria

An acceptance criterion is already almost a Noul. "An expired token returns 401 with the error code token_expired" is a sentence that is either true or false of a change, which is exactly what TypeSafe's Noul primitive grades: a yes-or-no statement, returning one probability from 0 to 1 [1]. The gap between the criterion and a good Noul check is in the wording, and this page is about closing it. Reveneau grades the acceptance criteria in its eval suite with Noul checks written from the specification before the code exists, and the rules below are the ones we apply.

The statement is read literally

TypeSafe's jaggedness page opens with the property that governs everything on this page: the model "answers the question you wrote, not the one you meant", and scoping words, negations, and implied conditions are read exactly as written [2]. A person reading an acceptance criterion fills in what the author meant. A decision model reads the sentence.

That has an immediate consequence for how you write. "The change handles expired tokens correctly" is a sentence a person understands and a model cannot grade, because "correctly" points at a standard that is not in the sentence. "The diff returns HTTP status 401 when the token's expiry time is in the past" names the standard. The difference is the same one that separates a testable spec line from a vague one, and writing specs an agent can verify covers the habit for the spec itself. A Noul is the spec line, tightened once more.

Rule one: positive statements only

TypeSafe says double negatives and indirection are answered less reliably [2], and a negation in a check has a second cost that is worse than reliability: it flips the meaning of the probability. If the statement is "the diff does not handle the expired-token case", a probability of 0.9 means the change is bad, and every threshold in the pipeline has to be inverted for that one check. Someone will get it wrong.

So every Noul asserts the good outcome. "The diff adds handling that returns 401 when the token is expired." High probability, good; low probability, bad; the same threshold logic for every check in the suite. If the criterion in the spec is written as a prohibition ("the endpoint must not return the user's email in the error body"), rewrite it as the positive form of what should be true ("the error body returned by the endpoint contains only the error code and message fields"). The rewrite is also more precise, which is usually the case.

Rule two: one thing per statement

"The diff returns 401 with the error code token_expired and logs the failure" is three claims in one sentence, and a probability of 0.6 on it tells you nothing about which claim the model doubted. TypeSafe's design guidance is to break a broad judgment into atomic questions asked in parallel [3], and since every question in a request is evaluated in parallel with almost no added response time [4], there is no cost to splitting.

Split until each statement could be wrong on its own. Then a low grade points at one thing, which is most of what a written rationale would have given you. The pattern for combining the results back into one verdict is in composite scoring for code quality; for acceptance criteria the combination is usually that every statement must pass.

Rule three: the evidence is in the state

The state is the only thing the model sees. TypeSafe names hidden context, relying on model knowledge instead of the state, as an anti-pattern [3], and separately warns that accuracy falls as the state grows with content unrelated to the decision [2]. Both point the same way: the state should hold exactly what the statement refers to.

For an acceptance-criterion check on a code change, that is the diff, or the part of it that touches the behaviour, and the spec line itself, quoted. If the statement says "the error code the spec names", the spec line with the error code is in the state. If the check is about an agent's trace, the trace is in the state and the plan is next to it. What is not in the state: the whole repository, unrelated files in the diff, the previous conversation with the agent. TypeSafe's context limit is 32k tokens for the state plus the longest question [5], and a large diff should be split by behaviour rather than truncated, which running Jev-graded evals on every pull request covers.

Rule four: name the evidence in the statement

A statement that says "the diff adds handling" is better than one that says "handling exists", because it tells the model where to look. Better still is a statement that names the observable artefact: "the diff adds a branch that returns status 401 and the error code token_expired when the token's expiry is in the past". A reader could find that branch. The model, reading literally, has the words to match against.

This is the same rule as the observable-behaviour rule for Score levels in turning a rubric into Score questions. A Noul is a one-level rubric.

A worked example

Spec line: "When a request carries an expired access token, the API returns HTTP 401 with the JSON error code token_expired, and the failure is logged with the request identifier and without the token value."

A person reads that as one requirement. It is four claims, and three of them are judged checks. The status code can be a deterministic test: send a request with an expired token to the running service and assert 401. That check stays in code and never goes to a model, in line with the rule to keep deterministic rules in code [3]. The other three become Noul statements, with a state of the diff plus the quoted spec line:

Statement one: "The diff adds a code path that returns the JSON error code token_expired when the access token's expiry time is in the past."

Statement two: "The diff logs the expired-token failure with the request identifier."

Statement three: "The log line written for the expired-token failure contains no part of the token value."

Each is positive. Each is one claim. Each names something in the diff. Statement three is the interesting one: the spec wrote it as a prohibition, and the positive rewrite ("contains no part of the token value") is still a positive statement about what the log line contains. It is a single positive claim and depends on no counting.

The three probabilities come back in one request. All three must pass for the criterion to pass. A low grade on statement two says the logging is the problem, and a person opening the change knows where to look.

Reading the probability

A Noul returns one probability and, unlike Choice and Score, no separate confidence number [1]. So the uncertain band is set on the probability itself. A statement at 0.93 passes. A statement at 0.12 fails. A statement at 0.55 is the model saying it cannot tell, and that change goes to a person, whose label is kept. The bands and how to set them by consequence are in thresholds, confidence and escalation, and the method for measuring where your own bands should sit is in calibrating Jev against your own human labels.

One caution on interpreting a middle probability. It can mean the change is borderline, or it can mean the statement was ambiguous and the model split its reading of the words. The second is more common in a new suite, and the label a person gives tells you which. If the reviewer says "obviously passes", the statement needs rewriting.

What a Noul cannot check

Anything that needs counting, comparing numbers, or ordering dates. TypeSafe lists all three as weaknesses [2], so "the diff adds at most two new dependencies" or "the retry delay is longer than the timeout" are deterministic checks, however tempting the yes-or-no shape is. Anything that needs knowledge outside the state, which is the hidden-context anti-pattern. And anything that needs a reason with the verdict, because a Noul returns a number and nothing else; what a Jev grade cannot tell you is the full list.

The habit

Write the criterion as a positive sentence about one observable thing, put the evidence next to it in the state, and split until each sentence could be wrong alone. Then the probability means what the sentence says. Every Noul in the Reveneau suite is written from the spec before the code exists, which is the order eval-driven development describes and the reason a grader can be trusted to grade against a standard the code did not write. The statement is the standard. The diff meets it or it does not.