Thresholds, confidence and escalation in an eval suite

When a Noul comes back at 0.74, something has to happen, and the thing that happens is the most important design decision in a Jev-graded suite. Pass it, and you have decided that three in four is good enough for this check. Fail it, and you have decided the opposite. Send it to a person, and you have admitted the grader was unsure and made the uncertainty someone's job. All three are defensible for some check. None is defensible for every check. This page is about choosing per check, and about what to do with the cases that go to a person. Reveneau grades its eval suite with Jev and routes every uncertain grade to a person, and the method below is the one we use.

The bands TypeSafe publishes

TypeSafe's confidence page gives three bands for its confidence number, which for Choice and Score questions is a single number from 0 to 1 computed from how spread out the probabilities are. At 0.9 and above, act automatically. Below 0.5, route to a human, request clarification, or fall back to a different system. In between, proceed with caution. The same page adds the sentence that turns those bands into a design: different actions within the same system should be gated at different levels depending on the consequences [1].

For a three-option Choice the docs give the confidence formula as three times the largest probability, minus one, divided by two [1], which means confidence 0.9 corresponds to a largest probability of 0.933 and confidence 0.5 to a largest probability of 0.667. Those are the docs' numbers for their example; they are a reference point rather than a rule for your suite.

A Noul has no confidence number, only the probability [2], so for a Noul the bands sit directly on the probability. A Score has both a combined score and a confidence number, and both matter: the score is compared to the pass line, and the confidence number decides whether the comparison can be trusted. The definitions are in Jev and System One models.

Three outcomes per check

The mistake teams make first is collapsing the bands to one line: above it pass, below it fail. That is what a unit test does, and it feels consistent. It is wrong for a probabilistic grade, because the grade near the line is the grader saying it does not know, and a single line converts "I do not know" into a confident verdict in one direction or the other. Every such case is either a change that failed for no reason the reviewer can see, which teaches the team to override the grader, or a change that passed when it should not have, which teaches nobody anything until production does.

So every check has three outcomes. Above the pass line, pass. Below the fail line, fail, and the change is blocked in the same way a failing test blocks it. Between the lines, route to a person. The width of the middle band is a choice: wide, and more changes go to people; narrow, and more uncertain grades become verdicts. A new suite starts wide and narrows as calibration shows where the grader can be trusted, which is the process in calibrating Jev against your own human labels.

Thresholds by consequence

TypeSafe's sentence about gating different actions at different levels is the rule that does the work. In an eval suite the "action" is passing a change, and the consequence of passing it wrongly varies by what the change touches.

A check on a flow that moves money, changes a permission, deletes data, or touches authentication has a high cost of wrong pass, so its pass line is high and its middle band is wide. We would set a Noul on such a check to pass only at 0.95 and above and to fail below 0.7, with everything between going to a person. A check on a naming convention has a low cost of wrong pass, so it can pass at 0.8 and fail at 0.4 with a narrow middle band, and a wrong pass costs a follow-up commit rather than an incident. The list of which flows are which is the same list as what to check in an eval suite, ordered by how much damage a failure does.

TypeSafe's own guardrail cookbook shows the shape with its numbers: two example policies, a review threshold of 0.35 and an action threshold of 0.70 for the strict policy or 0.85 for the permissive one, with routes checked in a fixed order [3]. Those are thresholds for screening a message rather than grading a diff, and the numbers do not transfer; the structure of a review line below an action line, set per policy, does.

The thresholds live in the same file as the weights from composite scoring for code quality, for the same reason: a threshold is pipeline configuration, and changing one is a reviewed change with a diff.

The Score's second condition

A Score at 2.3 on a scale where 2 passes looks like a pass. If its confidence number is 0.3, the probability was spread across levels, and the 2.3 is closer to a guess than a reading. TypeSafe's example makes the shape visible: a score of 1.43 with confidence 0.35 came from probabilities of 0.57 on level 1 and 0.43 on level 2 [4], and a pass line at 1.5 would fail it by a margin that the confidence number says is not meaningful.

So for a Score there are two conditions to pass automatically: the score is above the pass line, and the confidence number is at or above the automatic-action band, which we set following the docs' 0.9 for high-consequence checks and lower, after calibration, for the rest. A Score that passes on the number and fails on confidence goes to a person. This matters most on rubrics with many levels, where adjacent levels split the probability, and it is one of the reasons turning a rubric into Score questions argues for few levels.

What the person gets

A change routed to a person arrives with everything the grader saw and everything it returned: the diff, the quoted spec line or rubric, the question text, the probability or the score and its per-level probabilities, the confidence number where there is one, and the model version. The person's job is the same as a reviewer's in the older process described in using a model as a judge: read the diff against the criterion and decide.

The person records a label, pass or fail, and a short reason. The reason is the rationale the model could not write, and it is worth a sentence rather than a click, because it is what tells you later whether the question was ambiguous, the change was borderline, or the model was wrong.

Storing the label

This is the step that turns escalation from a cost into an asset. Every label is stored as a row: the change identifier, the question identifier and its text at the time, the state hash, the probability or score, the confidence number, the model version, the person's label, the reason, and the date. Nothing is discarded, including labels that agree with the grade, because agreement is data too.

Two uses follow. The first is the monthly re-check. Take every stored label, re-run the same questions on the same states against the current model version, and measure agreement between the grade and the label. A grader whose agreement has fallen shows up in this number before it shows up as a bad merge. LangChain's test, which measured agreement with human labels over repeated runs on five cases, is the shape of the measurement, and its authors' warning that low cost can amplify mistakes is the reason to run it on a schedule [5]: a cheap grader that is wrong in a consistent way is wrong many times before anyone notices, unless something is counting.

The second use is threshold adjustment. If reviewers pass 90 percent of the changes routed to them from a given check at probabilities between 0.6 and 0.8, the middle band on that check is too wide and its fail line can come down. If reviewers fail a share of changes the grader passed at 0.85, the pass line has to go up. Every such adjustment is a pull request against the thresholds file, with the calibration numbers in the description.

When the model version changes

jev-latest is an alias, and TypeSafe can point it at a new release; the current version, jev-1.13.0, was released on 15 September 2026, and jev-latest and jev-preview both point to it [6]. A new version can move every probability in the suite by a small amount, and a small movement across a threshold is a changed verdict. So the model version is logged on every request, a change in it triggers the full re-check against stored labels before the new version's grades are allowed to block, and the thresholds are re-derived if agreement moved. Pinning the exact version is the safer default for a blocking suite; the trade is that you choose when to move rather than having it chosen for you.

What escalation is for

Sending a change to a person is the grader doing the one thing a probabilistic check can do that a unit test cannot: say how sure it is. A suite that never routes anything to a person has either a grader that is always confident, which should worry you, or a middle band of zero width, which means uncertainty is being converted into verdicts nobody reviews.

At Reveneau the uncertain band goes to a person on every change, the label is kept, and the grader is re-checked against those labels on a schedule and on every model version change. The suite runs ten times faster than it did with a language-model grader, on our own measurement, and the people in it spend their time on the cases the grader was honest enough to flag. That trade is the point.