Calibrating Jev against your own human labels

LangChain's team published the most careful public test of Jev as a grader so far, and the honest thing about it is its size: five cases, run 100 times each per judge, with the authors calling the result "observational, not evidence" [1]. That is the right description for five cases. It is also the shape of the measurement you should run on your own suite, at a size that says something about your changes. Reveneau ran it before any Jev grade was allowed to block a merge and runs it again on every model version change, and this page is the method.

Why other people's numbers are not enough

A grader's accuracy is a property of the grader on a task, and your task is your changes. LangChain's 100 percent agreement on five weather-agent cases [1] and the 83 to 87 percent JevBench figure that Openlayer's README cites, which is intent classification on Banking77 and CLINC150 rather than grading [2], are results on other tasks by other people, and the README itself says calibration varies by task [2]. Neither number can be quoted about your suite, and a team that sets its thresholds from either has thresholds set by a benchmark it never ran.

The second reason is that the model can change underneath you. jev-latest is an alias that TypeSafe can point at a new release; jev-1.13.0 was released on 15 September 2026, and both jev-latest and jev-preview point to it today [3]. A new version can move every probability by a small amount, and a small move across a threshold is a changed verdict. A calibration run made once is a description of a version that will be replaced.

The third reason is the warning in LangChain's post that "low cost can amplify mistakes" [1]. A grader cheap enough to run on every change runs thousands of times, and a consistent error is a thousand consistent wrong grades unless something is counting. Calibration is the thing that counts.

Step one: the labelled set

Collect 50 to 200 cases from your own suite. A case is a change (a diff or a trace), the state you would send for it, the questions you would ask, and a label per question from a person who read the change against the criterion: pass or fail, with a sentence of reason. If your team already reviewed these changes, the labels exist; if not, this is a day of a senior engineer's time, and it is the cheapest day in the project.

Composition matters more than size. Include every change that failed in production in the period, because those are the cases the grader is for. Include changes that were escalated by a language-model judge if you had one. Include changes that were plainly fine, so that agreement on passes is measured too. And include the known-bad set from what a Jev grade cannot tell you: a planted approving comment, a moved count, a date comparison, each of which must fail. A set of 50 that is all easy passes tells you nothing.

The labels are the standard. A case where two reviewers disagree is not a case for the set until they agree, because the grader cannot be more consistent than the label, which is the same precondition using a model as a judge puts on any model grading.

Step two: the runs

Run every question on every case 10 times against the model version you intend to use. For a 200-case set with twelve questions per case, that is 2,000 requests, which at the published limits of 1,200 requests per minute and 250,000 tokens per second [3] is a few minutes with pacing, and at the public price of $0.042 per million input tokens it is under a dollar for a typical case size, as the pull request page works out. Log every run in full: model version, state hash, every probability, every confidence number.

Ten runs is the number that separates accuracy from stability. One run tells you whether the grade matched the label once. Ten tell you whether it matches every time, and a grader that matches nine times out of ten on the same input has a variance problem that a single run would have hidden. LangChain measured mean per-case variance on the quality score and found Jev's at 0.0000149 with the language models 92 to 913 times higher [1]; TypeSafe's own batching test found a standard deviation of 0.0 across repeats for 11 of 13 questions [4]. Those are the numbers to compare your own against. If your variance is far above them, the state or the question is unstable, and the fix is in the words before it is in the thresholds.

Step three: the measurements

For each question, four numbers.

Agreement: the share of case-runs where the grade, compared to a candidate threshold, matched the human label. Compute it across a range of thresholds, because the point of the run is to choose one.

False passes and false fails, separately, at each candidate threshold. A false pass is a change the person failed and the grader passed; it is the expensive direction. A false fail teaches the team to override the grader, which is the direction that ends the grader's usefulness. The older LLM-as-judge guidance said to track them apart, and that holds here.

Variance: for each case, how far the 10 probabilities spread. A case with a mean of 0.7 and a spread from 0.4 to 0.95 is a case the grader cannot decide, and it belongs in the middle band whatever the mean says.

Escalation rate: at each candidate pair of lines, how many cases would fall in the middle band. This is the number of changes a person will see per hundred, and it has to be a number the team can sustain.

Step four: thresholds from the result

For each check, choose the pass line as the lowest probability at which the false-pass rate on the labelled set is acceptable for that check's consequence, and the fail line as the highest probability at which the false-fail rate is acceptable. The gap between them is the middle band, and its width is the escalation rate you accepted. A money-moving check gets a false-pass tolerance near zero and a wide band; a naming check gets a looser tolerance and a narrow band, following thresholds, confidence and escalation. Start from TypeSafe's published bands of 0.9 and 0.5 [5] and move from there only on the strength of the measurement.

Write the thresholds into the reviewed file with a comment naming the calibration run: the date, the model version, the set size, the agreement at the chosen lines. That comment is what lets a reviewer, six months later, ask whether the run is still current.

Step five: repeat on every version change

The model version field on every request is what makes this possible. When it changes, or when you choose to move a pinned version, the full run happens again before the new version's grades are allowed to block. Compare agreement and variance per question against the previous run. A question whose agreement fell is a question whose threshold has to move or whose wording has to change; a question whose variance rose is one whose state has to be looked at. Nothing about the new version is assumed to be better or worse; the run says.

Run it monthly as well, against the stored labels from the uncertain band, even when the version has not changed, because the population of changes moves too: a new area of the codebase, a new agent, a new spec author. Openlayer's jevals ships a calibrate command that fits a threshold against your own labels and reports the error rate at it [2], which is the same step in tool form, and the labelled set it needs is the one this page builds.

The set is never finished

Every change a person labels from the uncertain band joins the set, with its state hash, its probabilities, its model version, and the reason. That is the mechanism that turns escalation from a cost into the grader's own eval suite, and it is the reason why we moved our eval grader to Jev calls the stored label the most useful artefact the new grader produces. The set grows with exactly the cases the grader found hard, which are the cases worth measuring on.

Two habits keep it useful. Prune labels whose question text has since changed, or re-label them against the new text, because a label against an old statement is a label for a different check. And keep the known-bad set fixed and separate, so it is a regression check rather than a moving target.

What this looked like for us

Before the first Jev grade blocked a merge at Reveneau, the judged checks ran in shadow, meaning both graders graded every change and only the old one decided, while the labelled set was built from our reviewers' decisions on those same changes. The thresholds came from the run, the model version was pinned, and the run is repeated on every version change and monthly against the stored labels. On our own suite the run is ten times faster than with the language-model grader. That figure came after the calibration, because a fast grader you have not measured is a fast way to be wrong. The sequence is in the migration checklist, and the definitions of the question types and the training method are in Jev and System One models.

Measure first. Then let it block.