Why we moved our eval grader to Jev

The slowest step in our pipeline was a paragraph nobody read. For every change, after the tests and the type check and the schema validation had passed, a language model received the diff and the acceptance criteria and wrote out its reasoning before ending with a verdict. The paragraph was the cost of getting the verdict. It took seconds per check, it was different every run, and when two runs disagreed there was no way to tell which paragraph to believe. We replaced that step with Jev this month, and this page is the record of what that meant in practice.

Two categories, and only one moved

Every check in the Reveneau suite falls into one of two categories, and the categories were already separate before Jev existed.

The first category has a deterministic answer. A unit test passes or fails. A schema validates or it does not. A migration run against a copy of production data either keeps every rule that must always hold or breaks one. A type checker either finds an error or finds none. These checks are cheap, repeatable, and exact, and no model has ever been part of them. The reasoning is the one TypeSafe gives in its own design guidance, keep control flow, deterministic rules, and side effects in code [1], and it applied before we had read that page. A model can only make an exact check less exact.

The second category has no exact answer. Does this change implement the acceptance criterion in the spec? Did the diff stay inside the files and behaviours the task named? Did the agent's trace follow the plan it was given? Does the generated text follow the instruction in the prompt? For these we needed a judge, and the judge was a language model. The design of that judge, its independence rule and its validation, is in using a model as a judge, without fooling yourself, and everything on that page about rubrics and known-bad examples still holds. What changed is the model behind the check.

What a judged check looks like now

Each judged check is one question, and the criteria are written into the question. Two shapes cover almost all of them.

For an acceptance criterion, the check is a Noul: a yes-or-no statement that the model scores with a single probability from 0 to 1 [2]. A spec line that reads "an expired token returns 401 with the error code token_expired" becomes a statement such as "The diff adds handling that returns status 401 with error code token_expired when the token is expired", and the state holds the diff and the spec line. The probability comes back, and code decides what it means. How to write those statements, and why one criterion often becomes three, is in writing Noul checks for acceptance criteria.

For a quality rubric, the check is a Score: 2 to 10 ordered levels described in words, one observable behaviour per level, and the model returns a position on that scale computed as each level number multiplied by its probability, added up [3]. We had rubrics already. Rewriting them for Score meant removing every number from them, because TypeSafe says Score levels are weak in numerical calibration [4], and describing each level as something a reader could see in the diff. The method is in turning a rubric into Score questions.

Some checks were one broad question before, of the form "is this change acceptable?". TypeSafe names broad questions as an anti-pattern and says to split a judgment into atomic questions asked in parallel and combine them in code with weighted arithmetic [1]. We split those. The combination lives in a file that a reviewer can diff, which composite scoring for code quality describes.

What happens to a grade

A probability is a number, and a number only becomes a verdict when code compares it to a threshold. Three outcomes exist for every judged check.

Above the threshold, the check passes. Below the fail line, the check fails and the change is blocked, in the same way a failing unit test blocks it. Between the two lines is the uncertain band, and a grade there goes to a person. That routing follows TypeSafe's own guidance for its confidence number, which says to act automatically at 0.9 and above and to route to a human, request clarification, or fall back to a different system below 0.5 [5], with the note that different actions in the same system should be gated at different levels depending on the consequences. A Noul carries no separate confidence number, only the probability, so for a Noul the bands are set on the probability itself [2]. We set them per check by what a wrong pass would cost, and the reasoning is in thresholds, confidence and escalation.

The person who reviews an uncertain grade records a label: pass or fail, in their judgment, with the reason. That label is stored with the change, the question, the model version and the probability. It is the most useful artefact the new grader produces, because it is the set we re-check the grader against. Calibrating Jev against your own human labels is the method.

The one number

On our own suite, measured against our previous language-model grader on the same changes, the full run is ten times faster. That is the figure, and we are careful about its shape. It is a run-time comparison of one suite against its own earlier version. We publish no defect rate, cost figure, count of evals, or claim about your suite alongside it, because we have not measured those in a way we could show the working for.

Why the run time changed by that much is a matter of arithmetic on the shape of the suite. A judged check that took seconds now takes a fraction of a second: LangChain measured 0.44 seconds per Jev call against 2.16 to 2.83 seconds for the language models in its five-case test [6], and Openlayer measured p50 244 ms and p95 371 ms on its own benchmark, meaning half of requests returned within 244 ms and 95 in 100 within 371 ms [7]. All the questions for one change now go in one request, which TypeSafe's batching cookbook measured at 10.0 times faster than sending them one at a time [8]. And the checks run in parallel inside the request, so adding a question barely changes the response time [9]. The deterministic checks took the same time as before; the judged checks stopped being the slowest part. How much a given suite speeds up depends on how much of it was judged, which is why we do not generalise the figure.

What did not change

The order of work is the same. At Reveneau the evals are written from the specification before the code exists, and the coding agent writes code until the suite passes. That order is the whole point of eval-driven development, and a faster grader does not alter it. A grader that returns in 300 ms instead of 3 seconds is still grading a check that somebody wrote from a spec line before the implementation was started.

The independence rule is the same. The model that writes the code is never the model that grades it. That was a rule with a language-model judge, argued in never let the model grade its own work, and a decision model satisfies it in a stronger way: Jev cannot write code at all, so it cannot have written the change it is grading.

The blocking is the same. A change that fails the suite is not released. A judged check that fails blocks a merge in exactly the way a failing test does.

And the accountability is the same. Reveneau takes responsibility for the whole project through production and after release. A grader, of any kind, is a tool inside that responsibility.

What we gave up

A language-model judge writes a reason. When it failed a change, an engineer could read the paragraph and see, sometimes, why. A Jev grade is a probability and nothing else. When a Noul returns 0.31 on the statement that the diff handles the expired-token case, the number says the model thinks the statement is probably false and says nothing about which line it looked at. We knew this going in, and the page what a Jev grade cannot tell you is the full account.

Two things make it workable. The first is that the questions are small. When a check is "the diff returns 401 for an expired token", a low grade points at one place to look. When a check was "is this change good?", a low grade pointed nowhere, and a paragraph of reasoning was the only way to find out where. Splitting the questions moved the explanation from the grader's output into the grader's input. The second is the uncertain band. A grade near the threshold goes to a person, and the person writes the reason the model could not.

We also gave up the ability to grade anything new without writing a question for it first. A language model, handed a diff and asked what is wrong with it, will sometimes name a problem nobody anticipated. A decision model answers only what was asked. We keep a language-model pass for that purpose on the changes where the cost of a missed unknown is highest, and it is advisory, which is the same status using a model as a judge recommends for any model grade that has not yet been shown reliable enough to block.

What we checked before trusting it

Jev was announced on 15 September 2026 [10], and the model version we grade with, jev-1.13.0, was released the same day [11]. A grader that new is not trusted on its vendor's word. Before any grade was allowed to block a merge we took changes our reviewers had already labelled, ran every question repeatedly against them, and measured how often the grade agreed with the label and how much it moved between runs. The thresholds came from that measurement, and the labelled set is now the grader's own eval suite, re-run whenever the model version changes, because jev-latest is an alias that TypeSafe can point at a new release [11]. We log the model version on every request so the day that happens is visible.

We also read TypeSafe's own list of the model's weaknesses [4] before writing a single question, and several of them apply directly to a diff: the model does not count reliably, reads dates as text, cannot reliably judge whether two numbers are near each other, and can be moved by adversarial text in the state. Every check that depends on counting or comparing numbers stayed deterministic. That list, applied to grading, is on the limits page.

If you are considering the same move

Start from your inventory. List every check in your suite and mark each one deterministic or judged. Only the judged ones are candidates, and if that list is short, the run-time change will be small and a different improvement may be worth more. If it is long, the migration checklist is the sequence we followed, including the period where both graders ran on every change and only the old one decided. The comparison of the two kinds of judge, with the third-party measurements and their sample sizes, is in LLM as judge vs a decision model, and the definitions are in Jev and System One models.

A faster grader is a means. The end is the same as it was: every change proven against the specification before it is released, by a check somebody wrote before the code existed.