Which checks in an eval suite should go to Jev and which should stay deterministic?

Any check with one right answer a program can compute stays deterministic: a test passing, a schema validating, a migration applying and rolling back. Jev takes only the checks that used to need a language model as judge, such as whether a change matches a written acceptance criterion or whether a diff stayed inside its scope. Sending a deterministic check to any model adds cost and a chance of error for no gain.

How do you write an eval check as a Jev question?

Each judged check becomes either a Noul, a yes/no statement that returns one probability from 0 to 1, or a Score, an ordered set of 2 to 10 levels described in words. The acceptance criterion and the standard for each level go into the question itself, because Jev returns no explanation, so the rubric is the only place the reasoning can live. In practice this means a person can read the check and know what it grades without running it.

What happens when a Jev grade falls below the threshold?

The change fails, exactly as it would on a failed unit test, and nobody can override it to release anyway. Below the threshold there is no review step, because the point of the suite is that a failed check stops the release without a person deciding whether to skip it that day. The threshold itself is set per check from labelled examples, so a scope question and a money-moving criterion carry different cutoffs.

What is the uncertain band and why does a person handle it?

The uncertain band is the range of probabilities between a clear fail and a clear pass, where the grader's answer is too spread out to act on. A person reads that change and labels it, and we keep the label. TypeSafe's own guidance is that a confidence below 0.5 should go to a human or a different system, and that different actions deserve different cutoffs depending on their consequences.

Why keep the human label after the change has been decided?

A grader that gives no written reason can only be checked by comparing its answers with answers people already gave. Every kept label becomes one more row in the set used to re-check the grader after a model version change or a rubric edit. Without that set, you would notice a drift in the grader only when a bad change reached production.

Did moving to Jev change how the evals are written?

No. Every eval is still written from the specification before the code exists, and that order is what makes a check test the code against what was asked for rather than against itself. A faster grader applied to a check written by reading the finished code is a faster way to confirm the code agrees with itself. The grader is the only part that changed.

How much faster is an eval suite with Jev as the grader?

On our own suite, measured against the language model that graded it before, the run is ten times faster. That is one suite and one comparison, so treat it as our statement about our own setup rather than a general rule. TypeSafe reports 70 to 500 milliseconds end to end per request, and LangChain measured 0.44 seconds per call in its own test against 2.16 to 2.83 seconds for the language models it compared.

What external evidence exists that Jev can grade as well as a language model?

Two published tests, both small and both from parties that built on Jev. LangChain ran five weather-agent cases 100 times per judge and found Jev matched the human pass/fail label 100 percent of the time, against 99.8 percent for GPT-5.6 Terra and 80.0 percent for Claude Sonnet 4.6, and the authors called the result observational rather than evidence. Openlayer's jevals benchmark reports cost and speed, and the accuracy figure its README cites comes from intent-classification benchmarks, which measure a different task from grading code.

What should I check before switching my own grader to Jev?

Take the judged checks your team has already labelled by hand, run the same checks through Jev, and compare the two sets of answers before any Jev grade is allowed to fail a change. Set each threshold from the disagreements you find, and keep a language model or a person for any check the comparison shows Jev gets wrong. Do this on your own checks, because the published tests used five agent cases and a vendor's own workflows.

Can Jev replace the model that writes the code?

No, and TypeSafe says so on its own coding-agents page: Jev cannot simply replace the language model that runs a coding agent, and it is not trained to generate text. It sits beside the writing model as the grader, which is also why it cannot share the writing model's habits. A grader with different training from the writer is the first thing that protects against a model rating its own work as correct.

We Moved Our Eval Grader to Jev: What Changed and What to Check First: Reveneau

This month we replaced the grader in our eval suite. For a year the checks that needed judgment, the ones where a program cannot compute the answer on its own, went to a language model with a prompt and came back as a paragraph and a verdict. They now go to Jev, the decision model TypeSafe AI announced on 15 September 2026. On our own suite, measured against the previous grader, the run is ten times faster.

An eval suite, for anyone new to the term, is the set of automated checks that every change has to pass before it is released, written from the specification before the code. Ours is how a small team releases AI-written code without a person reading every line. The grader is the part of the suite that decides whether a judged check passed. Here is what changed when we swapped it, what we left alone on purpose, and what you should check before doing the same.

1. The deterministic checks did not move

Most of a suite has one right answer that a program can compute. A unit test, a small automated check of one piece of code, passes or it does not. A schema validates, meaning the data has the shape the code expects. A database migration applies and can be reversed. Those checks never went to a language model, and they do not go to Jev now. Sending a check with a computable answer to any model adds cost and a chance of error for no gain.

TypeSafe's own design guidance says the same thing in its own words: keep control flow, deterministic rules, and side effects in code, and send the model only the judgment. We read that page after we had already made the same split, and it is the first thing we would tell anyone planning the switch. If a check can be a test, make it a test.

So the change touched one category of check only: the ones where a person used to have to read and decide. Whether a change matches a written acceptance criterion. Whether a diff, the list of lines a change added or removed, stayed inside the scope of the task. Whether an agent's trace, the record of the steps it took, followed the plan it was given. Whether generated text follows the instruction it was written to. Those four kinds are what the language model graded, and those four kinds are what Jev grades now.

2. Every judged check became a question with the rubric inside it

Jev does not write text. You send it a state, which is a string or a structured object holding whatever the decision needs (for us, the diff, the criterion, and sometimes the trace), plus a map of named questions, and it returns a typed answer for each question with probabilities. There are three question types. We use two.

A Noul is a yes/no statement and returns one probability from 0 to 1. "This diff changes only files under the payments module" is a Noul. A Score is a position on an ordered set of 2 to 10 levels that you describe in words, and the score is each level number multiplied by its probability, added up, so it can land between levels. The docs' own example returns 1.43 with probabilities of 0.57 on level 1 and 0.43 on level 2. "How completely does this change satisfy criterion 4" is a Score, with each level described in words: 0 means the criterion is not addressed, 1 means addressed with a gap a reviewer would send back, 2 means satisfied as written.

That last sentence is the whole change. With a language model as judge, the prompt carried the criterion and some instructions, and the paragraph that came back carried the reasoning. With Jev there is no paragraph, so the reasoning has to live in the question and in the level descriptions before the check runs. Rewriting every judged check this way took longer than connecting the suite to Jev. It was also the most useful part of the migration, because a check that a person can read and understand without running it is a better check, whichever model grades it. Several of ours turned out to be vaguer than we thought once there was no paragraph to supply the reasoning the question had left out.

One request carries every question about one change. Jev evaluates the questions in a request in parallel, and TypeSafe's docs say adding questions barely changes the response time. Their batching cookbook gives a measurement, on their own workload: 13 questions against the 53,777-character Wikipedia article on the GDPR, five repeats each way, cost $0.000497 and took 0.27 seconds as one batched call, against $0.006090 and 2.71 seconds as thirteen single calls. TypeSafe reports that as 12.2 times cheaper and 10.0 times faster. That is the vendor measuring the vendor's example, so treat the ratio as what batching did there and measure your own. On our suite, sending every question about one change in one request is part of where the ten times came from.

3. The threshold fails the change, and the uncertain band goes to a person

Every judged check has a threshold. A grade below it fails the change, the same way a failed test does, and nobody can override the failure to release anyway. That rule carried over from the previous grader exactly as it was.

The addition is the band above the fail line and below the clear pass, where the probabilities are spread out enough that acting on them would be a guess. A change that lands there goes to a person, who reads it and labels it pass or fail. We keep that label. TypeSafe's confidence guidance gives the same guidance: 0.9 and above, act automatically; below 0.5, route to a human or fall back to a different system; and, in their words, different actions within the same system should be gated at different levels depending on the consequences. A gate, here, is a check that blocks a change from being released. A scope question on a wording change and a criterion on the code that moves money do not deserve the same cutoff, so we set the band per check.

The kept label is what makes the whole thing auditable. A grader that returns probabilities and no sentence cannot explain a single decision, so the only way to check it is to compare its answers against answers people already gave. Every label a person writes in the uncertain band is one more row in that comparison set. When TypeSafe releases a new Jev version, or when we edit a rubric, we rerun the labelled set and look at where the grader and the people disagree, before the new version is allowed to fail anything. The current model is jev-1.13.0, and we name that exact version in our configuration rather than jev-latest, the alias that always points at the newest release, for exactly this reason.

4. What we did not change: the evals are still written from the spec, before the code

A faster grader does not fix a check written from the wrong source. Every eval in our suite is still written from the specification before the code exists, in the order our guide to eval-driven development describes. A check written by reading the finished code tests whether the code agrees with itself, and grading that check ten times faster gives you the same non-answer ten times sooner.

The other thing we kept is the separation between the model that writes and the system that grades. We wrote earlier this year about why you should never let the model grade its own work, and Jev happens to make that separation stronger. TypeSafe says it is trained with a method it calls Reinforcement Learning for Calibrated Decisions, and has not disclosed the architecture. Whatever it is, it is a different system with different training from the model that produced the diff, which is what a grader should be. TypeSafe's own coding-agents page says Jev cannot simply replace the language model that runs an agent. It sits beside that model, inside the application, for routing, rubric scoring and statement verification. That is the role we gave it.

Before you do the same: calibrate against your own labels first

The external evidence that Jev can grade is real, small, and written by people who built on it. Read it as that.

LangChain's test, published on 20 September 2026, ran five weather-agent cases 100 times per judge, 500 decisions per judge in total, against a human-labelled answer key. On the pass/fail decision, Jev agreed with the human label 100 percent of the time, GPT-5.6 Terra 99.8 percent, GPT-5.6 Luna 96.4 percent, and Claude Sonnet 4.6 80.0 percent. Jev took 0.44 seconds per call at $0.00035, against 2.16 to 2.83 seconds for the language models, and the full run cost $0.34 with Jev against $28.17 with Claude Sonnet 4.6. The authors call the result "promising, but early" and "observational, not evidence", say the experiment cannot tell them why Jev's scores varied less, and warn that low cost can amplify mistakes. Five cases is a small set, and they say so.

Openlayer's jevals, an early release under the MIT open-source licence published the same day, reports its own benchmark: Ragas, an evaluation library, running on GPT-4.1-mini at $2.60 per 1,000 samples and 22 to 35 seconds for 20 samples, against jevals on Jev at $0.03 per 1,000 and 0.8 seconds for 20. Those are cost and speed figures from the party that built the tool. The accuracy figure the README cites, 83 to 87 percent on Banking77 and CLINC150, comes from intent-classification benchmarks, which test whether a model can sort short customer messages into the right category, a different task from grading a code change against a criterion.

The published numbers tell you Jev is fast and cheap, and that it agreed with humans on five agent cases and one vendor's workflows. Whether it agrees with your team on your checks is a separate question, and the way to answer it is the same way we did: take the judged checks your people have already labelled by hand, run them through Jev with the rubric written into the question, and compare the two sets of answers before any Jev grade is allowed to fail a change. Set each threshold from the disagreements. Keep a language model or a person for any check the comparison shows Jev getting wrong, and read TypeSafe's own list of known weaknesses before you decide which checks those are: it reads questions literally, it does not count reliably, it reads dates as text, and text written to manipulate the model, if it appears in the state, can move the answer. Our guide to evals with Jev explains the calibration step in detail.

Credit to TypeSafe for publishing its weaknesses next to its benchmarks, and to the LangChain and Openlayer teams for publishing their numbers with their caveats attached. That habit made our own check faster to run.

A faster grader is worth having only once you know what a right grade looks like.

Sources

Introducing System One models and Jev: TypeSafe AI's announcement of 15 September 2026, with its latency claim and training method.
System One concepts, Noul, Score, Confidence, Introduction, Models: TypeSafe's documentation for the state and question model, the two question types used here, the confidence bands, parallel evaluation, and the current version.
How to build with System One and Coding agents: TypeSafe's guidance on keeping deterministic rules in code and on Jev's role beside a coding agent.
Parallel questions cookbook: TypeSafe's batching measurement on the GDPR article, jev-1.12.
Model jaggedness, jev-1.13: TypeSafe's published list of known weaknesses.
Can Jev be a better agent evaluator?: LangChain, 20 September 2026, five cases, 500 decisions per judge, with the authors' caveats.
jevals: Openlayer's MIT-licensed alpha and its own cost and speed benchmark.
TypeSafe AI releases Jev: MarkTechPost, 19 September 2026, on the undisclosed architecture.

Our eval suite now runs ten times faster. Here is what we changed.