Faster evals with Jev: how we grade AI-written code / Start here

Start here

LLM as judge vs a decision model

A language-model judge grades by generating: it writes a paragraph of reasoning and then a verdict, and both are new text every time. A decision model grades by scoring: it takes the answers you defined in advance and returns a probability for each, in one pass, with no text. That difference explains every measured gap between them. On Openlayer's own benchmark a Ragas eval on GPT-4.1-mini took 6 calls and $2.60 per 1,000 samples against 1 call and $0.03 for Jev; in LangChain's five-case test the language models took 2.16 to 2.83 seconds per call against 0.44. Both are small samples, and this page says where each holds.

Published September 22, 2026. Editorial.

Key takeaways

A language-model judge generates a rationale and then a verdict; a decision model scores predefined answers in one forward pass. The second is faster because it does less.
Openlayer's own benchmark: 6 LLM calls plus embeddings, $2.60 per 1,000 samples, 22 to 35 seconds for 20 samples, against 1 request, $0.03, and 0.8 seconds with Jev.
LangChain's test of five cases: agreement with the human label of 100 percent for Jev against 99.8, 96.4 and 80.0 percent for three language models, with per-case variance 92 to 913 times lower.
Both third-party tests are small and their authors say so. LangChain calls its result observational, not evidence, and warns that low cost can amplify mistakes.
A language model grading its own output rates it higher than a person would. A decision model cannot have written the code it grades, which removes that bias by construction.

Ask a language model whether a diff meets its acceptance criterion and it will tell you, in prose, why it thinks so, and then say yes or no. Ask a decision model the same thing and it returns 0.87. The two answers are produced by different mechanisms, and understanding the mechanism is what lets you predict where each grader is reliable and where it is not. Reveneau grades the judged checks in its eval suite with Jev, a decision model, and this page is the comparison we made before we did.

How a language-model judge grades

A language model generates text one token at a time, and a judge built on one is a prompt that asks for a verdict. The usual shape is a rubric, the artefact to grade, and an instruction to reason first and then answer. The reasoning is there because it improves the verdict, and it is also the cost: the model has to produce every word of the paragraph before it can produce the verdict, and a long paragraph takes seconds.

The output is new text each time. Two runs on the same input can reason differently and reach different verdicts. LangChain's team measured exactly this in its September 2026 test: on identical traces the language-model judges had per-case variance on the quality score between 92 and 913 times higher than Jev's [1]. A judge that changes its mind between runs makes a poor test suite, because a failing check might pass on re-run and the team learns to re-run instead of read. The full treatment of building such a judge well, including the rubric rules and the independence requirement, is in using a model as a judge, without fooling yourself. Nothing on that page is wrong. It describes a tool with a cost.

There is a second cost that is easy to miss. A language model asked to grade its own output rates it higher than an independent person would, for the same quality of work. That is self-preference bias, and it is why the independence rule exists. How to evaluate a vendor's eval suite explains what to ask a vendor about it, and the practical fix with two language models is a fresh context or a different model. The fix is a process rule, and process rules can be broken by a config change nobody notices.

How a decision model grades

A decision model does not generate text. TypeSafe describes Jev's input as a state plus a map of named questions, and its output as a typed answer per question with probabilities [2]. For a Noul, a yes-or-no statement, the output is one probability from 0 to 1 [3]. For a Score, a position on a scale of worded levels, the output is a probability per level and a score that combines them [4]. For a Choice, one of up to 255 named options with a probability each [5]. Every question in a request is evaluated in parallel, so the response time barely changes as questions are added [6].

The verdict is the probability. There is no paragraph before it, which is where the latency goes, and no free text at all, which is where the variance goes. TypeSafe reports 70 to 500 ms end to end for Jev against 3 to 329 seconds for the frontier language models in its own comparison [7], and the third-party measurements below are inside that range.

The consequence for grading is a trade. You get speed and repeatability. You give up the rationale, and you give up the ability to notice anything you did not write a question about. What a Jev grade cannot tell you is the list of what that costs in practice.

Openlayer's benchmark

Openlayer released jevals on 20 September 2026, an MIT-licensed library that runs agent evals and guardrails as one Jev request per trace, and its README carries a comparison with Ragas, an evaluation library, running on GPT-4.1-mini. On Openlayer's own benchmark, Ragas used 6.0 language-model calls plus embeddings per sample, cost $2.60 per 1,000 samples, and took 22 to 35 seconds for 20 samples. jevals on Jev used 1.0 request per sample, cost $0.03 per 1,000, and took 0.8 seconds for 20 samples. Jev's per-request latency was p50 244 ms and p95 371 ms, meaning half of requests returned within 244 ms and 95 in 100 within 371 ms [8].

Three caveats belong next to those numbers. The benchmark was run by the people who wrote jevals. It is one comparison against one library on one language model. And the README says the library is alpha, with its framework adapters written to the SDK docs and tested against fakes rather than run live [8]. The cost gap is arithmetic on the number of calls and follows from the mechanism, so it will hold in shape; the exact figures are Openlayer's.

LangChain's test

LangChain's team asked a different question: is the cheap grade also a correct grade? Their post of 20 September 2026 took five weather-agent cases with human-labelled correct answers and ran each judge 100 times per case, 500 decisions per judge, on identical traces. On the binary question of whether the trace passed, Jev agreed with the human label 100 percent of the time; GPT-5.6 Terra 99.8 percent; GPT-5.6 Luna 96.4 percent; Claude Sonnet 4.6 80.0 percent. Jev's mean per-case variance on the quality score was 0.0000149; Luna's was 433 times higher, Terra's 913 times, Claude's 92 times. Jev took 0.44 seconds per call at $0.00035; the language models took 2.16 to 2.83 seconds. The whole run cost $0.34 with Jev and $28.17 with Claude Sonnet 4.6 [1].

The authors are careful, and we repeat their care. They call the result "promising, but early" and "observational, not evidence". They say the experiment "cannot tell us why Jev's scores varied less". And they warn that "low cost can amplify mistakes": a grader that runs a thousand times an hour and is wrong in a consistent way produces a thousand consistent wrong grades. Five cases is a small set. The 100 percent figure is five traces graded correctly 100 times each, and it says nothing about the sixth trace.

What the test does establish is the mechanism's signature. Low variance between runs is what scoring predefined answers produces, and high variance is what generating a rationale produces. That part holds at any sample size.

The vendor's numbers, labelled as such

TypeSafe's own headline is that Jev finished a workflow decision in 0.114 seconds for $0.000081 against GPT-5.6 Terra at 8.566 seconds for $0.013880, which TypeSafe reports as 193.6 times faster and 444.6 times cheaper. MarkTechPost, reporting the launch on 19 September 2026, notes that TypeSafe's own capabilities team wrote the workflows and that TypeSafe says it cannot prove the price is unsubsidised [9]. That is a vendor's benchmark on the vendor's tasks. It agrees in direction with the third-party tests and is larger in size, which is the usual shape of a vendor benchmark, and it should be quoted with the attribution attached.

Which grader for which check

The mechanism gives a rule, and we apply it check by check rather than suite-wide.

A check goes to the decision model when the question can be written in advance, when the answer is one of a fixed set, and when a probability is enough. "The diff adds a 401 response for an expired token" is that shape. "The error message tells the user what to do next, on a three-level scale" is that shape. Most acceptance criteria and most rubric scores are that shape, and how to write them is in writing Noul checks for acceptance criteria and turning a rubric into Score questions.

A check stays with a language model when you need a written reason, when you want the grader to notice something you did not ask about, or when the judgment needs several steps of reasoning that no fixed question captures. "What is wrong with this change?" is that shape. So is anything involving counting, date arithmetic, or comparing two numbers, which TypeSafe's own jaggedness page lists as things Jev does badly [10]. Those checks should be deterministic where possible and go to a language model or a person where not.

A check goes to a person when the consequence of a wrong pass is high and the grade is uncertain, whichever model produced it. Thresholds, confidence and escalation covers the bands.

What we did with this

Reveneau moved every judged check that fit the first shape to Jev, kept an advisory language-model pass for the second shape on the changes where a missed unknown costs most, and routes the uncertain band to a person. On our own suite the run is ten times faster than with the language-model grader, and that figure is about our suite only. The account is in why we moved our eval grader to Jev. Before we trusted a single grade to block a merge, we measured agreement on our own labelled changes, which is the step calibrating Jev against your own human labels describes, and it is the step that turns other people's small samples into a number about your suite.

Two graders, two mechanisms. Pick by the shape of the question, and measure before you trust either.

Best for

Checks whose question can be written in advance and whose answer is one of a fixed set
Suites where the judged checks are the slowest step and run on every change
Teams that will label a calibration set from their own changes before trusting a grade

Avoid if

You need a written reason with every failing grade and have no person to supply it
The check depends on counting, date arithmetic, or comparing two numbers
You want the grader to notice failures nobody has written a question for

Check before you decide

Measure agreement with your own reviewers on 50 to 200 labelled changes, run repeatedly
Quote the third-party figures with their sample sizes and authors attached
Confirm the writer of the code is never the only grader, whichever model grades

Common questions

What is the mechanical difference between an LLM judge and a decision model?

A language-model judge generates text: it writes a rationale token by token and then a verdict, and both are new every run. A decision model scores answers you defined in advance and returns a probability for each in one pass, with no text. The first takes seconds and varies between runs; the second returns in a fraction of a second and returns the same number for the same input.

What did Openlayer's benchmark measure?

On Openlayer's own comparison, published in the jevals README, Ragas on GPT-4.1-mini used 6 language-model calls plus embeddings per sample, cost $2.60 per 1,000 samples, and took 22 to 35 seconds for 20 samples. jevals on Jev used 1 request per sample, cost $0.03 per 1,000, and took 0.8 seconds for 20. Jev's latency was p50 244 ms and p95 371 ms. It is one comparison, run by the library's authors.

What did LangChain's test find?

On five weather-agent cases with human labels, run 100 times per judge per case, Jev agreed with the label 100 percent of the time against 99.8, 96.4 and 80.0 percent for GPT-5.6 Terra, GPT-5.6 Luna and Claude Sonnet 4.6. Jev's per-case variance was 92 to 913 times lower, its calls took 0.44 seconds against 2.16 to 2.83, and the run cost $0.34 against $28.17. The authors call it observational, not evidence.

Why do the LangChain authors warn that low cost can amplify mistakes?

Because a grader that is cheap enough to run on every change runs thousands of times, and if it is wrong in a consistent way it produces thousands of consistent wrong grades with nothing in the pipeline to catch them. A slow, expensive judge gets fewer chances to be wrong. The defence is a labelled calibration set from your own changes, re-run on every model version, so a consistent error shows up as a drop in agreement.

Is 100 percent agreement on five cases meaningful?

It is five traces graded correctly 100 times each, which shows the grader was stable and right on those five and says nothing about a sixth. What transfers is the mechanism: scoring fixed answers produces low variance, and generating a rationale produces high variance. Treat the agreement figure as a reason to run your own calibration rather than as a result you can quote about your suite.

What is self-preference bias and does a decision model have it?

Self-preference bias is a language model rating its own output higher than an independent person would, for the same quality of work. It is the reason the writer of the code must never be the only grader. A decision model cannot generate text, so it cannot have written the diff it grades, which removes that bias by construction. The rubric still has to come from the spec before the code, which is the other half of the rule.

When should a check stay with a language-model judge?

When you need a written reason with every failing grade, when you want the grader to notice a failure nobody wrote a question for, or when the judgment needs several reasoning steps that no fixed question captures. Also when the check involves counting, date arithmetic, or comparing two numbers, which TypeSafe lists as weaknesses of Jev; those should be deterministic where possible and go to a language model or a person where not.

How should a vendor's own benchmark be read?

As the vendor's claim about the vendor's tasks. TypeSafe reports 193.6 times faster and 444.6 times cheaper than GPT-5.6 Terra on a workflow decision, and MarkTechPost notes that TypeSafe's own team wrote the workflows and that TypeSafe cannot prove the price is unsubsidised. The direction agrees with the third-party tests; the size is larger, which is the usual shape. Quote it with the attribution attached.

Does a decision model make an eval suite more repeatable?

For the judged checks, yes, and that is measurable. TypeSafe's batching cookbook reports a standard deviation of 0.0 across repeats for 11 of 13 questions, and LangChain measured per-case variance 92 to 913 times lower than the language models. A check that returns the same number for the same input can be re-run without the team learning to re-run failures until they pass.

How did Reveneau apply this comparison?

Every judged check whose question could be written in advance with a fixed answer set moved to Jev. Checks that need a written reason or an open-ended search for problems stayed with an advisory language-model pass on the highest-consequence changes. Uncertain grades go to a person. On our own suite the run is ten times faster than with the language-model grader, and we measured agreement on our own labelled changes before any grade could block a merge.

References

More in Start here

Why we moved our eval grader to Jev

Reveneau writes all of its code with AI and releases nothing that has not passed an eval suite written from the specification. Inside that suite, the checks with no exact answer used to go to a language model that read the change and wrote a verdict. This month those checks go to Jev, a decision model that returns a probability for each question we wrote in advance. On our own suite, measured against our previous grader, the run is ten times faster. This page describes the practice: which checks moved, which did not, what a grade looks like, and what happens when the grade is unsure.

Previous: Why we moved our eval grader to Jev Next: Turning a rubric into Score questions