Faster evals with Jev: how we grade AI-written code

Every eval suite for AI-written code has a slow step, and until this month ours was the grader. A test either passes or fails, and that takes milliseconds. Whether a diff meets the acceptance criterion it claims to meet has no exact answer, so for those checks we asked a language model to read the change, write a paragraph of reasoning, and end with a verdict. Each of those calls took seconds, the paragraph changed between runs, and the verdict sometimes changed with it. We now send those checks to Jev, a decision model from TypeSafe that answers fixed questions with probabilities instead of writing text, and on our own suite the full run finishes ten times faster than it did with the language-model grader. This guide is the method behind that change, what it cost us to make, and what it cannot do.

Two things did not change and should be said first. Every eval at Reveneau is still written from the specification before the code exists, and a change that fails the suite is still not released. What changed is the mechanism inside one kind of check.

Two kinds of check in every suite

The first kind has an exact answer. A unit test passes or it does not. A schema validates or it does not. A migration applied to a copy of production data keeps every rule that must always hold, or it does not. These checks are deterministic: run them twice on the same input and you get the same result. They stay in code and never go to any model, because a model can only lower their reliability. The list of what belongs in this category is in what to check in an eval suite, and the pipeline shape is in evals in CI for coding agents.

The second kind has a better answer and a worse answer with no single string to compare against. Does this diff implement the requirement in the linked spec line? Did the change stay inside the files and behaviours the task named? Did the agent's trace follow the plan it was given, or did it call a tool the plan never mentioned? Does the generated error message tell the user what to do next? For these, the older method is a language model as judge: give it the artefact and a rubric, ask for a verdict. We wrote up that method, its independence rule, and how to validate it in using a model as a judge, without fooling yourself. It works. It is also the reason that page warns about cost and latency, because a judge that adds minutes to every pull request gets avoided.

This hub is about the second kind of check only.

What a decision model is, in one paragraph

TypeSafe announced Jev on 15 September 2026 as the first public System One model, which it describes as a new class of model built to make fast, structured decisions that software can use directly [1]. It does not generate text. The input is a state (a string, a JSON object, or an array of text values, text only) plus a map of named questions, and the output is a typed answer per question with a probability attached [2]. There are three question types. A Choice picks one of up to 255 named options. A Score places the state on an ordered scale of 2 to 10 levels that you describe in words. A Noul is a yes-or-no statement that returns a single probability from 0 to 1 [3][4]. Every question in a request is evaluated in parallel, so adding questions barely changes the response time [5]. Definitions, the training method, and the full list of what the model is and is not are in the companion guide Jev and System One models; this page uses them without repeating them.

The consequence for an eval suite is simple to state. A language model judge produces a paragraph and then a verdict, and both are new text each time. A decision model produces a probability for a question you wrote in advance, and nothing else. The grade is the number. The reasoning, if you want it, has to come from somewhere other than the grader.

What a language-model judge costs, measured by other people

Two third parties have published side-by-side measurements, and both are worth reading with their sample sizes stated.

Openlayer released jevals on 20 September 2026, an MIT-licensed library that runs agent evals and guardrails as one Jev request per trace. On Openlayer's own benchmark, Ragas on GPT-4.1-mini used 6 language-model calls plus embeddings per sample, cost $2.60 per 1,000 samples, and took 22 to 35 seconds for 20 samples; jevals on Jev used 1 request per sample, cost $0.03 per 1,000, and took 0.8 seconds for the same 20 samples. Openlayer measured Jev at p50 244 ms and p95 371 ms per request, meaning half of requests returned within 244 ms and 95 percent within 371 ms [6]. That is one comparison, run by the people who wrote the library.

LangChain's team ran a different test, published 20 September 2026 under the title "Can Jev be a better agent evaluator?". They took five weather-agent cases with human-labelled correct answers and ran each judge 100 times per case, 500 decisions per judge. On the yes-or-no question of whether the trace passed, Jev agreed with the human label 100 percent of the time, GPT-5.6 Terra 99.8 percent, GPT-5.6 Luna 96.4 percent, and Claude Sonnet 4.6 80.0 percent. Jev's mean per-case variance on the quality score was 0.0000149, with the language models between 92 and 913 times higher. Jev took 0.44 seconds per call at $0.00035; the language models took 2.16 to 2.83 seconds per call. The whole run cost $0.34 with Jev and $28.17 with Claude Sonnet 4.6 [7]. The authors call the result "promising, but early" and "observational, not evidence", say the experiment cannot tell them why Jev's scores varied less, and warn that low cost can amplify mistakes. Five cases is a small set, and we repeat that every time we quote the number.

The vendor's own headline is larger. TypeSafe reports a workflow decision finishing in 0.114 seconds for $0.000081 against GPT-5.6 Terra at 8.566 seconds for $0.013880, which it presents as 193.6 times faster and 444.6 times cheaper; TypeSafe's own capabilities team wrote the workflows, and TypeSafe says it cannot prove the price is unsubsidised [8]. Treat that as the vendor's claim about the vendor's tasks.

Our own figure is narrower than any of these and we keep it that way on purpose: on our own suite, measured against our previous grader, the run is ten times faster. We are not publishing a dollar figure, a defect rate, or a count of evals, because we have not measured them in a way we could show the working for. The full account of the switch is in why we moved our eval grader to Jev, and the mechanics of the two kinds of judge are compared in LLM as judge vs a decision model.

Designing the questions

A decision model answers the question you wrote, and TypeSafe's own list of known weaknesses opens with exactly that: it "answers the question you wrote, not the one you meant", and scoping words, negations, and implied conditions are read exactly as written [9]. So the design work moves from writing a judge prompt to writing questions, and three shapes cover most of an eval suite.

A rubric becomes a Score. You write 2 to 10 ordered levels in words, one observable behaviour per level, and the model returns a position on that scale as each level number multiplied by its probability and added up, which is why a score can land between levels: the docs' own example is a score of 1.43 with probabilities of 0.57 on level 1 and 0.43 on level 2 [3]. Numeric rubrics do not work here, because TypeSafe says Score levels are weak in numerical calibration [9]. The method, with a worked rubric for a code change, is in turning a rubric into Score questions.

An acceptance criterion becomes a Noul. One criterion, one yes-or-no statement, read literally, with the evidence placed in the state: the diff, the spec line, the trace. The worked example of one spec line becoming three statements is in writing Noul checks for acceptance criteria.

A broad judgment becomes several small ones combined in code. TypeSafe's design guidance names broad questions ("Is this spam?") as an anti-pattern and tells you to break a judgment into atomic questions asked in parallel, then combine them in code with weighted arithmetic [10]. "Is this change good?" is the broad question every eval suite is tempted to ask. Composite scoring for code quality shows the split, example weights, and why the arithmetic belongs in a file a reviewer can diff.

The same guidance has a sentence we treat as the rule for the whole suite: keep control flow, deterministic rules, and side effects in code [10]. The model answers questions. Code decides what happens next.

Thresholds, and what happens in the middle

A probability is not a verdict until code compares it to a number. TypeSafe's confidence page gives the bands it recommends: 0.9 and above, act automatically; below 0.5, route to a human, request clarification, or fall back to a different system; the middle, proceed with caution. The same page says different actions within the same system should be gated at different levels depending on the consequences [11]. In an eval suite that means a check on a flow that moves money fails at a higher grade than a check on a log message, and the uncertain band goes to a person.

That person's label is the most valuable output of the whole process, and we keep every one. It tells you, months later, whether the grader was right in the cases it was unsure about, and it becomes the set you re-check the grader against. Thresholds, confidence and escalation covers the bands, the per-check thresholds, and the storage. Noul answers carry no separate confidence number, only the probability, so the band for a Noul is set on the probability itself [4].

Running it on every pull request

The pipeline shape is the one described in evals in CI for coding agents, with one request per change added after the deterministic checks pass. All the questions for one change go in one request, because the docs say the response time barely changes with the number of questions and TypeSafe's own batching cookbook, on jev-1.12, measured 13 questions in one call at $0.000497 and 0.27 seconds against thirteen separate calls at $0.006090 and 2.71 seconds [12]. The published limits are 1,200 requests per minute and 250,000 tokens per second, with 64k tokens of context per request and 32k for the state plus the longest question [13], so a large diff has to be split or trimmed before it is sent. The price is $0.042 per million input tokens with output tokens free [13], which makes the cost of a grade small enough that the arithmetic fits in a sentence. All of that, plus what to log and what a failing grade blocks, is in running Jev-graded evals on every pull request.

Grading an agent trace

When the thing under test is an agent, the artefact is a trace: the sequence of model calls, tool calls, and results. The questions change shape. Did the agent choose the right tool? Did it stay in scope? Did it loop? Did it complete the goal? Openlayer's jevals ships 37 evals across agent, security and quality groups with adapters for the OpenAI Agents SDK, LangGraph and the Claude Agent SDK, and its README says it is alpha [6]. LangChain's test is the other public reference point. Both, and the shape of a trace check, are in grading agent traces with Jev. The broader question of whether an agent is ready for production at all is the subject of AI agents in production, and the runtime side of the same model, guardrails and routing, is in Jev in production.

What a grade cannot tell you

A Jev grade is a probability with no written rationale. When a check fails you learn that the model put a low probability on the statement, and nothing about why. TypeSafe's jaggedness page lists the weaknesses that matter most to a grader: it does not count reliably, it cannot judge whether two numeric values are near each other, it reads dates as text rather than as ordered quantities, double negatives and indirection are answered less reliably, accuracy falls as the state grows with unrelated content, and adversarial content in the state can move the answer [9]. A diff can contain all six at once. TypeSafe's phrase "can't hallucinate" means the answer always matches the schema; the answer can still be wrong.

There are also checks that should stay with a language model or with a person: a new failure mode nobody has written a question for, and design judgment, which is what human review is for and what evals vs tests vs code review explains. The full list is in what a Jev grade cannot tell you.

Calibrating against your own labels

The published tests are other people's tasks. Before a grade is allowed to block a merge on yours, you need agreement measured on your own cases: 50 to 200 labelled examples from your own suite, each run 10 times, agreement and variance measured the way LangChain measured them, and thresholds set from the result. Repeat it on every model version change, because jev-latest is an alias that can move to a new version without your pipeline changing [13]. Keep the labelled set as the grader's own eval. The method is in calibrating Jev against your own human labels.

Migrating an existing suite

If you already run a language-model judge, the switch is a sequence, and the migration checklist is printable: inventory every check, classify deterministic against judged, rewrite each judged check as questions, label a calibration set, run both graders on the same changes with only the old one deciding, compare, set thresholds, switch, and keep the human band. A team that has no suite at all should start with how to write your first eval suite and adding evals to a codebase that has none before any of this applies.

Where a decision model sits next to the coding agent

One boundary is worth stating because it is easy to get wrong. Jev does not write code and does not run the coding agent. TypeSafe's own page on coding agents says it is not a drop-in replacement for the language model that runs the agent; it is used inside the application for routing, rubric scoring, and statement verification [14]. In an eval suite it is the grader. The agent that writes the code is a different system, and the rule that the writer must never be the only grader, argued in never let the model grade its own work, is one of the reasons a separate decision model is attractive: it cannot have written the code it is grading. Product teams weighing the same model for decisions outside engineering will find that treatment in decision models for product teams.

Where Reveneau fits

Reveneau writes all of its code with AI, and a large eval suite, written from the specification, has to pass before any change is released. We grade that suite's judged checks with Jev. Each such check is a Noul or a Score question with the rubric written into the criteria; a grade below the threshold fails the change; a grade in the uncertain band goes to a person, and that person's label is kept so we can re-check the grader later. On our own suite the run is ten times faster than it was with a language-model grader, and that is the only number we attach to the switch.

The saving goes where our other savings go. We use AI instead of hiring more engineers, so the same build takes a small team, and a faster suite means the team waits less between a change and its verdict. If you are choosing a development partner, the questions to ask about any vendor's grader are in how to evaluate a vendor's eval suite, and they apply to us. Our AI development work is where this practice lives, and if you want to see how a judged check is written and graded on a real change, talk to us.