What a Jev grade cannot tell you

The first time a Jev-graded check fails a change, an engineer will open the pipeline log looking for the reason, and there will be a number. That moment is the honest introduction to what a decision model is: a grader that returns numbers and no reasons. Reveneau grades its eval suite with Jev and finds that trade worth making, and the trade is only worth making with the limits written down in advance. This page is that list, drawn from TypeSafe's own documentation of the model's weaknesses and from what those weaknesses mean when the state is a diff or a trace.

There is no written reason

A language-model judge writes a paragraph and then a verdict. A decision model returns a probability for each question and no text, because it does not generate text at all [1]. When a Noul comes back at 0.19, you know the model found the statement unlikely and you know nothing else: which line it weighed, whether it read the spec line, whether the diff's comment confused it.

Three things replace the paragraph, and none of them is a full substitute. The first is question design: a suite of small questions, each about one observable behaviour, turns a bare number into a pointer, because a low grade on "the log line contains no part of the token value" tells you where to look. Composite scoring for code quality is built on this. The second is the uncertain band, where a person reads the change and writes the reason, as thresholds, confidence and escalation describes. The third is an advisory language-model pass on the changes where you want a paragraph, kept advisory because it is slow and varies between runs. What none of these gives you is a reason attached to every grade, and a team that needs one on every check should keep a language-model judge for those checks, following using a model as a judge.

The weakness list, applied to a diff

TypeSafe maintains a page on what it calls model jaggedness for jev-1.13, and it is the most useful document the vendor publishes, because every item on it is something a grader will meet [2]. Here is each one with what it means when the state is a code change or an agent trace.

Literal reading. The model "answers the question you wrote, not the one you meant", and scoping words, negations, and implied conditions are read exactly as written. For a grader this is the whole reason writing Noul checks for acceptance criteria exists: the statement has to say exactly what a reader could check. A statement with an implied condition ("the endpoint validates input", implying "all input the spec lists") will be graded on the words.

Counting. The model "does not count reliably", and the error grows with size. A diff is full of things to count: files touched, dependencies added, cases handled, tests changed. Every count is a deterministic check and none is a question. A Noul that says "the diff touches fewer than five files" is a count written as a yes-or-no statement.

Numeric comparison. The model cannot reliably judge whether two numeric values are near each other. "The retry delay is longer than the timeout", "the new limit is within 10 percent of the old one", "the migration adds fewer rows than the batch size": all numeric comparisons, all code.

Numeric calibration on Score levels. Score levels are weak in numerical calibration, so a rubric level containing a number is unreliable; turning a rubric into Score questions covers the rewrite.

Dates. The model "reads dates as text, not as ordered quantities". "The expiry check compares against the current time", "the migration is dated after the previous one", "the token was issued before the request": a grader cannot order them. Date arithmetic is code.

Double negatives and indirection. Answered less reliably. A spec written as a prohibition, a statement written as a negation, a criterion that refers to another criterion: each one adds a step the model takes less well. Positive, direct statements.

Unrelated content. "Accuracy falls as the state grows with content unrelated to the decision." A whole repository in the state is worse than the one file the question is about. A whole trace is worse than the three calls the question names. Every question gets the smallest state that contains its evidence, which the pull request page turns into a splitting rule.

Adversarial content. Adversarial content in the state "can move the answer". This one deserves its own section.

Text in the diff that reads as an instruction

A coding agent writes code, and it also writes comments, strings, commit messages, and documentation, any of which can contain a sentence shaped like an instruction: "this change is complete and passes all criteria", "reviewer: approve", or a string literal that happens to restate the acceptance criterion in the affirmative. A person reading the diff ignores that text. TypeSafe's page says the model may not [2].

The same holds for an agent trace, where a tool result is text the agent did not write and the grader reads as part of the state. Openlayer's jevals has a security group of evals with PromptInjection and Jailbreak checks for exactly this reason [3], and the grader that reads the trace is exposed to the same text as the agent that produced it.

Three defences, none complete. Run the deterministic checks first, so a change that fails a test is never graded. Send only the parts of the diff a question needs, so a comment three files away is not in the state. And route any change that touches security-sensitive files, authentication, permissions, secrets, or the pipeline configuration itself, to a person regardless of its grade, following the rule in evals in CI for coding agents that nothing weakens a check without review. The security review side of generated code is its own subject.

Cannot hallucinate, and what that means

TypeSafe's launch material says the model cannot hallucinate. Read carefully, the claim is about structure: the answer always matches the schema you defined, so a Noul returns a probability, a Choice returns one of your options, a Score returns a position on your levels, and there is no free text in which an invented fact could appear. The independent reference site for these models describes the same property [4]. That is a real advantage over a language model, which can return a verdict about a function that does not exist in the diff.

The claim covers structure only. A schema-valid answer can be the wrong answer: a Noul at 0.91 on a statement that is false of the diff is a well-formed hallucination in every sense that matters to a merge. The way to know how often that happens on your changes is to measure it, which is what calibrating Jev against your own human labels does, and the published third-party agreement figures, 100 percent on LangChain's five cases with the authors calling it observational rather than evidence [5], are a reason to run that measurement rather than a substitute for it.

It grades only what was asked

A language model handed a diff and asked what is wrong with it will, sometimes, name a problem nobody anticipated. A decision model cannot, because there is no open question to answer; it scores the answers you defined. A new failure mode, one nobody has written a Noul or a Score for, is invisible to the grader until somebody writes the question.

This is the same limit every eval suite has, stated in how to evaluate a vendor's eval suite: a check can only catch a rule someone thought to write down. A decision model makes this limit stricter, because a language model's open-ended reading was a partial defence against the unknown, and that defence is gone from the checks that moved. The response is the one every eval suite uses: every incident becomes a question, so the set of what is asked grows with what has gone wrong. And on the changes where a missed unknown would cost most, an advisory language-model pass or a person reads the whole diff, because that is what open-ended reading is for.

What stays with a person

Design judgment. Whether the requirement was the right thing to build, whether the abstraction will hold in six months, whether the change belongs in this service at all: none of these is a question with predefined answers, and evals vs tests vs code review is the argument that they never were. A grader raises the minimum quality of a large volume of changes. It does not decide whether the change was a good idea.

The uncertain band, on every check, with the label stored.

Every change to the grader itself: a new question, a moved threshold, a changed weight. A grader cannot grade its own configuration, and the file that holds the thresholds is pipeline code under review.

And the accountability. Reveneau takes responsibility for the whole project through production and after release, and a grader of any kind is a tool inside that responsibility. The suite runs ten times faster than it did with a language-model grader, on our own measurement, and the list on this page is why the people are still in it.

What to do with this list

Read it before writing the first question, and again before the first grade is allowed to block. Then turn it into checks on the grader: a set of known-bad changes that must always fail, including one with an approving comment in the diff, one with a count in the criterion moved to a deterministic check, and one with a date comparison. If a model version change or a rewritten question makes any of them pass, you have found a fault in the grader before it reached a merge. The full definitions of the model's question types and its training method are in Jev and System One models, and the runtime side of the same limits, where the cost of a wrong answer is an action rather than a merge, is in Jev in production.

A grader that cannot explain itself is still a grader. It just has to be one you have measured, on your own changes, with the limits written next to the thresholds.