What to do with a grader that gives no reason

The first engineering lead we showed a Jev grade to asked where the reason was. Their suite, like ours until this month, used a language model as the judge for every check that needed judgment, and every verdict came back with a paragraph. The Jev grade for a scope check on an AI-written change looked like this: 0.91. One number. No sentence.

That question is the right one to ask before you let a grader with no written reason fail a change, and we asked it ourselves before we moved our eval suite's grader to Jev. Here is where the reason lives when the grader gives none, the three cases where a number is the wrong output, and why the uncertain band should stay with a person.

1. The reason is in the question you wrote and the state you passed

Start with what Jev returns. A Noul, a yes/no statement, returns one probability from 0 to 1 and nothing else. A Choice or a Score returns a probability per option plus a confidence number, which is computed from how spread out those probabilities are; for three options the docs give the formula as three times the largest probability, minus one, divided by two. That is the whole output. Jev is not trained to generate text, so there is no paragraph to ask for.

So the reason has to exist before the check runs. It is the question you wrote, the rubric you wrote into it (for a Score, the words describing each of the 2 to 10 levels), and the state you passed, which for us is the diff (the lines the change added or removed) and the acceptance criterion, and nothing more. The audit record for one decision is five things: the question with its rubric, the state, the probabilities, the threshold, and the model version. Anyone can rerun that record.

Rerunning it matters because the answer comes back the same. TypeSafe's own batching cookbook reports a standard deviation of 0.0 across five repeats for 11 of its 13 questions. LangChain's test, five agent cases run 100 times per judge, measured Jev's mean per-case variance on the quality score at 0.0000149, with GPT-5.6 Luna 433 times higher, GPT-5.6 Terra 913 times higher, and Claude Sonnet 4.6 92 times higher. The authors call this observational and say the experiment cannot tell them why Jev varied less. It can tell you that on those five cases, the same input gave the same grade.

Compare that with the paragraph. A written reason from a language-model judge is text the model produced next to its verdict. Rerun the same diff and you get a different paragraph, and sometimes a different verdict under it. Two paragraphs for two verdicts on one diff is a record of what the model wrote, and it tells you less about why the grade came out as it did than a number you can reproduce.

One qualification, because TypeSafe's own marketing invites the misreading. Their line that Jev "can't hallucinate" means the answer always matches the schema: a yes/no question never comes back as an essay. On TypeSafe's own workflow evaluation, as reported by DataCamp, Jev had 0 percent structured-output errors against 45.5 percent for Claude Haiku 4.5, and on the same workflows Jev agreed with the reference answer 67.8 percent of the time against 67.9 percent for GPT-5.6 Terra, 74.1 percent for GPT-5.6 Sol, and 73.1 percent for Claude Opus 5. Those are TypeSafe's workflows and TypeSafe's numbers. What they show is that the shape of the answer is guaranteed and the correctness is ordinary: a well-formed answer can still be wrong, and calibration against your own labels is what measures how often.

2. A number is enough for a gate you have already calibrated

A gate is a check that blocks a change from being released. What a gate needs is the same answer for the same input, a threshold set from examples people have labelled, and a record. A paragraph adds nothing here, because nobody acts on it. A change that fails the scope check is sent back, and the engineer fixing it needs the question and the diff, both of which are in the record.

TypeSafe's guardrail cookbook shows what a policy written in numbers looks like. One request screens a message with questions such as jailbreak (an attempt to make the model ignore its rules), harmful request, medical advice and self-harm, plus a severity score from 0 to 3, and the two example policies set a review threshold at 0.35 and an action threshold at 0.70 for the strict policy or 0.85 for the permissive one, with the routes ranked support, block, review, pass. Nobody reads a paragraph for each of a million messages. The thresholds are the explanation, and the person who set them is the person who explains them.

Our suite works the same way on a smaller scale. "This diff changes only files the task named" is a Noul with a threshold. Below it, the change fails. Above the clear-pass line, it proceeds. The confidence guidance TypeSafe publishes, 0.9 and above act automatically, below 0.5 route to a human, is where we started, and each check's cutoff moved from there once we had compared Jev's answers with our own labels.

3. Three cases where a probability is the wrong output

A failure nobody wrote a question for. Jev, in TypeSafe's own words, answers the question you wrote, not the one you meant. A probability of 0.97 on "satisfies criterion 4" says nothing about a problem criterion 4 never described. A language model reading the diff with an open question, "what is wrong with this change", can produce a sentence naming something you did not ask about. So we keep a language-model pass for open-ended review, and we treat what it names as a candidate for a new eval question. It never counts as a grade, because a check that grades against a standard the model invented on the spot is the problem we wrote about when we said never let the model grade its own work.

A design judgment. Whether the approach is the right one, whether the structure will still fit when the second feature is added, whether the change belongs in this service at all. A Score with 2 to 10 levels cannot hold an argument. A person writes the argument, and the eval suite's job is to make sure that person spends their time on this and on nothing a script could have checked.

Anything a regulator or an auditor wants explained in words. A probability with a threshold is a policy. It is a good policy when the threshold came from labelled examples and the disagreement rate at that threshold is known, but it is still numbers, and a person on your team has to write the sentence that says what the policy is and why. Keep the person, and hand them the record so the sentence is short.

4. The uncertain band stays with a person, for concrete reasons

Between the fail line and the clear pass is the band where the probabilities are too spread out to act on. We send those changes to a person, who labels them, and we keep the label to re-check the grader later. The reasons a change lands there are on TypeSafe's own jaggedness page, and each one maps to a kind of check.

Literal reading. Scoping words, negations and implied conditions are read exactly as written, with nothing inferred. A criterion that says "unless the field is optional" becomes two Nouls, and code combines them. Double negatives are answered less reliably, so we rewrite them.

Counting. Jev does not count reliably and the error grows with size. "Touches at most three files" is a count, so a script computes it and Jev never sees it.

Numeric comparison and dates. Jev cannot reliably judge whether two numbers are near each other, its Score levels are weak in numerical calibration, and it reads dates as text rather than as ordered quantities. "Response time stays under 200 milliseconds" and "deprecated before 1 June 2026" both belong in code.

Unrelated content. Accuracy falls as the state grows with content unrelated to the decision. We pass the diff and the criterion, never the whole repository.

Text written to manipulate the model. Adversarial content in the state can move the answer. A diff can carry a comment that reads like an instruction to the grader, so for scope questions we strip comments from the state, and any change whose diff contains instruction-shaped text goes to a person regardless of its grade.

A second model shares several of these weaknesses, which is why the band goes to a person and why the person's label is worth keeping. Our guide page on what a Jev grade cannot tell you lists each weakness beside the check it affects, and the guide to System One models explains what the model is and how it differs from a language model.

Credit to TypeSafe for publishing the jaggedness list at all, and for putting it in the documentation next to the benchmarks. And to the LangChain authors for writing "observational, not evidence" in their own results.

A grade you can rerun is worth more than a reason you cannot check.

Sources

• Noul, Choice, Score and Confidence: TypeSafe's documentation for what each question type returns and how confidence is computed.
• Model jaggedness, jev-1.13: TypeSafe's published list of known weaknesses, including literal reading, counting, dates, unrelated content and adversarial text.
• Parallel questions cookbook: TypeSafe's repeatability measurement, standard deviation 0.0 on 11 of 13 questions.
• LLM guardrails cookbook: TypeSafe's example policy with review and action thresholds.
• Can Jev be a better agent evaluator?: LangChain, 20 September 2026, variance figures and the authors' caveats.
• Jev: TypeSafe's System One model explained: DataCamp's report of TypeSafe's workflow evaluation, including the structured-output error rates.