Grading agent traces with Jev

A coding agent's pull request can be graded from its diff. An agent that books a flight, answers a support ticket, or runs a migration leaves no diff, only a trace, and the trace is what has to be graded. Reveneau grades agent traces in its eval suite with Jev, using the same Noul and Score questions as for a diff, over a different state. This page is what those questions look like, what the two published third-party toolkits do, and where both stop.

What a trace is, for grading purposes

A trace is the ordered record of one agent run: each model call with its input, each tool call with its arguments, each tool result, and the final output. For grading, the state is the trace plus whatever the questions refer to: the plan or task the agent was given, the list of tools it was allowed to use, and the acceptance criteria for the run. Same rules as for a diff: only what the questions name, because accuracy falls as the state grows with unrelated content [1], and the state has to fit in 32k tokens with the longest question [2], so a long trace is cut to the calls the questions are about.

The reason a trace needs a judged check at all is that most of what matters about an agent run has no exact answer. Whether the agent called the tool named in the plan is deterministic and stays in code. Whether the tool it called was the right one for the step, given the plan, is a judgment. How to evaluate an AI agent covers the full evaluation of an agent as a product; this page is the grading mechanism for the trace-level checks in an eval suite.

The four questions

Tool choice. For each step in the plan, a Noul: "At the step where the plan says to look up the booking, the agent's tool call is to the booking lookup tool with the booking reference from the user's message as its argument." State: the plan step and the corresponding tool call. A Choice can do the same job when the question is which of several named tools was right.

Scope. A Noul over the whole trace: "Every tool call in the trace is to a tool named in the allowed list." That one is deterministic if the allowed list is exact, and the judged version is the one that matters: "Every action the agent took was within the task the user described." State: the task and the sequence of tool calls with their arguments. AI agent permissions and guardrails and how to audit an AI agent's tool permissions are the design side of the same question.

Loop detection. Here counting is a temptation and TypeSafe says the model does not count reliably [1]. So the deterministic check counts: three calls to the same tool with the same arguments is a loop, and code says so. The judged check is the one that needs judgment: "The agent's repeated calls to the search tool each use different arguments in response to the previous result." State: the repeated calls and the results between them.

Goal completion. A Score with worded levels: level 0, the final output does not address the task; level 1, the output addresses the task but the acceptance criterion is not met; level 2, the acceptance criterion is met by the final output. State: the task, the criterion, and the final output. Written following turning a rubric into Score questions, with one observable behaviour per level.

All four kinds go in one request per trace, with the composite in code as composite scoring for code quality describes: scope and any safety statement as gates, the rest weighted.

Openlayer's jevals

Openlayer released jevals on 20 September 2026 under the MIT licence. It runs agent evals and guardrails as one Jev request per trace, with 37 built-in evals across three groups: agent (ToolChoice, Grounded, StayedInScope, LoopDetection, GoalCompletion, and others), security (PromptInjection, Jailbreak, PII, PHI, SecretsExposure, and others), and quality (Faithfulness, Hallucination, InstructionFollowing, CustomRubric, and others). It ships adapters for the OpenAI Agents SDK, LangGraph, and the Claude Agent SDK, a text-file format (YAML) for eval definitions, gates, a server that lets AI tools call it, and a command-line tool [3]. The names of the agent evals map onto the four questions above, and the README's CustomRubric is the Score pattern.

Three things from the README belong next to any use of it. The library calls itself alpha, and says its framework adapters were written to the SDK docs and tested against fakes rather than run live, and that its direct TypeSafe backend was tested against a mock [3]. Its benchmark, the Ragas comparison at $2.60 against $0.03 per 1,000 samples and 22 to 35 seconds against 0.8 seconds for 20 samples, was run by the library's own authors [3]. And the accuracy figure it cites, JevBench at 83 to 87 percent on Banking77 and CLINC150, is a result on intent-classification benchmarks, which is a different task from grading a trace; the README itself says calibration varies by task [3]. The README also has a calibrate command that fits a threshold against your own labels and reports the error rate at that threshold, which is the right tool and the reason calibrating Jev against your own human labels exists as a page. Our AI News item on the release is here.

LangChain's LangSmith test

LangChain's team asked whether Jev could serve as an evaluator inside LangSmith, its tracing and evaluation product, and published the test on 20 September 2026. They took five weather-agent cases with human-labelled correct answers and ran each judge 100 times per case, 500 decisions per judge, on identical traces. On the binary does_pass question, Jev agreed with the human label 100 percent of the time, GPT-5.6 Terra 99.8 percent, GPT-5.6 Luna 96.4 percent, and Claude Sonnet 4.6 80.0 percent. Jev's mean per-case variance on the quality score was 0.0000149, with the language models 92 to 913 times higher. Jev took 0.44 seconds per call at $0.00035 against 2.16 to 2.83 seconds for the language models, and the run cost $0.34 with Jev against $28.17 with Claude Sonnet 4.6 [4].

The authors' own framing is the one to keep: "promising, but early", "observational, not evidence", the experiment "cannot tell us why Jev's scores varied less", and "low cost can amplify mistakes" [4]. Five cases is a small set, and a trace grader that is right on five weather cases has not been shown to be right on a sixth domain. What the test does show is the signature of the mechanism: a grader that scores fixed answers returns the same number for the same trace, and a grader that generates a rationale does not, which is the argument LLM as judge vs a decision model makes in full.

LangChain has also published an integration, the langchain-typesafe package with a TypeSafeClassifier, used for model routing and for a layer that runs before each tool call and blocks risky ones before they execute [5]. That is the runtime use of the same questions, and it belongs to Jev in production rather than to this hub.

Grading in the suite and screening at runtime

The four questions above can be asked at two moments. In an eval suite, they are asked about a completed trace, after the run, to decide whether the agent's behaviour on a fixed set of cases is acceptable before a change to the agent is released. At runtime, the scope and safety questions are asked about a proposed tool call, before it executes, to decide whether to allow it. Openlayer's gates and LangChain's pre-call layer are the runtime shape; the Claude Agent SDK adapter in jevals, for example, is a hook that runs before a tool call and returns allow, ask, or deny [3].

The questions are the same and the thresholds are different. In the suite, a change to the agent that lowers the goal-completion score on the fixed cases fails, and the fixed cases are the agent's regression suite. At runtime, a single tool call with a low scope probability is blocked and the run goes to a person. Reveneau does the first in its eval suite for every change to an agent, and the case set grows the way any eval suite grows: every incident becomes a trace with a label. Can you trust an AI agent is the argument for why both moments are needed.

Where trace grading stops

Everything on what a Jev grade cannot tell you applies, and three items apply harder to traces than to diffs. Counting: how many tool calls, how many retries, how many tokens, all deterministic, never judged. Adversarial content: a tool result can contain text that reads as an instruction, TypeSafe says adversarial content in the state can move the answer [1], and Openlayer's own security group of evals exists because the same tool result can move the agent; the grader that reads the trace is exposed to the same text as the agent that produced it. And the unknown failure: a trace grader answers the questions it was given, and an agent that fails in a way nobody wrote a question for gets graded on the questions that exist. Keep a person on the traces from the highest-consequence cases, and keep an advisory language-model pass where a missed unknown costs most.

What to build first

Take the ten runs of your agent that mattered most in the last month, including the failures. Write the plan, the allowed tools, and the acceptance criterion for each as text. Write the four questions for each, following the rules on this hub, and label each run by hand. That is a ten-case calibration set and a ten-case regression suite at once, and it is what the migration checklist asks for before any grade blocks a change. Grow it from incidents. The published tests are other people's five cases; yours are the ones that count.