Faster evals with Jev: how we grade AI-written code / Run it on every change

Run it on every change

Running Jev-graded evals on every pull request

A Jev-graded check belongs in the same pipeline as the tests, after them, with the power to block. The shape is one request per change carrying every question, sent after the deterministic checks pass, with the probabilities compared to thresholds in code. The published limits are 1,200 requests per minute and 250,000 tokens per second, with 32k tokens for the state plus the longest question, so a large diff is split by behaviour rather than sent whole. The price is $0.042 per million input tokens with output tokens free, which puts a typical grade under a tenth of a cent. This page gives the pipeline, the logging, and the arithmetic.

Published September 22, 2026. Editorial.

Key takeaways

Run the deterministic checks first and send a change to Jev only when they pass. A change that fails a test has no need of a grade.
One request per change carries every question. TypeSafe's own batching test measured 13 questions in one call at 0.27 seconds against 2.71 seconds for thirteen calls.
Respect 1,200 requests per minute and 250,000 tokens per second, retry 429 and 529 with a delay that doubles each time, and split any state over 32k tokens by behaviour.
Log the model version, every probability, the question identifiers, and a hash of the state on every request. Without the version you cannot see the day the grader changed.
A failing grade blocks the merge like a failing test. An uncertain grade blocks until a person labels it. Neither is advisory once calibrated.

The pipeline that runs a Jev-graded check is the pipeline you already have, with one stage added after the tests. Everything about making the block real, splitting checks by speed, and refusing to let an agent weaken a check, which evals in CI for coding agents covers, still applies. This page is the added stage: what it sends, what it gets back, what it logs, what it blocks, and what it costs. Reveneau runs this stage on every change, and on our own suite the run is ten times faster than it was with a language-model grader.

Where the stage sits

After the deterministic checks and before the merge. The order matters for two reasons. A change that fails a unit test or a schema check has no need of a grade, and skipping the request saves the cost and keeps the failure message clear: the test failed, and that is the message. And the judged checks are the ones that depend on the change being complete; grading a diff that does not compile wastes the grade.

The stage has three steps. Build the state and the questions for this change. Send one request. Compare the answers to the thresholds and decide pass, fail, or route to a person.

Building the request

The state is the evidence the questions refer to: the diff, split into the parts that touch each behaviour, and the quoted spec lines, conventions sections, or plan text the questions name. Nothing else, because TypeSafe says accuracy falls as the state grows with unrelated content and names hidden context as an anti-pattern [1]. The context limit is 64k tokens per request, with 32k for the state plus the longest question [2], and a diff that would exceed 32k tokens is split by behaviour into two or more requests, each with the questions that apply to its part. Truncating a diff to fit is the wrong fix, because the truncated part is the part the model never sees.

The questions are the Noul statements and Score rubrics from writing Noul checks for acceptance criteria, turning a rubric into Score questions, and the composite set from composite scoring for code quality. All of them go in the one request. TypeSafe's docs say every question is evaluated in parallel and adding questions barely changes the response time [3], and its batching cookbook measured it on jev-1.12: 13 questions against a 53,777-character article in one call cost $0.000497 and took 0.27 seconds, against $0.006090 and 2.71 seconds for thirteen single calls, which TypeSafe reports as 12.2 times cheaper and 10.0 times faster [4]. The same cookbook reports that answers did not depend on what else was in the request, with a standard deviation of 0.0 across repeats for 11 of the 13 questions [4], so batching does not trade accuracy for speed.

The request goes to a single endpoint, POST to api.typesafe.ai/v1/systemone, with a secret key sent in each request; TypeSafe publishes code libraries for calling it, a Python one for 3.10 and above and a JavaScript one for Node 20 and above, both reading the key from TYPESAFE_API_KEY [5]. The key lives in the pipeline's secret store, the same place as every other credential, and never in the repository.

Rate limits and backoff

TypeSafe's published limits are 1,200 requests per minute and 250,000 tokens per second [2]. For a pipeline that sends one request per change, the request limit is 20 per second and is the one that binds on a typical change: at 8,480 tokens per request, the token limit would allow 29 requests per second, so the request limit is reached first. At a state near the 32k limit, the token limit allows 7 requests per second and binds before the request limit does. Either way, a pipeline that grades a hundred changes an hour is far from both, and a pipeline that re-grades a stored calibration set of 200 cases 10 times each, which calibrating Jev against your own human labels asks for, needs to pace itself.

The API returns 429 for a rate limit and 529 when TypeSafe is temporarily overloaded, alongside 401 for an invalid key and 422 for a validation error, and the docs say to use exponential backoff, meaning a retry delay that doubles each time [5]. The pipeline retries 429 and 529 with that doubling delay and a cap, and it does not retry 401 or 422, because those are bugs in the request rather than conditions that will pass. Openlayer's README notes that in one of its benchmark runs a few connections hung and its client's 15-second timeout and retry let the run finish, at the cost of a p95 of a minute for that run [6]; a timeout with a retry is part of the stage, and a request that fails every retry marks the change as ungraded and routes it to a person rather than passing it.

What to log

Every request writes one record, and the record is what makes the grader auditable. It holds: the change identifier; the model version field from the response, which is the only way to see the day jev-latest moves to a new release, since jev-latest and jev-preview are aliases that TypeSafe can repoint [2]; the identifier and text of every question; a hash of the state (a short fingerprint of its exact contents), so a re-run can be confirmed identical; every probability, every per-level probability on a Score, and every confidence number; the thresholds applied; the outcome; and the latency. If a person labels the change, the label and reason join the same record, as thresholds, confidence and escalation describes.

The record is also what the monthly re-check reads. Without the state hash and the model version, a drop in agreement cannot be separated into the model having changed and the questions having changed, and without the probabilities you cannot see whether a threshold should move.

What a failing grade blocks

The same thing a failing test blocks: the merge. A judged check that returns below its fail line marks the change as failed, the pipeline reports which question failed and its probability, and the change does not reach the main branch. An uncertain grade marks the change as blocked pending review, and the person's label, pass or fail, is what releases or fails it.

Neither outcome is advisory once the grader is calibrated. The older guidance for a language-model judge, in using a model as a judge, was to keep model grades advisory until a validation sample showed agreement, and the same holds here: a Jev-graded check is advisory during the shadow period the migration checklist describes and blocking after it. Once blocking, the rule from the CI guide applies in full: nobody, and no agent, weakens a check to get a change through. A threshold change is a pull request.

What the failure message says is a design choice worth a sentence. A decision model returns no rationale, so the message is the question text and the probability: "Noul expired_token_error_code: 0.21. The diff adds a code path that returns the JSON error code token_expired when the access token's expiry time is in the past." That is enough for an engineer to open the diff and look at one thing. It is also enough for a coding agent, which reads the failed question and tries again, in the loop that eval-driven development describes; a question written as an observable behaviour is a better instruction to an agent than a paragraph of a judge's reasoning was.

The cost

The public price is $0.042 per million input tokens, with output tokens free [2]. Everything below is arithmetic on that price and nothing else.

A typical change: a state of 8,000 tokens (the relevant parts of the diff plus the quoted spec lines) and twelve questions of 40 tokens each, 8,480 tokens in total. At $0.042 per million that is $0.000356 per grade. A thousand such grades cost $0.36. Ten thousand cost $3.56.

A large change at the state limit: 32,000 tokens of state and the same twelve questions, 32,480 tokens, $0.001364 per grade. A thousand cost $1.36.

The calibration re-run: 200 stored cases, 10 repeats each, at the typical size, is 2,000 requests and 16,960,000 tokens, $0.71 per full re-check. Monthly, that is under a dollar for the measurement that tells you whether the grader still agrees with your reviewers.

For comparison, Openlayer's own benchmark put its agent evals at $0.03 per 1,000 samples on Jev against $2.60 with Ragas on GPT-4.1-mini [6], and LangChain's five-case test cost $0.34 with Jev against $28.17 with Claude Sonnet 4.6 [7]. Both are other people's workloads; the arithmetic above is what your pipeline would pay at the public price for the shapes described, and it excludes engineering time, which is the real cost of the migration.

Two failure modes of the stage itself

The first is the request that never arrives. A pipeline that treats a network failure as a pass has built a grader that passes changes when the vendor is down. The stage treats no grade as no merge, and routes the change to a person.

The second is the state that leaks. A diff written by a coding agent can contain text that reads as instructions, and TypeSafe's jaggedness page says adversarial content in the state can move the answer [1]. The stage runs the deterministic checks first, sends only the parts of the diff a question needs, and routes any change touching security-sensitive files to a person regardless of its grade. What a Jev grade cannot tell you covers the rest of that list.

The stage, in one paragraph

Tests first. Then one request with the diff, the spec lines, and every question. Retry 429 and 529 with a growing delay, treat any other failure as no grade and no merge. Log the version, the state hash, and every number. Compare to thresholds in a reviewed file. Fail the merge below the line, hold it for a person in the middle, pass it above. Reveneau's pipeline is that paragraph, and the grade it produces is the same probability whether the change was written by a person or an agent, which is the point of having a grader at all.

Best for

Pipelines that already block merges on tests and can add one stage after them
Teams with a secret store for the API key and a log store for the per-request record
Suites where the judged checks were the slowest stage

Avoid if

The pipeline cannot block the merge when the API is unreachable
Diffs routinely exceed 32k tokens and cannot be split by behaviour
Nobody will log the model version and the probabilities per request

Check before you decide

The stage runs only after the deterministic checks pass
429 and 529 are retried with exponential backoff; 401 and 422 are not retried
A failed grade and an ungraded change both block the merge

Common questions

Where in the pipeline does the Jev stage run?

After every deterministic check passes and before the merge. A change that fails a test needs no grade, and skipping the request keeps the failure message clear and saves the cost. The stage builds the state and questions for the change, sends one request, compares the answers to thresholds in a reviewed file, and marks the change passed, failed, or held for a person.

Should each question be a separate request?

No. Put every question for one change in one request. TypeSafe's docs say questions are evaluated in parallel with almost no added response time, and its batching cookbook measured 13 questions in one call at $0.000497 and 0.27 seconds against $0.006090 and 2.71 seconds for thirteen calls, with a standard deviation of 0.0 across repeats for 11 of the 13, so batching costs no accuracy.

What are the rate limits and which one binds first?

1,200 requests per minute and 250,000 tokens per second. At a typical 8,480-token request the token limit would allow 29 requests per second, so the 20-per-second request limit binds first. At a 32,480-token request the token limit allows 7 per second and binds first. A pipeline grading a hundred changes an hour is far from either; a calibration re-run of 2,000 requests needs pacing.

How should the pipeline handle a 429 or 529 response?

Retry with exponential backoff, as TypeSafe's docs say, meaning a delay that doubles after each failed attempt, with a cap, and a per-request timeout. 401 (invalid key) and 422 (validation error) are bugs in the request and are not retried. A request that fails every retry leaves the change ungraded, and an ungraded change blocks the merge and goes to a person. The stage never treats an unreachable API as a pass.

What must be logged on every request?

The change identifier, the model version field from the response, the identifier and text of every question, a hash of the state, every probability and per-level probability and confidence number, the thresholds applied, the outcome, and the latency. The model version is the only way to see the day jev-latest moves, and the state hash is what lets a monthly re-run be confirmed identical.

What does a failing Jev grade block?

The merge, in the same way a failing test does. The pipeline reports the question identifier, its text, and the probability, which is enough for an engineer or a coding agent to open the diff and look at one thing. An uncertain grade holds the change until a person labels it. Once the grader is calibrated, neither outcome is advisory, and no one weakens a threshold to get a change through.

How much does grading one pull request cost?

At the public price of $0.042 per million input tokens with output free: a state of 8,000 tokens plus twelve 40-token questions is 8,480 tokens, $0.000356 per grade, so 1,000 grades cost $0.36. A change at the 32k state limit is 32,480 tokens, $0.001364 per grade, $1.36 per thousand. Those figures are arithmetic on the price and exclude engineering time.

What happens when a diff is larger than the 32k-token state limit?

Split it by behaviour into two or more requests, each carrying the part of the diff and the questions that apply to it. Do not truncate, because the part cut off is the part the model never sees, and do not send unrelated files, because TypeSafe says accuracy falls as the state grows with unrelated content. A diff that cannot be split by behaviour is a change that should have been two changes.

How does the failure message work without a written rationale?

The message is the question text and the probability, for example a Noul at 0.21 on the statement that the diff adds a code path returning the error code token_expired for an expired token. Because each question names one observable behaviour, the message points at one place in the diff. A coding agent reads the failed question and retries, which is a clearer instruction than a judge's paragraph.

Where does the API key live?

In the pipeline's secret store, the same place as every other credential, and never in the repository or in a log record. TypeSafe's code libraries read it from the TYPESAFE_API_KEY environment variable. The per-request log record holds the model version, the state hash, and the numbers, and nothing that would let a reader of the logs call the API.

References

More in Run it on every change

Thresholds, confidence and escalation in an eval suite

A probability becomes a verdict only when code compares it to a threshold, and the design of those thresholds decides what the grader is worth. TypeSafe's guidance gives the starting bands: act automatically at 0.9 and above, route to a person below 0.5, proceed with caution in between. In an eval suite that means a check on a flow that moves money fails at a higher grade than a check on a log message, the uncertain band always goes to a person, and that person's label is stored, because it is the data you re-check the grader against every month.

Grading agent traces with Jev

When the thing under test is an agent, the artefact is a trace: the ordered record of model calls, tool calls, and tool results. The questions change shape from whether a diff meets a criterion to whether the agent chose the right tool, stayed inside its scope, avoided looping, and completed the goal. Two third parties have published tooling for this. Openlayer's jevals runs 37 built-in evals under an MIT licence and calls itself alpha. LangChain tested Jev as a LangSmith evaluator on five cases and called the result promising but early. This page covers what a trace check asks and how far the public work goes.

Checklist: migrating an existing eval suite to Jev

This is the sequence for moving the judged checks in an existing eval suite from a language-model judge to Jev, as a numbered checklist with a definition of done for each step. It is printable. The order matters: the inventory comes before any rewriting, the calibration set comes before any threshold, and both graders run side by side before either is switched off. A team that follows it ends with a suite where the deterministic checks are untouched, every judged check is a question in a reviewed file, every threshold names the run that set it, and the uncertain band still goes to a person.

Previous: Thresholds, confidence and escalation in an eval suite Next: Grading agent traces with Jev