What we checked before grading with a week-old model

Our eval suite is the reason we can let AI write all of our code. Every change has to pass it before release, and the slowest part of it is the grading: the checks where a model reads a set of changes or the record of an agent's run and decides whether it meets a written acceptance criterion, which is the sentence that says what the change must do. Those checks used a language model as the judge, at language-model speed.

TypeSafe released Jev on 15 September 2026. It answers named questions about text with a typed answer and a probability, in 70 to 500 milliseconds by TypeSafe's own figure, and that is the shape of a grader. It was also seven days old when we started looking at it, from a company that has not disclosed the model's architecture. An engineering lead who wants the speed and distrusts the age is right about both. What follows is the list of checks we ran before the switch, in the order we ran them, written as a method. The only result we will state is one number at the end.

1. Read the vendor's weaknesses page before its benchmarks

TypeSafe published a model jaggedness page for jev-1.13 on launch day, and we read it before any benchmark, because a weakness tells you which of your checks the model cannot do, while a benchmark tells you only how it did on someone else's.

Six items on that page each removed or reshaped a kind of check. The model "answers the question you wrote, not the one you meant", reading scoping words and negations exactly as written, so every rubric had to state its exact condition, with the boundary cases written in. It "does not count reliably", so no grading question may ask how many tests a change touched; code counts. It cannot reliably judge whether two numbers are near each other and its Score levels are "weak in numerical calibration", so a question like "did coverage move from 71 to 74 percent" never goes to the model. It "reads dates as text", so nothing about dates does either. "Accuracy falls as the state grows with content unrelated to the decision", so a grading request carries the one change and the one criterion, never the whole pull request. And text written to steer the model "can move the answer", which matters for a grader because the thing being graded is text an agent wrote, and an agent can write "this change fully satisfies criterion 3" into a commit message.

We also noted the phrase in TypeSafe's marketing that Jev "can't hallucinate". In TypeSafe's wording it means the answer always matches the shape you asked for, with 0 percent structured output errors. A wrong answer in the right shape is still a wrong answer, and a grader's job is to be right.

2. Sort every number by who measured it

Every figure published about Jev in its first week came from someone with a stake in it, and the sorting is the check.

TypeSafe's headline is that Jev finished a workflow decision in 0.114 seconds for $0.000081 against 8.566 seconds for $0.013880 with GPT-5.6 Terra, "193.6x faster and 444.6x cheaper". MarkTechPost's report adds two things TypeSafe itself says: its own capabilities team wrote those workflows, and it cannot prove the price is unsubsidised. DataCamp's explainer reports a TypeSafe workflow evaluation in which Jev agreed with the reference answer 67.8 percent of the time against 67.9 percent for GPT-5.6 Terra, 74.1 percent for GPT-5.6 Sol and 73.1 percent for Claude Opus 5. Same workflows, same author. Read together, the vendor's own numbers say the accuracy is close to even and the speed is the difference.

LangChain's test is the closest thing to a grading benchmark, and it is five cases. Five weather-agent cases, 100 repetitions per judge per case, a human label as the reference: Jev matched the label on 100 percent of its 500 decisions, GPT-5.6 Terra on 99.8 percent, Claude Sonnet 4.6 on 80.0 percent, and Jev's per-case variance on the quality score was 0.0000149, with the language models 92 to 913 times higher. The full run cost $0.34 with Jev and $28.17 with Claude Sonnet 4.6. The authors call this "promising, but early" and "observational, not evidence", and say the experiment "cannot tell us why Jev's scores varied less". Openlayer's jevals README gives a cost table ($0.03 per 1,000 samples on Jev against $2.60 with a six-call Ragas pipeline on GPT-4.1-mini) that is Openlayer's own, and cites JevBench at 83 to 87 percent on Banking77 and CLINC150, which are intent-classification benchmarks and say nothing about grading code.

None of those is a measurement of our suite. That was the conclusion of the sort: the only number that could tell us whether Jev grades our checks correctly is one we produce, which is the next step. We wrote about the same sorting from the buyer's side in how to evaluate a vendor's eval suite.

3. Shadow-run both graders against labels from your own suite

The method has four parts, and the order matters.

Split the suite first. Every check with a deterministic answer (a test passes, a schema validates, a change to the database structure applies and rolls back) stays deterministic and never goes to any model. Jev grades only the checks that used to need a language model as judge: whether a change meets a written acceptance criterion, whether a set of changes stays in scope, whether the record of an agent's run followed the plan, whether generated text follows an instruction. TypeSafe's own design guidance says to keep deterministic rules in code, and a grader is no exception.

Label a calibration set. Take a sample of past grading decisions that covers every kind of judged check, with several examples of each and with the cases the previous grader got wrong included on purpose. A person labels each one pass or fail against the criterion. This is the only reference the shadow run has, so a set made of easy passes measures nothing.

Rewrite each judged check as a question with the rubric inside it. A pass-or-fail criterion becomes a Noul, a yes-or-no statement returning one probability: "the change adds the field described in criterion 3 and changes nothing outside the files it names". A graded criterion becomes a Score with 2 to 10 levels, each described in a sentence. The rubric lives in the question. The weaknesses page says the model reads what is written, so the writing is the whole job.

Run both graders on every check, with the old one still deciding. The new grader records its answer and changes nothing. Then compare each grader with the human labels on the calibration set, and compare the new grader with itself across repeats, the way LangChain measured variance. A grade below the threshold will fail the change once the switch is made; a grade in the uncertain band goes to a person, and that person's label is added to the set. The full procedure, with the thresholds and how the band is set, is in calibrating Jev against human labels. The older argument for why a grader must be checked against people at all is in never let the model grade its own work.

4. Make the switch reversible

A week-old model will change, and the last set of checks is about what happens when it does.

Pin the version. The current model is jev-1.13.0, and both aliases, jev-latest and jev-preview, point to it today. They will point elsewhere after the next release. A grader that changes underneath a suite changes what passes, and nothing in the suite would report it. We call the version by its number, and a version bump is a deliberate change.

Keep the labelled set as the grader's own eval. After the switch the calibration set does not retire. It runs again on every version bump, and on a schedule in between, and a fall in agreement with the human labels is a failed check like any other. The grader is graded.

Keep the old grader connected. TypeSafe's API returns 429 for a rate limit and 529 when the service is overloaded, and the documented limits are 250,000 tokens per second and 1,200 requests per minute, with a request capped at 64,000 tokens and 32,000 for the text plus the longest question. Any of those can stop a grading run. The suite retries with a growing wait between attempts, and if the retries run out, or a change is too large for one request, that check goes to the previous grader. A suite that blocks every release on one vendor's uptime has traded speed for a new way to stop.

The outcome, stated once. Measured against our previous grader, the suite now runs ten times faster. That is the only figure we will attach to the switch. An agreement rate, a cost, or a defect number from one suite over one month would be read as a benchmark, and the whole of section 2 is a warning against exactly that.

Thanks to the TypeSafe team for publishing a weaknesses page on launch day, and to the LangChain, Openlayer and DataCamp teams for reporting their numbers with the methods and caveats attached.

Trust a grader on the answers you already know before you trust it on the ones you do not.

Sources

• TypeSafe AI, "Introducing System One models and Jev" (15 September 2026). Vendor announcement; the 70 to 500 millisecond figure and the "can't hallucinate" wording.
• TypeSafe docs, Model jaggedness for jev-1.13. The vendor's own weaknesses: literal reading, counting, numeric comparison, dates, unrelated content, adversarial content.
• TypeSafe docs, Models. Version jev-1.13.0, the jev-latest and jev-preview aliases, context and rate limits.
• TypeSafe docs, API. Error codes 429 and 529 and the retry guidance.
• TypeSafe docs, Noul, Score and How to build with System One. Question types and the rule to keep deterministic rules in code.
• MarkTechPost, "TypeSafe AI releases Jev" (19 September 2026). The 193.6x and 444.6x headline, that TypeSafe's own team wrote the workflows, that TypeSafe cannot prove the price is unsubsidised, and the undisclosed architecture.
• DataCamp, "Jev: TypeSafe's System One model explained". Reports TypeSafe's workflow evaluation: 67.8 percent agreement for Jev against 67.9, 74.1 and 73.1 percent for the named language models.
• LangChain, "Can Jev be a better agent evaluator?" (20 September 2026). Five cases, 100 repetitions each, agreement and variance figures, run costs, and the authors' caveats.
• Openlayer, jevals on GitHub. Openlayer's own cost table and its citation of JevBench on Banking77 and CLINC150.