How Reveneau uses Jev

This page describes one company's practice, Reveneau's, and it is written as practice rather than as a result. Reveneau generates all of its code with AI. Every change has to pass an eval suite written from the specification before it is released, and the company takes responsibility for the whole project through production and after release. The suite is where Jev now sits, and this page says exactly where, why, and what we will and will not claim about it.

Why the grader matters to us more than to most

When a machine writes the code, the writing stops being the hard part. What remains is deciding what correct means and proving it on every change without a person having to remember. That is the argument of our eval-driven development guide, and it is why the grader is on the path of every release we make. A slow grader makes the suite run less often. A grader that cannot explain its verdict, or that gives a different verdict on the same input twice, makes the suite less trusted. Both problems were real for us with a language model as judge, and the second was the worse one.

Two kinds of check, kept apart

The suite has two kinds of check, and the split is the most important design decision in it.

Checks with a deterministic answer. A test passes or fails. A schema validates or does not. A migration applied to a copy of production data holds its invariants or breaks them. A performance threshold is met or missed. A build compiles. These have one right answer that code can compute, and they never go to a model. They never did, and they never will, because a model that gets a deterministic check wrong 1 percent of the time is worse than code that gets it right every time. TypeSafe's own guidance says the same: "Keep control flow, deterministic rules, and side effects in code" [1].

Checks that need a judgment. Does this change do what the acceptance criterion says, as written? Does this diff stay inside the files and behaviour the plan listed? Did the agent's trace follow the plan it was given? Does this generated text follow the instruction it was given? These are questions about the relation between two texts, and no line of code answers them. They used to go to a language model as judge. They now go to Jev.

How each judgment check is written

Each judgment check is one question to Jev, of one of two types.

A Noul for anything that is pass or fail. The acceptance criterion is the statement, written as one positive clause in the words the specification uses. "The export endpoint returns one row per active user." "Every file changed in the diff appears in the plan's file list." "The response addresses the question the user asked." Jev returns a probability that the statement holds for the state [2].

A Score for anything where partial credit matters. How completely a response covers a rubric with several parts, on a scale described in words from "covers none" to "covers all". Jev returns a probability-weighted value on that scale [3].

The state for each check is exactly what the check needs: the criterion, the diff and the plan for a scope check; the instruction and the output for an instruction check; the plan and the trace for a plan-adherence check. Nothing else goes in, because TypeSafe documents that accuracy falls as the state fills with unrelated content [4], and because a shorter state is a cheaper one. Every count, comparison and date the check depends on is computed in code and written into the state as a fact, for the reasons on where Jev is weak. All of a change's judgment checks go in one request, since TypeSafe evaluates every question in parallel [5], and many questions in one call covers why that is the cheap shape.

The rubric lives in the criteria. That was true before the switch and it is the reason the switch was possible: a criterion written so that a machine can check it, in the sense of writing specs an agent can verify, is already most of a Noul question. The work of the migration was mostly rewording criteria that had negations or implied conditions in them, because TypeSafe documents that Jev reads those literally [4]. How to write the state and the questions is the page we wrote from that experience.

What happens to a grade

Three outcomes, set per check according to what a wrong answer costs. This is the part of the suite where Reveneau's own thresholds live, and they are ours to defend.

Below the threshold, the change fails. A scope check has a high threshold, because a diff outside the plan is never acceptable and a false fail costs only a re-run. A rubric-coverage Score has a threshold set from labelled cases, because the cost of a false fail and a false pass are closer.

In the uncertain band, the change goes to a person. That person reads the diff and the criterion and gives a label. The label is kept, with the state and the question, so that the grader can be re-checked against it later. This is the band where the model has said it is unsure, and the labels from it are the most useful data we collect: they tell us whether a criterion needs rewording, whether a threshold has drifted, and, when TypeSafe releases a new version, whether the calibration curve moved. TypeSafe's confidence docs describe the band as "Proceed with caution" and the low end as "Route to a human", and add that different actions should be gated at different levels depending on the consequences [6], which is the rule we apply.

Above the threshold, the change passes that check. It still has to pass every deterministic check, and a person still reviews the diff once everything passes. The reviewer stays. What Jev removed is the wait for a judgment that used to take seconds per check and now takes a fraction of a second for the whole set.

The one figure, and why only one

On our own suite, measured against our previous grader, the run is ten times faster. That is the figure we state, and it is the only one.

We do not attach a dollar figure, because we have not measured the cost in a way we could show the working for. We do not attach a defect rate, because we have not run the controlled experiment that would separate the grader's effect from everything else that changed this month. We do not attach a count of evals, a count of changes, or a count of clients, because a consultancy whose whole pitch is verification should not publish numbers it cannot verify. The pillar of our eval-driven development guide makes the same refusal about evals in general, and it applies here.

What we will say is what changed in practice. The grading step used to be the slowest part of the suite. It is now one of the fastest. The suite therefore runs more often, and a failing change is caught sooner. And the grades are stable: the same state and the same question return the same probability on repeat, which TypeSafe's own cookbook measured as a standard deviation of 0.0 across five repeats for 11 of 13 questions [7], and which matches what we see. A grader that returns the same verdict twice is a grader an engineer can argue with, and that was the thing we wanted most.

What we checked before switching

Jev was five days old when we started, and a grader on the path of every release deserves more caution than a five-day-old model would usually get. We did four things.

We re-ran the labelled cases. Every judgment check had a set of past verdicts from the language model judge, some of them corrected by a person. We sent the same states and criteria to Jev and compared. The disagreements were the useful part, and most of them traced to a criterion with a negation or an implied condition, which is on TypeSafe's jaggedness page [4] and which rewording fixed.

We set thresholds per check from those cases rather than from TypeSafe's default bands. The 0.9 and 0.5 bands in the confidence docs [6] were the starting point, and the labelled cases moved most of them.

We pinned the model version. The current model is jev-1.13.0, and the jev-latest alias moves with each release [8]. A grader whose threshold silently shifted would pass or fail changes for a reason nobody chose, so the suite names the version and a version change is itself a change to be tested against the stored labels.

We kept the fallback. If the API returns a 429 or a 529, the docs say to back off and retry [9]. The suite does, with a ceiling, and past the ceiling the judgment checks are marked as not run rather than passed. A check that did not run is never a pass.

The fuller account is in what we checked before grading with a week-old model, and the story of the move is in we moved our eval grader to Jev.

What stays as it was

Every deterministic check stays in code, and that is permanent. The language model that writes the code stays; TypeSafe's own coding-agents page says Jev "is not a drop-in replacement" for that model [10], and we agree. The human review of a passing diff stays. And a few checks stay on the old grader, because Jev's errors on them clustered in a way that rewording the criterion left unfixed.

What this means if you are choosing a partner

Asking whether we use AI tells you little, because everyone does. Ask what the check is, who wrote it, whether it runs on every change, what grades it, and what happens when the grade is uncertain. This page is our answer to those questions, and how to evaluate a vendor's eval suite is the list of questions we would want you to ask anyone, including us.

If you want to build the same thing, evals with Jev is the how-to, written from this setup. If you want to know what the model is before you decide, the pillar, Jev and System One models, is the plain guide. And if you want to talk it through with the team that runs this suite, contact us.