Checklist: migrating an existing eval suite to Jev

Reveneau followed this sequence when it moved its own eval suite's judged checks to Jev, and the result on our suite is a run ten times faster than with the language-model grader. The checklist is written so that another team can follow it without us. Each step has a definition of done; do not start the next step until the current one meets it. Print it, and cross the steps off.

Step 1. Inventory every check

List every check in the suite, one row each: name, what it asserts, where it runs in the pipeline, and how long it takes. Include the checks nobody has looked at in a year.

Done when: every check in the pipeline configuration appears in the list, and the list's total run time matches the pipeline's measured run time within the accuracy of your timing. A check missing from the list is a check the migration will break by accident.

Step 2. Classify each check as deterministic or judged

Mark each row. Deterministic: the check has an exact answer that code produces, such as a test, a type check, a schema validation, a lint rule, a rule that a migration must keep true, a performance threshold. Judged: the check has no exact answer and a model or a person currently decides, such as whether a diff meets a criterion, stays in scope, follows conventions, or whether a trace followed its plan. TypeSafe's design guidance is the rule: keep control flow, deterministic rules, and side effects in code [1].

Look hard at the judged rows for checks that should be deterministic. A judged check that asks a model to count files, compare a number to a limit, or order dates is a check that should never have been judged, because TypeSafe lists counting, numeric comparison, and dates among the model's weaknesses [2]. Rewrite those as deterministic checks now, before any of them reaches the new grader.

Done when: every row has one mark, every deterministic row stays exactly as it is, and no judged row depends on a count, a numeric comparison, or a date ordering. The deterministic list is closed for the rest of the migration; what to check in an eval suite is the reference for what belongs there.

Step 3. Rewrite each judged check as questions

For each judged row, write the questions Jev will be asked: Noul statements for acceptance criteria following writing Noul checks for acceptance criteria, Score rubrics with worded levels following turning a rubric into Score questions, and a composite in code following composite scoring for code quality where the old check was one broad question. Write down, per question, exactly what goes in the state.

The rules that every question must meet: positive, one claim, one observable behaviour, no number, no negation, evidence named in the statement and present in the state, state limited to what the question refers to. Every question gets an identifier that will appear in logs and failure messages.

Done when: every judged row maps to one or more questions with identifiers, every question passes the rules above on a read by a second engineer, and the state for each question is specified and fits within 32k tokens with the longest question [3]. The questions are written from the spec, as every eval at Reveneau is, and a question that was written by reading the current code is rewritten.

Step 4. Label a calibration set

Collect 50 to 200 recent changes with a human label per question, following calibrating Jev against your own human labels. Include every production failure in the period, every change the old judge escalated, a share of plainly fine changes, and a fixed known-bad set that must always fail: a planted approving comment in the diff, a moved count, a date comparison, a real criterion failure.

Done when: the set has at least 50 cases, every case has a label per question from a person who read the change against the criterion with a written reason, no case has unresolved disagreement between reviewers, and the known-bad set is stored separately and marked as fixed.

Step 5. Build the stage and log everything

Add the Jev stage to the pipeline after the deterministic checks, as running Jev-graded evals on every pull request describes: one request per change carrying every question, retries on 429 and 529 with a delay that doubles each time, treats any other failure as no grade and no merge, and a full log record per request with the model version, a hash of the state (a short fingerprint of its exact contents), every probability and confidence number, and the latency. The API key lives in the secret store. The model version is pinned; jev-1.13.0 is the current release, and jev-latest is an alias that can move [3].

Done when: the stage runs on every change, produces a complete log record every time, and has no ability to affect the merge yet. Its outcome is recorded and ignored.

Step 6. Shadow-run both graders

Run the old judge and the new stage on every change for a period long enough to cover the calibration set's variety, with only the old judge deciding. Meanwhile, run every question on the calibration set 10 times against the pinned version and record agreement with the labels, false passes and false fails at candidate thresholds, variance per case, and escalation rate per candidate band. This is the measurement LangChain made on five cases and called observational rather than evidence [4]; on your own set it is the evidence.

Done when: the calibration run is complete and logged, the shadow period has covered at least the kinds of change in the calibration set, and every question has its four numbers.

Step 7. Compare

Per question, compare the new grader's agreement with the human labels against the old judge's agreement on the same cases, if the old judge's decisions on them were kept. Compare variance: the new grader should be far more stable between runs, and a question where it is not has an ambiguous statement or an overloaded state. Compare escalation rate against what the team can sustain. Read every false pass on the known-bad set by hand.

Done when: every question is in one of three states: ready (agreement acceptable for its consequence, variance low, known-bad set fails), rewrite (agreement or variance not acceptable, back to step 3 for that question), or keep-old (the check needs a written reason or open-ended reading and stays with the language-model judge or a person, following what a Jev grade cannot tell you). No question is in a fourth state.

Step 8. Set thresholds in a reviewed file

For every ready question, set the pass line, the fail line, and the middle band from the calibration numbers, by consequence, following thresholds, confidence and escalation: a money-moving, permission, deletion, or authentication check gets a high pass line and a wide band; a convention check gets a lower line and a narrow band. Set the composite weights and the gate list. Put all of it in a file in the repository with a comment on each number naming the calibration run: date, model version, set size, agreement at the chosen lines. Start from TypeSafe's published bands of 0.9 and 0.5 [5] and move only on the strength of the measurement.

Done when: the file is merged through review, every number in it has a comment naming its run, every correctness and security question is a gate, and the pipeline reads the file rather than an environment variable or a dashboard.

Step 9. Switch

Turn the new stage's outcome from recorded to blocking for the ready questions, and turn the old judge off for those questions. Keep the old judge, advisory, for the keep-old questions. Announce the change to the team with the failure message format, which is the question text and the probability, so nobody looks for a paragraph that will not arrive.

Done when: a change that fails a ready question is blocked by the pipeline, a change in the middle band is held for a person, the old judge no longer decides any ready question, and the first week of blocked changes has been read by a person to confirm the blocks were right.

Step 10. Keep the human band, and keep measuring

The uncertain band goes to a person on every change, the label and reason are stored with the log record, and the labelled set grows from those labels. The full calibration run repeats on every model version change before the new version blocks, and monthly against the stored labels. The known-bad set runs on every change to a question or a threshold. Escalation count is tracked as a health signal: it should fall as calibration narrows the bands and it should never be zero.

Done when: the monthly re-check is scheduled and its first run has happened, the version-change re-check is wired to the model version field in the logs, and the escalation count for the last month is a number someone can quote. This step stays open for as long as the suite runs.

What the finished migration looks like

The deterministic checks are byte-for-byte what they were. Every judged check is one or more questions with identifiers, in a reviewed file, written from the spec. Every threshold and weight names the run that set it. The model version is pinned and logged. The uncertain band goes to a person whose label is kept. The old judge, where it survives, is advisory and scoped to the checks that need a written reason. And the run time is whatever your measurement says it is; ours is ten times faster than it was, on our own suite, and yours is a number you should measure on the same changes before and after rather than take from us.

The step teams most want to skip is the shadow period, because by then the questions are written and the stage works. It is the step that turns the migration from a guess into a measurement, and the eval-driven development guide exists to make the same point about every check: a check you have not seen fail on a known-bad case is a check you do not know works. The same applies to the grader. Finish the checklist. Then let it block.