Versions, drift and monitoring a decision model

On the day this guide was written, TypeSafe's models page listed one model, jev-1.13.0, and two aliases, jev-latest and jev-preview, both pointing to it [1]. The SDK, TypeSafe's client library, defaults to the alias jev-latest [2]. So a team that installs the SDK, writes a few questions and ships is running on an alias, and TypeSafe's own page says what that means: "An alias moves when a new release ships, so the answers behind it can change without a change on your side" [1].

Everything on this page follows from that sentence. A decision model is a dependency whose behaviour can change underneath you, and the routine for a dependency like that is the same as for any other: pin it, test the new version before you take it, and watch production for the changes you did not expect.

Pin the version, read the model field

Production code should name the versioned ID, jev-1.13.0, rather than an alias. The aliases are for development, where you want the newest behaviour, and for the re-run described below, where you want to test it. TypeSafe's models page says the release date of each version is available from the /v1/models endpoint [1], which is also how a monitor can learn that a new version exists before anyone moves the pin.

Every response from the API (the web service your code calls) carries a model field that reports "the versioned ID that answered" [1][3]. Log it on every decision, next to the question ID, the probabilities and the action. That one field is what lets you match a change in behaviour to a release, and it is the check that the pin is actually in effect: a log that shows a version you did not pin means a code path is still on the alias.

Keep a labelled set, and re-run it on every change

A labelled set is a collection of real states with the answer a person agreed is right. It is the only confirmed answers a decision model produces, and every page in this guide has said where the labels come from: the human path in confidence-gated routing, the confirmation step in gating tool calls, the review queue in guardrails, the curator queue in extraction and verification. Each of those produces a state, the model's probabilities, and a person's decision. Keep all three.

The set is re-run in three situations. Before moving the version pin: run the set against the new version, compare each answer's probability with the recorded one, and look at the cases that crossed a threshold. Before changing a question's wording or criteria: the jaggedness page says the model answers the question you wrote, and a rewording is a new question [4]. Before changing a threshold: the set says how many past decisions would have moved.

The comparison is per case and per threshold rather than an average. A version that moves the mean jailbreak probability by 0.01 can still move forty cases across the 0.35 review threshold. Report the number of cases that changed route, list them, and have a person read them before the pin moves. This is the same discipline as running an eval suite before a deployment, and evals in CI for coding agents describes the pipeline shape, CI being the automated checks that run on every change.

TypeSafe's own cookbooks are dated and versioned for this reason: the guardrail numbers came from jev-1.12 on 2026-08-15 [5], the entity alignment run from jev-1.12 on 2026-08-11 [6]. When you read a threshold on a vendor page, check which version produced it, because the current model is a later one.

What drift looks like in a decision model

Drift is a change in the relationship between inputs and answers over time. With a decision model it has three sources, and the monitoring has to tell them apart.

Model drift: the version changed, or the pin was not in effect. The model field in the log answers this in one query.

Input drift: the traffic changed. A new customer segment, a new language, a product launch that changes what people ask about, a marketing campaign that brings a different kind of message. The model is the same and the answers are different because the questions are, in effect, being asked about different things.

Question drift: someone changed a question, a criterion or an option description, and the labelled set was not re-run. Version the question definitions in the same repository as the code, so a change is a reviewed diff.

The signal that catches all three is the probability distribution. For each question, track weekly: the share of answers in each confidence band (below 0.5, 0.5 to 0.9, 0.9 and above); for a Choice, the share per option and the none rate; for a Noul, the share above the action line, in the review band, and below; for a Score, the mean and the share at or above each gate. A histogram per question per week, compared with the previous four, shows a shift before the review queue does. TypeSafe's self-consistency cookbook reports a mean per-question standard deviation of 0.0102 across 15 repeated calls on the same input [7], which is TypeSafe's own measurement on one claim, and it means that on a stable model with stable input the distribution should barely move; if it moves, something upstream did.

Two derived rates matter most because they cost money and attention. The review-queue rate (the share of decisions that went to a person) rising with flat traffic means either the model got less confident or the inputs got harder. The default-used rate from rate limits, retries and latency rising means the model is not answering at all.

Calibration, checked monthly

Beyond the distribution, check whether the probabilities mean what they say. Bucket the month's human-reviewed decisions by the model's probability (0.3 to 0.4, 0.4 to 0.5, and so on) and compute the share in each bucket where the person agreed with the model's answer. On a calibrated model the 0.8 bucket agrees about 80 percent of the time. If the 0.8 bucket agrees 60 percent of the time, the model is overconfident on your inputs and the act thresholds should move up; if it agrees 95 percent of the time, they can come down. TypeSafe describes the training method as Reinforcement Learning for Calibrated Decisions [8], and calibration on your inputs is the thing that method promises and that only your data can confirm.

This check is only as good as the review sample, which is biased towards the uncertain band by design. Add a small random sample of high-confidence decisions to review each month, so the 0.9 and above bucket has data too.

Language

TypeSafe's models page says English is the primary training language and where accuracy is currently best, with other languages, including Chinese, Japanese and Korean scripts, supported but less reliable, and it recommends testing on your own content [1]. For monitoring that means two things. Tag each decision with the detected language of the untrusted text, and track the distribution and the calibration per language. And treat a change in the language mix as input drift: a product that was 95 percent English and is now 70 percent has a different model performance even though nothing else changed. A labelled set per language you serve is the minimum before the thresholds in that language are trusted.

What to alert on

Alerts should be few and each should name an action. Ours, per integration:

A model field in the log that is not the pinned version. Action: find the code path on the alias.

A new version listed at /v1/models. Action: schedule the labelled-set re-run.

The review-queue rate for any question above its weekly baseline by a stated margin. Action: read a sample from the queue and check the histogram.

The none rate on a router rising. Action: check whether a new kind of request needs a handler.

The default-used rate rising. Action: check the 429, 529 and timeout rates.

The monthly calibration check moving a bucket by more than a stated amount. Action: revisit the thresholds for that question.

Everything else is a dashboard rather than an alert. A histogram per question per week, the model version, the language mix, and the count of labels added, on one page, read at the same time each week.

Where Reveneau fits

Reveneau runs this routine on the decision points in the products it builds and on the grader in its own eval suite. The suite's Noul and Score questions are pinned to a version, the labels a person adds in the uncertain band are kept to re-check the grader, and the labelled set is re-run before the pin moves. On our own suite, measured against our previous grader, the run is ten times faster with Jev grading, and the re-run is what lets us keep that speed without trusting a version we have not checked. Evals with Jev explains the grader; when evals give false confidence is the warning about a check that passes for the wrong reason, which is what an unpinned grader can become. For the definitions behind every term on this page, Jev and System One models is the place to start, and decision models for product teams is the version for the person who has to sign off on running one.