Should your product use a decision model? A guide for founders and CTOs / Govern

Govern

Measuring a decision model in production

Six numbers tell you whether a decision model is still doing its job: agreement with a fresh sample of human labels, the escalation rate, the rate at which a person reverses the model's action, latency at the median and the 95th percentile, cost per decision, and drift in the shape of the probabilities over time. Re-label a sample monthly and after every model version change. A seventh number, repeat consistency, matters for any decision used as a gate, and LangChain's five-case test is the only published measurement of it so far, which is why you measure it on your own cases.

Published September 22, 2026. Editorial.

Key takeaways

Agreement is measured against fresh human labels on a sample of live traffic, on a schedule. The pilot's set goes stale.
Escalation rate and reversal rate move in opposite directions when a threshold is wrong. Read them together.
Latency is reported at the median and the 95th percentile. Openlayer measured 244 and 371 milliseconds on its own benchmark; yours will differ.
Drift in the probability distribution is the earliest warning, because it shows before agreement drops.
Repeat consistency is the measure that makes a decision safe to use as a gate. Measure it on your own cases and after every version change.

A decision model in production is a feature that gives the same kind of answer thousands of times a day, and the only way to know whether those answers are still good is to check a sample of them against a person, on a schedule, and to watch a few numbers between checks. This page lists the numbers, says how to get each, and gives the schedule. It is written for the person who owns the feature after launch, and it assumes the audit record from auditability when the model gives no written reason is being written for every decision, because every metric below is computed from it.

1. Agreement with sampled human labels

What it is: the share of decisions where the model's answer, at the threshold in use, matches the label a person gave to the same case, on a sample drawn from live traffic after launch.

How to get it: each month, draw a sample of 100 to 200 decisions across the period, stratified so that the uncertain band and each action are represented, and have a person label them blind. Compute agreement per action, and report the human agreement rate from a second labeller on a subset beside it, as in the two-week pilot. The pilot's labelled set is the first calibration set, and it goes stale, because the traffic a feature sees in month six is different from the traffic in the pilot. The monthly sample is what keeps the measurement honest.

What moves it: a model version change, a change in the questions, a change in the traffic. The record's version field tells you which.

2. Escalation rate

What it is: the share of decisions that fell below the action threshold and went to a person or a fallback.

How to get it: count the escalated actions in the record. Report it per action, because a routing decision at a 0.7 threshold and a spam close at 0.95 will escalate at different rates by design.

What it tells you: a rising escalation rate with steady agreement means the traffic got harder or the model got less confident. A falling escalation rate with falling agreement means the model got confident about the wrong answers, which is the worse case and the one drift catches first. Raising a threshold raises agreement by sending more cases to people; the escalation rate is the cost of that, and it should always be reported next to the agreement it bought.

3. Action-reversal rate

What it is: the share of automatic decisions that a person later undid. A ticket routed to billing and moved by an agent to technical. A post blocked and restored on appeal. A document marked complete and sent back.

How to get it: join the record's action field to whatever later system records the correction. This is the one metric that measures the model against reality rather than against a labeller, and it is the one a product owner should read first, because a reversal is a user or a colleague telling you the answer was wrong.

What it tells you: reversal rate per action is the truest agreement number you have, on the cases where a wrong answer had a consequence. If it rises while sampled agreement holds, the sample is not representative of the cases that matter.

4. Latency at the median and the 95th percentile

What it is: the time from sending the request to receiving the answer, reported at two points. The median is the time half of requests beat. The 95th percentile is the time 95 of every 100 requests beat, and it is the number a user waiting on the decision feels.

How to get it: timestamp the request and the response in the record. Openlayer measured a median of 244 milliseconds and a 95th percentile of 371 milliseconds on its own benchmark [1]; TypeSafe reports 70 to 500 milliseconds end to end [2]; LangChain measured 0.44 seconds per call [3]. Yours will differ with state size and network, and the specification's cutoff is the one that matters.

What it tells you: a rising 95th percentile with a steady median means a growing tail, which is usually the state growing. TypeSafe's jaggedness page says accuracy also falls as the state grows with content unrelated to the decision [4], so a latency tail is often an accuracy warning as well.

5. Cost per decision

What it is: total spend divided by decision count, per month, per action.

How to get it: tokens sent per decision, from the record, multiplied by the list price of $0.042 per million input tokens with output free [5], and reconciled against the vendor bill. Report it beside the pilot figure and beside the cutoff in the specification.

What it tells you: a rising cost per decision with a steady decision count means the state grew. That is a product change somebody made, and it should be findable. A price change from the vendor shows here first, which is why the contract clause in early-access risk and vendor lock-in exists.

6. Drift in the probability distribution

What it is: a change over time in the shape of the probabilities the model returns, before any change in the actions it takes.

How to get it: from the record's stored distributions, plot per week the share of decisions in each confidence band (above 0.9, 0.5 to 0.9, below 0.5, following TypeSafe's published bands [6]) and the mean probability of the chosen option. Compare each week to the launch week.

What it tells you: drift is the earliest warning, because a model can become less sure, or sure in a different way, weeks before agreement on the monthly sample drops. A shift in the bands with no version change means the traffic changed. A shift on the day of a version change means the version did. Either one triggers an out-of-schedule labelling sample.

7. Repeat consistency, for any decision used as a gate

A gate is a check that must pass before an action runs: a guardrail before a message posts, a tool-call check before an agent acts, a grade before a code change merges. A gate that gives different answers to the same input on different runs is a gate that sometimes lets through what it blocked yesterday, and no threshold fixes that.

What is published: LangChain ran five agent cases through each judge 100 times and measured the variance of the quality score across those repetitions. Jev's mean per-case variance was 0.0000149; the three language models were 92, 433 and 913 times higher [3]. TypeSafe's own batching cookbook reports a standard deviation of 0.0 across five repeats for 11 of 13 questions on jev-1.12 [7]. LangChain calls its result "promising, but early", says it is "observational, not evidence", and says the experiment "cannot tell us why Jev's scores varied less" [3]. Five cases is a small set and the authors say so.

How to get yours: once a day, send the same 20 cases from the calibration set and record the probabilities. The variance across days on a pinned version is your repeat consistency. It should be close to zero. If it is not, write it down and lower the confidence you place in the gate. Reveneau runs this on the checks its eval suite grades with Jev, because a grader that passed a change on Monday and fails the same change on Tuesday is a grader nobody trusts; the suite runs ten times faster with Jev than it did with a language model grading it, on our own suite, and the consistency check is part of why the speed is usable.

The re-labelling schedule

Monthly: a sample of 100 to 200 live decisions, labelled blind, agreement and escalation per action computed and recorded. Escalated cases with their human labels join the calibration set.

On every model version change: the full calibration set run against the new version before the move, agreement and escalation compared with the current version, the comparison recorded, and the move made on a named person's decision. TypeSafe lists jev-1.13.0 as the current version with the aliases pointing to it [5], so at writing no such change has happened; the procedure is written so the first one is measured.

On a drift trigger: an out-of-schedule sample of 50 to 100 decisions from the period where the bands moved, labelled within the week.

Quarterly: a review of the thresholds against the accumulated agreement and reversal data, with any change recorded in the specification with a date.

What to put on one page

One table, per action, updated monthly: agreement, human agreement rate, escalation rate, reversal rate, median latency, 95th percentile latency, cost per decision, share in each confidence band, repeat variance, model version. Ten columns. A product owner who reads that table monthly knows whether the feature is working and can name the month it stopped. How to measure whether AI is actually helping covers the outcome metrics above this table, and metrics for AI code quality is the equivalent table for the eval suite.

Between the monthly tables, when any of the signals on when to move back to a language model or to rules appears in them, that page has the decision. The pillar guide has the rest of the sequence. Six numbers, one table, one owner. A decision model that is measured this way can be trusted for exactly as long as the table says it can.

Best for

Any decision-model feature past its first month in production
Gates, meaning checks that must pass before an action runs, where repeat consistency decides trust
Teams with an audit record that stores the full probability distribution per decision

Avoid if

Do not rely on the pilot's labelled set as the ongoing measure; it goes stale within months
Do not report agreement without the escalation rate it was bought with
Do not adopt a new model version without running the calibration set against it first

Check before you decide

Confirm the monthly sample is drawn from live traffic and labelled blind
Confirm reversal data is joined to the decision record per action
Confirm the daily repeat check runs on a pinned version and its variance is recorded

Common questions

What should we measure once a decision model is live?

Six numbers per action, monthly: agreement with a fresh blind-labelled sample of live decisions, escalation rate, the rate at which people reverse the model's action, latency at the median and 95th percentile, cost per decision, and the share of decisions in each confidence band, which shows drift. Add a seventh, repeat variance on a fixed set of cases, for any decision used as a gate. All seven come from the audit record.

Why does the pilot's labelled set go stale?

Because the traffic a feature sees in month six differs from the traffic in the pilot: new products, new customer types, new ways of writing the same request. Agreement on the old set stays high while agreement on live traffic falls. Draw 100 to 200 live decisions each month, label them blind, and let escalated cases with their human labels join the calibration set, so the set follows the traffic.

How do escalation rate and agreement relate?

Raising the action threshold sends more cases to people, which raises agreement on the automatic decisions at no cost in accuracy and raises the escalation rate as the cost. Report them together per action. Rising escalation with steady agreement means harder traffic or a less confident model. Falling escalation with falling agreement means the model became confident about wrong answers, which is the worse case and the one drift monitoring catches first.

What is the action-reversal rate and why read it first?

The share of automatic decisions a person later undid: a ticket re-routed, a blocked post restored on appeal, a document sent back. It comes from joining the decision record to the system that records the correction. It measures the model against reality rather than against a labeller, on the cases where a wrong answer had a consequence, so a product owner reads it before the sampled agreement rate.

What latency figures have been published for Jev?

Openlayer measured a median of 244 milliseconds and a 95th percentile of 371 milliseconds on its own benchmark. TypeSafe reports 70 to 500 milliseconds end to end. LangChain measured 0.44 seconds per call against 2.16 to 2.83 seconds for the language models in its test. Your figures will differ with state size and network; report the median and the 95th percentile from your own record against the cutoff in your specification.

What is drift and how do we see it early?

A change in the shape of the probabilities the model returns before any change in the actions it takes. Plot weekly the share of decisions in each confidence band, above 0.9, 0.5 to 0.9, and below 0.5, following TypeSafe's published bands, and the mean probability of the chosen option, against the launch week. A shift with no version change means the traffic moved; a shift on the day of a version change means the version did. Either triggers an extra labelling sample.

Why does repeat consistency matter for a gate?

A gate is a check that must pass before an action runs. If it gives different answers to the same input on different runs, it sometimes lets through what it blocked the day before, and no threshold fixes that. LangChain measured Jev's mean per-case variance at 0.0000149 across 100 repetitions on five cases, with language models 92 to 913 times higher, and called the result observational. Measure yours daily on 20 fixed cases.

How often should we re-label?

Monthly, a blind sample of 100 to 200 live decisions. On every model version change, the full calibration set against the new version before the move, with the comparison recorded. On a drift trigger, 50 to 100 decisions from the period where the bands moved, labelled within the week. Quarterly, a review of thresholds against accumulated agreement and reversal data, with changes dated in the specification.

What happens on a model version change?

The new version is run against the full calibration set before anything moves. Agreement and escalation rate per action are compared with the current version's, the comparison is recorded, and a named person decides. TypeSafe lists jev-1.13.0 as the only version at writing, with jev-latest and jev-preview pointing to it, so the procedure has not yet been exercised; it is written so the first change is measured rather than noticed.

How does cost per decision change in production?

It moves when the state grows, when the decision count changes, or when the vendor changes the price. Compute it monthly from tokens sent per decision times $0.042 per million input tokens, output free, and reconcile against the bill. A rising cost with a steady count means the state grew, which is a product change someone made. A price change shows here first, which is what the contract's notice clause is for.

How does Reveneau measure its own grader?

With the same checks on a smaller scale: human labels from the uncertain band are kept and re-run against the grader, a fixed set of cases is re-graded to watch repeat variance, and a version change is a change like any other that runs the labelled cases before adoption. We publish one figure from this: the suite runs ten times faster with Jev grading it than with a language model, on our own suite. We publish no agreement rate.

References

More in Govern

Auditability when the model gives no written reason

A Jev answer is a probability with no paragraph explaining it, because the model is not trained to generate text. That removes one thing from the audit record and leaves everything else: the state the model saw, the questions it was asked, every probability it returned, the threshold applied, the action taken, the model version, and the human label if the case was escalated. For most product decisions that record is more complete than what a language-model feature keeps today. Where a policy or a regulation requires a written reason, the decision layer must hand off to a person or to a language model, and this page says how.

When to move back to a language model or to rules

Five signals say a feature should move back from a decision model: the option set keeps changing, users need a reason, agreement drops after a version change and does not recover, the task turned out to need counting or dates, or English-only became a problem. A decision model is one tool among rules, language models and people, and none of the five is a failure of judgment; each is a task turning out to be a different shape than it looked. With the questions, thresholds and labelled set in your repository, the move is about a week of work. Without them it is a rebuild.

Previous: Auditability when the model gives no written reason Next: When to move back to a language model or to rules