Should your product use a decision model? A guide for founders and CTOs / Plan

Plan

A two-week pilot for a decision model

This is the pilot we would run before putting a decision model into a product, written day by day with what done means on each day. Week one picks one decision, writes the questions, labels 100 to 200 real cases, and runs the model in shadow mode, meaning it answers every case and nothing acts on its answer. Week two compares those answers against the current method on agreement, latency and cost, and decides against cutoffs written before the run. It is a plan, stated plainly as one. We have not published a result from it.

Published September 22, 2026. Editorial.

Key takeaways

Pick one decision that passes the four-part sorting rule, with the highest volume and the lowest consequence of a wrong answer.
Write the pass marks on day one, before any result exists: the agreement rate, the latency at the 95th percentile, and the cost per decision the feature needs.
Label 100 to 200 real cases by hand, two labellers on at least 30 of them, so you know how much two people disagree before you measure the model against either.
Run in shadow mode for at least three working days so the sample includes the traffic you did not think of.
Decide on the numbers against the written cutoffs. A pilot that ends in a debate had no cutoffs.

A pilot has one job: to produce three numbers you can compare against three cutoffs you wrote down before you had the numbers. Everything below is arranged to make that happen in ten working days. If a day's "done" is not met, the next day waits. That rule is what keeps a two-week pilot from becoming a two-month one.

We say this once and plainly. Reveneau runs pilots of this shape when a client's feature passes the sorting rule, and we grade our own eval suite with Jev, which runs ten times faster than it did with a language model as the grader, on our own suite. We have not published a result from this pilot plan, so nothing below is a measured outcome. It is the plan.

Before day one: pick the decision

Choose one decision, using which features are System One tasks. Among the candidates that pass all four tests, pick the one with the highest daily volume and the lowest consequence of a wrong answer. Support ticket routing is the usual pick: high volume, a wrong route costs a delay rather than money, and someone already does it today, so there is a current method to compare against.

Write the decision in one sentence and the answer set as a list. "Given the text of a new support ticket, which of these six queues should it go to." That sentence is the pilot's scope. Anything that is not that sentence is out.

Done means: one sentence, one list, and the name of the current method (a person, a rule, a language model) written at the top of a shared document.

Week one

Day 1: cutoffs and roles. Write the three pass marks. Agreement: the share of labelled cases where the model's answer, at the threshold you will use, matches the human label; pick the number the feature needs, and if the current method's agreement with a second human is known, use that as the floor. Latency: the 95th percentile, meaning the time within which 95 of every 100 requests return, that the feature can tolerate; for a decision inside a user request that is usually under one second. Cost: the price per decision above which the change is not worth making, from the cost arithmetic. Name the owner of the labelled set and the person who will make the go or no-go call on day 10. Done means: three numbers and two names in the document, dated.

Day 2: write the questions. Turn the decision into Jev's shapes: a Choice with the queues as options, or several Nouls if the routing is several yes-or-no facts combined in code. Follow TypeSafe's own guidance: keep control flow and deterministic rules in code, split broad judgments into atomic questions asked together, and put every fact the decision needs into the state rather than relying on model knowledge [1]. Write the questions as plain text in your repository, with the version of the model they were written against, jev-1.13.0 at writing [2]. Done means: the questions file is in the repository and a second person has read it and found no question whose answer depends on something not in the state.

Day 3: label the set, first pass. Pull 100 to 200 real cases from the last month, sampled across the day and the week rather than from one afternoon. One person labels each with the answer the current method should have given, and records the label, the labeller and the date. Done means: every case has one label, and the set is in the repository beside the questions.

Day 4: label the set, second pass. A second person labels at least 30 of the same cases without seeing the first labels. Count the disagreements. That count is your human agreement rate, and no model can be expected to beat it against either labeller. If it is below the agreement cutoff you wrote on day 1, the cutoff was wrong or the question is, and you fix that before continuing. Done means: the two-labeller agreement rate is in the document.

Day 5: shadow mode on. Wire the model into the path where the decision is made, so that every real case is sent to it and its answer is recorded with the state, the probabilities, the confidence number, the model version and the timestamp, and nothing acts on the answer. The current method keeps deciding. Set the confidence threshold you plan to use, starting from TypeSafe's published guidance of 0.9 and above for acting automatically and below 0.5 for routing to a person [3], and record which band each answer fell in. Done means: the first hundred shadow answers are in a table with every field filled.

Week two

Days 6 to 8: let it run. Three working days of shadow traffic, so the sample includes the Monday queue, the quiet Wednesday afternoon and whatever arrives that nobody planned for. While it runs, do two things. Run the labelled set through the model once a day and record agreement each time; if it moves between runs on the same version, you have found inconsistency and you write it down. LangChain measured repeat consistency on five cases across 100 repetitions per judge and found Jev's mean per-case variance on the quality score to be 0.0000149 [4]; you are measuring the same thing on your own cases. And read 20 shadow answers a day by hand, chosen from the uncertain band, to see what kind of case lands there. Done means: three days of shadow answers, three daily runs of the labelled set, and 60 hand-read cases with notes.

Day 9: compute the three numbers. Agreement on the labelled set at the chosen threshold, counting an escalation to a person as neither right nor wrong but as an escalation, and reporting the escalation rate beside the agreement rate. Latency at the median and the 95th percentile from the shadow log. Cost per decision from the tokens sent and the list price, $0.042 per million input tokens with output free [2], and from the bill if the vendor console shows one. Compare each against the day 1 cutoff. Then compute one more number the cutoffs did not ask for: agreement on the shadow traffic between the model and the current method, on a sample a person has checked, because the labelled set was chosen and the shadow traffic was not. Done means: a one-page table with the three numbers, the escalation rate, the shadow agreement, and the cutoff beside each.

Day 10: decide. The person named on day 1 reads the table and makes one of three calls. Go: all three numbers meet the cutoffs, and the feature moves to the specification with the pilot's questions, threshold and labelled set as its first inputs. Retune: agreement missed the cutoff and the hand-read cases show why, so the questions are rewritten and week one's days 2 to 5 run again, once. No go: agreement missed and the hand-read cases show the task needs something the model does not do, such as counting or reading dates [5], or latency or cost missed with no fix identified, and the feature stays with the current method. Done means: the call, the reason, and the date, at the bottom of the document.

What the pilot does not tell you

It does not tell you how the model behaves on traffic you did not sample, which is why the shadow period is three days and not three hours. It does not tell you what the next model version will do, which is why the labelled set stays in the repository and is re-run on every version change; measuring a decision model in production sets the schedule. And it does not tell you whether the vendor's price will hold, which is a contract question covered in early-access risk and vendor lock-in.

Common ways this pilot goes wrong

Labelling from the model's answers. If the labeller sees the model's answer before writing the label, the agreement rate measures persuasion rather than accuracy. Label first, blind.

Sampling one afternoon. A hundred tickets from a single afternoon share a shape. Sample across the period the feature will serve.

Skipping the second labeller. Without a human agreement rate, an 88 percent model agreement rate cannot be judged, because two people might agree at 85 percent on the same set.

Moving the cutoff on day 9. The cutoffs exist to be written before the numbers. If they were wrong, say so and record the new ones, dated, so the change is visible.

Testing the alias. Point the pilot at the pinned version, jev-1.13.0, and record it, so that a version change during the pilot cannot silently move the numbers [2].

The pillar guide has the sequence this pilot sits in. How to structure a pilot before a full engagement is the general version for any AI feature, and how to evaluate an AI agent covers the case where the decision sits inside an agent. Ten days, three numbers, one call. Print this page and fill it in.

Best for

One high-volume, low-consequence decision with a current method to compare against
Teams that can spare one labeller for two days and a second for half a day
Features where shadow mode can be wired in without touching the current path

Avoid if

Do not pilot on a decision where a wrong answer costs money or a customer before the pilot is done
Do not start the pilot without the three cutoffs written and dated
Do not let the labeller see the model's answer before writing the label

Check before you decide

Confirm the two-labeller agreement rate is recorded before the model's agreement rate is computed
Confirm the shadow log stores state, probabilities, confidence, model version and timestamp for every case
Confirm the labelled set ran against the pinned version and not the alias

Common questions

Which decision should the pilot use?

The one that passes all four tests of the sorting rule and has the highest daily volume with the lowest consequence of a wrong answer. Support ticket routing usually wins: hundreds of tickets a day, a wrong route costs a delay rather than money, and a person or rule already routes them, so a current method exists to compare against. Write it as one sentence with the answer set as a list before day one.

How many cases should we label?

One hundred to two hundred real cases, sampled across the day and the week the feature will serve rather than from one afternoon. One person labels all of them blind, and a second person labels at least 30 of the same cases without seeing the first labels. The disagreement count between the two people is your human agreement rate, and it is the floor against which the model's agreement rate is judged.

What is shadow mode and why run it for three days?

Shadow mode sends every real case to the model and records its answer, probabilities, confidence, model version and timestamp while the current method keeps deciding and nothing acts on the model's answer. Three working days are the minimum because a single afternoon's traffic shares a shape, and the cases you did not think to sample are the ones that break a threshold. Read 20 uncertain-band answers a day by hand while it runs.

What cutoffs should we write on day one?

Three. Agreement: the share of labelled cases the model gets right at the threshold you will use, with the two-labeller human rate as the floor. Latency: the 95th percentile the feature can tolerate, usually under one second for a decision inside a user request. Cost: the price per decision above which the change is not worth making, from the cost arithmetic page. Write them dated, before any result exists, and change them only in writing.

What confidence threshold should the pilot use?

Start from TypeSafe's published guidance: 0.9 and above to act automatically, below 0.5 to route to a person, the middle band to proceed with caution, and different thresholds for actions with different consequences. Record which band every shadow answer fell in. On day 9, report agreement at the threshold you will use and the escalation rate beside it, because raising the threshold raises agreement by sending more cases to people.

How do we measure cost per decision in the pilot?

Two ways, and record both. From the tokens sent per decision multiplied by the list price of $0.042 per million input tokens with output free, which gives a figure with no vendor benchmark in it. And from the vendor console bill divided by the decision count, if the console shows one. The pilot figure replaces every published comparison in your planning, because it carries your state size and question count.

What does a go decision lead to?

The specification. The pilot's questions file, its threshold per band, and its labelled set become the first three inputs to the specification for the production feature, which also has to state the escalation path, the fallback when the API is down, the audit record, the owner of the labelled set, and the model version policy. Reveneau writes its eval suite from that specification, so every threshold becomes a check that runs on every change.

What does a retune look like?

Agreement missed the cutoff and the 60 hand-read uncertain cases show why: a question that reads a negation literally, a fact missing from the state, or a broad question that should be several atomic ones. Rewrite the questions following TypeSafe's guidance, rerun days 2 to 5 once, and recompute on day 9. One retune is allowed. A second means the task or the cutoff is wrong, and the pilot ends with a no-go and a reason.

Why re-run the labelled set daily during the shadow period?

To measure repeat consistency on your own cases. A decision used as a gate, a check that must pass before an action runs, has to give the same answer to the same input. LangChain measured Jev's mean per-case variance at 0.0000149 across 100 repetitions on five cases, but that is their cases and a small set. If agreement moves between daily runs on the same pinned version, write it down; it changes what threshold you can trust.

Has Reveneau published a result from this pilot?

No. This is the plan we run when a client's feature passes the sorting rule, and it is written as a plan. The one first-party figure we publish about Jev is that our own eval suite, which Jev now grades, runs ten times faster than it did with a language model grading it, measured on our own suite. That figure is about run time, and it says nothing about agreement, cost per decision or any client's result.

What if the current method is a language model?

Then the comparison is direct. Run both against the labelled set at their chosen thresholds, report agreement for each, and report latency and cost per decision side by side from the same shadow period. Split any language-model call that both decides and drafts, and compare only the deciding half. If the language model's agreement is higher and the gap matters more than the cost and latency difference, the answer is no-go, recorded with the numbers.

References

More in Plan

Writing a specification for a decision-model feature

A specification for a decision-model feature has to state eight things: the decision in one sentence, the allowed answers as a closed list, a threshold per consequence, the escalation path when confidence is low, the fallback when the API is down, the fields in the audit record, the calibration set and its owner, and the policy for model version changes. Each of the eight becomes an automated check. Reveneau writes its eval suite from the specification before the code exists, and a decision-model feature is where that practice matters most, because the behaviour that matters lives at the thresholds.

Previous: Early-access risk and vendor lock-in Next: Writing a specification for a decision-model feature