Writing a specification for a decision-model feature

A specification for a language-model feature spends most of its words on the prompt and the shape of the output. A specification for a decision-model feature spends most of its words on what happens at the edges: the confidence at which the product acts, the confidence at which it asks a person, and what it does when the model is not there at all. That shift is why the eight sections below are in the order they are.

At Reveneau every change has to pass an eval suite written from the specification before it is released, and the suite is now graded by Jev, which runs ten times faster than it did with a language model grading it, on our own suite. The specification is where those evals come from. A section that cannot be turned into a check is a section that is not finished, and each of the eight below ends with the check it produces.

1. The decision, in one sentence

"Given the text of a new support ticket, decide which of the six queues it goes to." One decision per specification. If the sentence has an "and" in it, there are two decisions and two specifications, because they will have different thresholds and different fallbacks. The sentence also names the input, and the input has to be text, since the model's state accepts no images or audio [1].

The check: an eval that asserts the feature sends exactly one request per decision and that the state contains only the fields the sentence names.

2. The allowed answers, as a closed list

The queues, the labels, the yes-or-no statements, or the levels of a scale described in words. This is the list a Choice question offers, up to 255 options, or the 2 to 10 levels a Score question carries, or the statement a Noul evaluates [1]. The list is closed: an answer outside it is a bug, and the specification says so. If the product needs an "other" outcome, it is on the list as an option, with its own threshold and its own route.

Write the questions in full, in the exact words the product will send, in this section. TypeSafe's jaggedness page says the model "answers the question you wrote, not the one you meant" and reads scoping words and negations literally [2], so the words are the behaviour. Follow the vendor's guidance to split broad judgments into atomic questions and keep the combining arithmetic in code [3], and write that arithmetic here too.

The check: an eval that runs every question against the calibration set and asserts every answer is on the list; an eval that asserts the combining arithmetic produces the expected outcome for a hand-written table of probability inputs.

3. A threshold per consequence

Every action the feature can take gets its own confidence threshold, in this section, as a number. TypeSafe's confidence page gives the starting guidance: 0.9 and above to act automatically, below 0.5 to route to a person or fall back, the middle band to proceed with caution, and the instruction that "different actions within the same system should be gated at different levels depending on the consequences" [4]. Routing a ticket to billing might act at 0.7. Closing a ticket as spam might act at 0.95, because the cost of a wrong close is a lost customer and the cost of a wrong route is a delay. Noul questions return one probability and no separate confidence number [1], so for those the threshold is on the probability itself, and the specification says which.

TypeSafe's guardrail cookbook shows the shape: a review threshold of 0.35 and an action threshold of 0.70 for a strict policy or 0.85 for a permissive one, with route precedence of support, then block, then review, then pass [5]. Copy the shape; your numbers come from the pilot's labelled set.

The check: an eval per threshold that feeds a case just above and just below it and asserts the right action; an eval that asserts the highest-consequence action is never taken below its threshold on any case in the calibration set.

4. The escalation path when confidence is low

Where a below-threshold case goes, who sees it, how long it can wait, and what happens to the label that person gives. The last part is the one teams leave out. The person's label is the most valuable data the feature produces, because it is a fresh human answer on exactly the cases the model found hard, and it is how the calibration set grows. The specification says the label is stored with the case and joins the calibration set at the next refresh.

The check: an eval that sends a low-confidence case and asserts it appears in the escalation queue within the stated time, with the state, the probabilities and the model version attached; an eval that asserts a label given in the queue is written back.

5. The fallback when the API is down

A sequence, in order, for each error the API documents: 401 for an invalid key, 422 for a validation failure, 429 for the rate limit, 529 for "TypeSafe is temporarily overloaded" [6]. Retry with exponential backoff, which the docs ask for, up to a stated count and a stated total wait. Then one of: ask a language model the same questions, apply rules, route to a person, or fail closed and tell the user. The order depends on the consequence of each action and the specification states it per action, because a moderation block might fail closed while a routing decision might fall to a default queue.

The check: an eval that cuts the API and asserts the fallback runs to completion within the stated time; an eval that asserts a 422 is logged as a bug rather than retried, because a validation failure will not fix itself.

6. The audit record

The fields written for every decision, whether or not it was escalated: the state as sent, the questions as sent, every probability returned, the confidence number where one exists, the threshold applied, the action taken, the model version, the timestamp, and the human label if the case was escalated. The record has no written reason from the model, because the model produces none [2], and the specification says so, with the hand-off rule from auditability when the model gives no written reason for the cases where a written reason is required.

The check: an eval that makes one decision and asserts every listed field is present and non-empty in the record.

7. The calibration set and its owner

The set of labelled cases the feature is measured against: where it lives (in the repository, beside the questions), how many cases it has (100 to 200 to start, growing from escalations), how it was labelled (blind, two labellers on a sample, with the human agreement rate recorded), who owns it by name or role, and when it is next refreshed. The pilot's labelled set is the first version. Measuring a decision model in production gives the refresh schedule.

The check: an eval that runs the full calibration set on every change and fails the change if agreement at the stated thresholds drops below the stated cutoff. This is the eval that turns the specification into a guarantee, and it is the one we would not release without.

8. The model version policy

The pinned version, jev-1.13.0 at writing [7], written as the version and never as the jev-latest alias, because an alias moves on the vendor's schedule. The rule for a change: a new version is run against the calibration set, its agreement and its escalation rate are compared against the current version's, the comparison is recorded, and the move happens on a named person's decision. What happens if the old version is withdrawn before that: the fallback in section 5 runs until the comparison is done. And the contract terms that back this up, from early-access risk and vendor lock-in.

The check: an eval that asserts the request names the pinned version string; a check in the pipeline that fails any change to that string unless a comparison record is attached.

How this connects to eval-driven development

The eight sections produce eleven checks, and every one of them is written before the feature's code. That is the practice the eval-driven development guide describes for all software, and writing specs an agent can verify is its general version. For a decision-model feature, two of the checks are themselves graded by a decision model: whether the escalation queue's content matches the acceptance criterion, and whether a change to the questions stayed in scope. Evals with Jev is the engineering guide to grading those, and never let the model grade its own work is the rule that the grader is never the model whose feature is being graded.

What a finished specification looks like

Two to four pages. One sentence at the top. A list of questions in their exact words. A table of actions with a threshold each. Two sequences, one for low confidence and one for the API being down. A field list for the record. A paragraph naming the set, its size, its owner and its refresh date. A version string and a rule. Eleven checks, one under each section, each of which will exist as a running eval before the feature does.

If any of the eight is missing, the feature is not specified, and an engineer or an agent building it will fill the gap with a guess. What a good technical spec looks like covers the rest of the document; the pillar guide covers the decision that led here. Write the thresholds first. Everything else in the feature exists to serve them.