Confidence-gated routing: act, confirm, or escalate

TypeSafe's confidence-routing pattern uses a voice banking assistant, and it is the clearest example of the idea because the two actions in it have such different costs. A balance inquiry runs at 0.6 confidence, because in TypeSafe's words the worst case is the user having to listen to a balance read-out they did not ask for. A transfer at the same confidence asks the user to confirm. A transfer above 0.85 runs on its own [1]. One model, one question, three outcomes, and the outcome depends on what a mistake would do.

That is the whole pattern. The rest of this page is about setting the numbers and checking them against outcomes.

What the confidence number measures

Confidence on a Choice or Score answer is a single number from 0 to 1 derived from the shape of the probability distribution. TypeSafe's confidence page describes it as collapsing that shape: probability concentrated on one option means a confident answer, probability spread across several means an uncertain one [2]. For a Choice with three options the formula is three times the largest probability, minus one, divided by two, so a distribution of 0.9, 0.05, 0.05 gives 0.85 and an even split gives 0.

Two things follow. A Noul question has no confidence number, because its one probability already says how sure the model is: 0.5 is maximum uncertainty and 0.02 or 0.98 are near-certain [3]. And the full probabilities map is always in the response for the cases where one number is too coarse, such as a Choice where the top two options are the ones that matter [2].

The word calibrated appears in TypeSafe's description of the training method, Reinforcement Learning for Calibrated Decisions [4]. Calibrated means that when the model says 0.8, it should be right about 80 percent of the time on inputs like that one. Whether it is calibrated on your inputs is something you measure, and the logging section at the end is how.

TypeSafe's bands

The confidence page gives three bands and a sentence for each [2]. Below 0.5: the model is unsure, do not guess, route to a human, ask for clarification, or use a different system instead. Between 0.5 and 0.9: proceed with caution, and depending on the stakes gather more information, flag for review or ask the user to confirm. At 0.9 and above: "Act automatically" when the stakes are low, and for a high-stakes action such as a financial transfer, proceed with confirmation.

The same page says the thresholds depend on your domain and on how the model performs on it, and that you should start conservative and adjust from experience [2]. We read the bands as defaults for a first deployment, and the gate table below as the thing you actually maintain.

A gate per consequence

The sentence from TypeSafe's confidence page that we build around is that different actions within one system should be gated at different levels depending on the consequences, and that a read-only operation tolerates lower confidence than a destructive one [2]. We turn that into a table with one row per action, and each row has four columns: the action, the threshold to act, the threshold below which it goes to a person, and what happens in between.

For a read (show the balance, open the order, display the article): act at 0.6, escalate below 0.4, and in between act anyway with the answer visible so the user can correct it.

For a reversible write (add a tag, move a ticket to a queue, save a draft): act at 0.8, escalate below 0.5, and in between act and record it for a periodic review.

For a message to a user (send a reply, notify a customer): act at 0.85, escalate below 0.5, and in between show the user what will be sent and ask for a confirmation.

For an irreversible action (a payment, a delete, an external send to a third party): act at 0.95 or never, escalate below 0.7, and in between require a confirmation from a person with the probabilities shown.

Those numbers are our starting defaults and every one of them gets moved by the logged outcomes. Gating agent tool calls before they run applies the same table to an agent's tools, where the rows are tool names.

Escalate to a reasoning model only when uncertain

The most useful version of the middle band, in our work, is escalation to a more expensive model rather than to a person. TypeSafe's extraction cascade cookbook is the reference example, and a cascade here means a chain of stages where each stage is used only if the previous one did not settle the question [5].

Stage one is gpt-5.4-mini, at $0.75 per million input tokens and $4.50 per million output tokens in the cookbook's figures, extracting fields from a document. Stage two is jev-1.12, at $0.042 per million input and free output, asked one Noul question per field about whether that field is wrong (hallucinated, off target). Stage three is gpt-5.5, at $5.00 and $30.00, which re-extracts only when any field's flag exceeds 0.7 [5]. The gate is a maximum rather than an average, so one confident flag is enough to escalate.

The worked example in the cookbook shows why. The cheap model invented a description field that the source did not contain, which was schema-valid and looked fine. Jev flagged it at 0.95 on hallucinated and 0.85 on off target, both above the 0.7 gate, and the reasoning model returned an empty string [5]. On TypeSafe's 100-prompt test, the cookbook reports that the cascade's quality-against-cost position sits above and to the left of every single model, with the reasoning model alone at about 0.81 quality for around $0.10 per extraction. Those are TypeSafe's own prompts and TypeSafe's own scoring, so read the shape rather than the exact figures. The shape is: the cheap path handles most items, the decision model checks every item at decision-model cost, and the expensive path is paid for only where a check fails.

The same shape works for routing. A Choice with confidence in the middle band can be re-asked to a reasoning model with the top two options and the reason each might apply. That is one expensive call for the share of traffic that is uncertain, instead of one for every request.

Clarification as an outcome

The low band's second option in TypeSafe's list is "request clarification" [2]. For a conversational product this is often the best outcome. When a Choice returns 0.45 confidence split between return and exchange, asking "Do you want a refund or a replacement?" resolves it in one turn and produces a label for free. The clarification question can be picked with a second Choice question whose options are the clarifying questions, which is the pattern in reverse.

The failure to avoid is a loop of clarifications. Cap it at one, and send the second uncertain answer to a person.

Score questions have their own gate

A Score returns the expected level as a number that can sit between levels, with its own confidence [6]. The jaggedness page warns that score levels "are weak in numerical calibration" and that the number should be used for threshold checks rather than to compute an exact magnitude between levels [7]. So a severity score of 1.43 on a 0 to 3 scale means "closer to 1 than 2", and a gate at 2.0 is a sound use of it; treating 1.43 as 43 percent of the way from mild to serious is not something the model supports.

TypeSafe's intent-routing example gates a complaint on both axes: it escalates to a person when the complexity score is above 1 or when the score's confidence is below 0.5 [8]. Two conditions, either one enough.

What to log, and how to tune the gates

Every gated decision should be logged with six fields: the question ID, the model version from the response, the full probability map, the confidence, the threshold that applied, and the action taken (acted, confirmed, escalated, clarified). When a person was involved, log their decision too.

With that log, two checks run monthly. Calibration: bucket decisions by confidence (0.5 to 0.6, 0.6 to 0.7, and so on) and compare the model's answer with the person's or with the later outcome in each bucket. If the 0.8 bucket is right 60 percent of the time, the model is overconfident on your inputs and the gates move up. Cost of the middle band: count how many decisions were confirmed unchanged. If nearly all confirmations pass unchanged, the act threshold is too high for that action and can come down.

TypeSafe's self-consistency cookbook is worth reading on the second point. It routes a Noul answer to a person when the probability is in an inclusive band from 0.30 to 0.70, and its authors state that the escalation is application logic over the returned probability, with no second question and no second API call [9]. Across 15 repeated calls on one insurance claim, TypeSafe reports a mean per-question standard deviation of 0.0102 for Jev, and shows that on the judgment calls the answers sat near the 0.5 threshold (0.43 to 0.53 on one question) [9]. That is what the middle band is for: an input that is genuinely on the boundary should land there consistently, and a person should see it.

Where Reveneau uses this

Reveneau uses the gate table in every agent it builds and in its own eval suite. In the suite, a Noul or Score grade below the threshold fails the change, a grade in the uncertain band goes to a person, and that person's label is kept to re-check the grader later; on our own suite, measured against our previous grader, the run is ten times faster with Jev grading. Evals with Jev explains that setup. For the wider question of when a feature needs a person in the loop at all, how to decide if a feature needs a human in the loop is the shorter piece, and making an AI agent reliable covers the rest of the reliability work around the gate.