Intent routing with Jev

TypeSafe's intent-routing pattern uses a customer-service example with four intents: order_status, product_question, return_exchange and complaint, each with a one-line description [1]. The request text is the state, the intents are the options of a Choice question, and the answer comes back as a choice, a probability per option and a confidence number. Code then does one check before it trusts the answer: if confidence is below 0.5, the request goes to a human agent [1]. Everything else in the pattern is detail on those two sentences.

The reason the pattern matters is where the expensive work happens. TypeSafe's description is that the classification happens in "a single quick call" and the expensive resources are used only for requests that need them [1]. The specialist model, the reasoning model or the person is called after the route, and only for the share of traffic the route sends there.

Writing the options

A Choice question accepts up to 255 options [2]. Each option is a name plus a description, and the description is what the model reads, so it carries the whole meaning. TypeSafe's Choice docs recommend that when options are often confused with each other, the description should say what the option covers, what it excludes, and give an example, instead of a single phrase [2].

In practice that means writing the options the way you would write a routing rule for a new employee. "return_exchange: the customer wants to send an item back or swap it. Excludes questions about whether a product can be returned before buying, which is product_question. Example: I want to return the second one, the first was fine." The exclusion line is what stops two neighbouring options from splitting the probability between them, which is what lowers confidence.

The jaggedness page adds a rule that applies to every option description: the model answers exactly the question you wrote, and scoping words, negations and implied conditions are read literally [3]. So an option described as "anything about billing" will attract a message about a wrong shipping address if the message mentions a charge. Write the boundary explicitly.

The none of these option

TypeSafe's Choice docs say to include an "other" or "none of the above" option whenever the list might not cover every input, so the model can say that none of the others fit [2]. We treat this as mandatory for a router. Without it, a request that matches nothing is forced into the nearest option with a flat probability distribution, and the only signal you get is low confidence. With it, you get two signals: a low confidence on a real option means the model is split between handlers, and a confident "none" means the request is outside what the product handles.

Those two cases need different handling. A split answer can go to a person with the top two options shown, so the person's choice becomes a label for later. A confident "none" is a product question: either a new handler is needed, or the request should be answered with a fixed message. Log them separately, because the counts drive different decisions.

Confidence and the human path

The confidence number on a Choice answer is computed from how concentrated the probabilities are: a single peak means high confidence and probability spread across several options means low [2]. TypeSafe's confidence page gives the general bands, 0.9 and above to act automatically and below 0.5 to route to a human, ask for clarification or use a different system [4]. The intent-routing example uses 0.5 as its cutoff for the human path [1].

For a router the middle band has a specific meaning. A confidence of 0.7 with the probability split between return_exchange and complaint is a request that is both, and a good product handles both: route to the returns handler and flag the complaint. That is why we keep the full probability map in the log and in the handler's input, and why TypeSafe's confidence page says the fuller probabilities property is always available when the single number does not fit your case [4]. Confidence-gated routing covers the bands in detail, and the point here is only that the router should pass the probabilities on rather than the choice alone.

Long lists: rank every option, then re-check the top few

Two limits push a router towards two stages. The first is the 255-option cap. The second is that a Choice over many options with long descriptions makes a large state, and accuracy falls as the state grows [3].

TypeSafe's skill-suggestion cookbook is the reference example. The task is to pick which one of 182 skills, from Nous Research's Hermes catalogue, a coding agent should load for a request, or none [5]. Stage one asks a single Choice question over all 182 skills using truncated descriptions, plus three Noul questions about whether the request needs an action, a documented procedure, or only an explanation; if the mean of those three is below 0.30 nothing is suggested, and otherwise the top three candidates go on. TypeSafe reports 0.16 to 0.31 seconds for this stage. Stage two re-checks those three with their full descriptions and opening sections, asks a "fits" question for each, and rejects all of them if the best fits score is below 0.30; TypeSafe reports 0.09 to 0.12 seconds [5].

On TypeSafe's own test of 488 requests (315 covered by exactly one skill, 173 by none), wrong skill loads fell from 16.8 percent to 7.3 percent and needless loads from 9.8 percent to 4.0 percent, with the agent running on a Claude Haiku model [5]. The cookbook also reports the cost of a wrong suggestion: 37 requests that used to fail now succeed, and 7 that used to succeed now fail because of an incorrect suggestion. That trade is the thing to measure in your own router, because a confident wrong route is worse than a request that waits for a person.

The shape generalises. Stage one is cheap and wide: short descriptions, one Choice, a probability per option. Stage two is narrow and careful: the top few candidates with everything the model needs to tell them apart, and a threshold under which the answer is "none". Both stages are one request each, so the whole route is two round trips of under a third of a second each in TypeSafe's measurements.

Routing to a model rather than a handler

A router can pick a model as well as a handler. LangChain's harness post describes a ModelRouterMiddleware that assesses each request with Jev and selects a model on stated criteria, in its example between a fast model for direct lookups and a powerful one for complex decisions, choosing the least costly model that can complete the task [6]. The mechanics are the same Choice question; the options are models instead of queues.

We use this shape inside agents where most turns are simple and a few are hard. The route runs on every turn at decision-model cost, and the expensive model is called for the share of turns that need it. Single agent or multi-agent in our agents guide covers when that split is worth the added complexity.

What a route costs

TypeSafe's public price is $0.042 per million input tokens, and output tokens are free [7]. A route over a 500-token message with a 200-token option list costs $0.0000294 at that price. Over a million requests a month that is $29.40. The two-stage router costs the same twice plus the larger stage-two state. These are arithmetic on the list price and say nothing about discounts or future changes, and TypeSafe's own benchmark comparison against language models was written by TypeSafe's capabilities team [8], so treat the list price as the one number you can rely on and measure your own request sizes.

Latency is TypeSafe's reported 70 to 500 milliseconds end to end [9]; Openlayer measured a p50 of 244 milliseconds and a p95 of 371 milliseconds on its own guardrail benchmark, where a p50 is the time half of requests beat and a p95 the time 95 percent beat [10]. A router on the request path adds that to every request, so Rate limits, retries and latency covers timeouts and the fallback route when the model is unreachable.

How we build a router

Reveneau builds the router as code with the options in a versioned file next to the handlers, so a new handler and its option description land in the same change and the eval suite checks both. Every route is logged with the model version, the full probability map, the confidence and the handler chosen. Requests that went to a person keep the person's choice as a label, and that labelled set is re-run on every model version change before the pin moves. Versions, drift and monitoring covers that loop. For the person deciding whether a router belongs in the product at all, decision models for product teams is the shorter version.