Many questions in one call: speculative fan-out

The cheapest thing you can do with Jev is ask it more questions. That is an odd sentence to write about a paid API, and it follows from how the price and the latency work: you pay per input token, the state is most of the input, and every question in a request is evaluated in parallel against that one state. TypeSafe's introduction puts it as "Adding questions barely changes the response time" [1]. This page explains why, gives TypeSafe's own numbers, and sets out when fanning out is the right call and when it wastes money.

Why a second question is almost free

A Jev request carries a state and a map of questions. The bill is $0.042 per million input tokens, with output free [2]. The state is where the tokens are: a support ticket, a diff, a document. A question is a sentence. So a request with one question and a request with thirteen questions differ in input by twelve sentences, which against a document of any size is a rounding error.

Latency works the same way. TypeSafe evaluates every question in a request in parallel [1], so the response time is set by the state, and adding a question adds almost nothing. Compare that with a language model, where every question means another generated answer, and every generated answer means more tokens at the output price and more seconds of waiting.

TypeSafe's measurement

The batching cookbook is TypeSafe's own test of this, so read it as the vendor measuring its product on a workload it chose. The state was the Wikipedia article on the GDPR, 53,777 characters. The model was jev-1.12. Thirteen questions were asked two ways, five repeats each: once as a single request carrying all thirteen, and once as thirteen separate requests carrying one each [3].

The batched request cost $0.000497 and took 0.27 seconds. The thirteen single requests cost $0.006090 and took 2.71 seconds in total. TypeSafe reports that as "12.2x cheaper, 10.0x faster" [3]. The cost ratio is close to 13 because the state was sent thirteen times in the single-request version and once in the batched one. The time ratio is close to 10 because the thirteen single requests ran one after another, each paying the full round trip.

The second finding matters more for engineering. TypeSafe reports that the answers did not depend on what else was in the request: across the five repeats, the standard deviation was 0.0 for 11 of the 13 questions [3]. So a question's probability is the same whether it is asked alone or alongside twelve others. That independence is what makes it safe to add a question to a live request without re-checking the thresholds on the ones already there. Two of the thirteen showed some variation across repeats, which is worth remembering: identical is the usual case and near-identical is possible.

The pattern: speculative fan-out

TypeSafe's patterns page names four patterns, and speculative fan-out is the one this page is about [4]. The idea is to ask every question you might need in one request, up front, and let code decide afterwards which answers to use.

The alternative is sequential: ask whether the ticket is billing, wait, and if so ask whether it is a refund request, wait, and if so ask whether the amount is disputed. Three round trips, each paying the state again. Fan-out asks all three at once. Code reads the first answer and, depending on it, reads the second and third or ignores them. The answers you ignore cost twelve sentences of input and nothing in time.

The word speculative is precise. You are asking questions whose answers you may never read, on the bet that the cost of asking is lower than the cost of a second round trip. On TypeSafe's numbers that bet wins whenever the state is longer than a sentence or two.

The pattern combines well with the design guidance to break broad judgments into atomic questions [5]. Once you are asking five narrow Nouls instead of one broad question, you are already fanning out, and the cost is the same as asking the broad question would have been.

Where fan-out is the right call

Screening. TypeSafe's guardrails cookbook screens a message in one request with questions such as jailbreak, harmful_request, medical_advice and self_harm, plus a 0 to 3 severity score [6]. Every incoming message gets every question, code applies the route precedence support, block, review, pass, and the whole screen costs one request.

Eval grading. A change has several acceptance criteria, and each is a Noul. Ask all of them in one request with the criterion set, the diff and the plan as the state. Openlayer's jevals library does this for agent traces: one Jev request per trace, with 37 built-in evals across agent, security and quality categories available to include [7]. On Openlayer's own benchmark that came to $0.03 per 1,000 samples and 0.8 seconds for 20 samples, against $2.60 per 1,000 and 22 to 35 seconds for 20 with a language-model-based alternative that made 6 model calls plus embeddings per sample [7].

Triage. A support ticket needs a queue, a priority, a personal-data flag and an abuse flag. Four questions of three types, one request, and the response time is set by the ticket length.

Agent supervision. LangChain's harness uses a TypeSafe classifier as a middleware that checks a tool call before it executes [8]. Every tool call gets the same set of questions, and the check adds one short round trip rather than several.

This is how Reveneau's eval suite works. Every criterion for a change goes to Jev in one request with the criterion, the diff and the plan as the state. Deterministic checks stay in code and never touch the model. The uncertain band goes to a person and the label is kept. On our own suite the run is ten times faster than it was with the previous grader, and fan-out is part of the reason: the grading step used to be a sequence of language model calls, and it is now one request per change. How Reveneau uses Jev and evals with Jev have the detail.

Where fan-out wastes money

Fan-out is a bet, and there are three situations where it loses.

The questions need different states. Fan-out works because one state serves every question. If question A needs the ticket body and question B needs the customer's order history, putting both in one state makes the state longer for every question, and TypeSafe documents that accuracy falls as the state grows with content unrelated to the decision [9]. Two requests with two focused states lose no accuracy and cost close to the same in tokens.

Most answers are never read and the state is long. The cost of a question is the sentences it adds to the input. If the state is 30,000 tokens and you add twelve questions of which you will read one, the twelve sentences cost almost nothing, so fan-out still wins. If the state is 30,000 tokens and the alternative was to not send the request at all on most paths, then the state is the cost, and a cheap deterministic check that skips the request entirely on the common path wins.

The questions depend on each other's answers. Fan-out assumes the questions can be asked without knowing the answers to the others. Sometimes the second question only makes sense given the first answer, and its text has to change. That is a genuine sequence, and TypeSafe's design guidance says to keep control flow in code [5]: ask the first question, branch in code, and ask the second only where it applies.

The limits that bound a request

Two hard limits shape how far fan-out can go. The request is capped at 64k tokens, of which the state plus the longest question may use 32k [2]. And the rate limits are 250,000 tokens per second and 1,200 requests per minute [2]. Fan-out pushes you towards fewer, larger requests, which is the right direction under a requests-per-minute cap: 1,200 requests per minute is 20 per second, and a service that fanned out thirteen questions into thirteen requests would hit that cap thirteen times sooner. Jev pricing, rate limits and context window goes through the limits and the error codes.

A Choice question can carry up to 255 options and a Score question 2 to 10 levels [10], so a single request can carry a large decision surface. The practical limit is readability: a request with 40 questions is hard to review and hard to debug when one misfires. Group questions by decision and keep each group's request small enough that a person can read it.

A worked example, with the arithmetic

Take a 4,000-token support ticket and four questions of 20 tokens each. One request: 4,080 input tokens, which at $0.042 per million is $0.000171. Four sequential requests: 4 x 4,020 tokens, which is 16,080 tokens and $0.000675. The batched request costs a quarter of what the sequential version costs, and on TypeSafe's latency figures returns in one round trip instead of four. Over a million tickets, the batched version costs $171.36 and the sequential version $675.36. The arithmetic is simple, which is why it is worth doing before you build.

The pillar, Jev and System One models, puts this pattern alongside the other three TypeSafe names. How to write the state and the questions is the page to read before you fan out, because a request with thirteen questions is only as good as its worst question. And Jev in production covers what fan-out means for monitoring, where the answers you ignored are still worth logging.