What Jev returns: Choice, Score and Noul

Every request to Jev carries a state and a map of named questions, and every question is one of three types, which TypeSafe's docs call primitives [1]. Getting the type right is most of the work of using the model well, because each type returns a different shape and each shape fits a different structure in code. This page goes through the three, with the exact limits from the docs, and ends with how to name them.

Choice: one option from a list

A Choice question gives the model a list of options and asks it to pick one. The list can hold up to 255 options. The answer has three parts: the chosen option, a probability for every option in the list, and a confidence number from 0 to 1 [2].

The probability per option is the part most teams under-use. If you ask which of five support queues a ticket belongs to and the answer comes back as billing at 0.51 and technical at 0.46, the chosen option is billing, and the probabilities tell you the model could barely separate the two. That is a different situation from billing at 0.97, and your code should treat it differently. The confidence number summarises that spread in one figure, and calibrated probabilities and confidence, explained covers how it is computed and how to set thresholds on it.

In code, a Choice question is a switch statement. Each option is a case, and the probability is the guard that decides whether to take the case automatically or hand it to a person. Two rules make the list work. Every option should be something your code can act on, because an option nobody handles is a branch to nowhere. And the options should be mutually exclusive, because if two of them overlap the model will split its probability between them and the confidence number will drop for a reason that has nothing to do with the text.

Score: a position on an ordered scale

A Score question describes a scale of 2 to 10 levels in words and asks where the text sits. The answer is a probability per level, and the score itself is each level number multiplied by its probability, added together [3]. That means it can land between levels.

The docs' own example makes this concrete. A three-level scale, numbered 0, 1 and 2, returns probabilities of 0.0 for level 0, 0.57 for level 1 and 0.43 for level 2. The score is 0 x 0.0 + 1 x 0.57 + 2 x 0.43, which is 1.43, and the confidence is 0.35 [3]. Read that as: the model is sure the text is above level 0, and it leans towards level 1 while giving level 2 a real chance. A score of 1.43 with confidence 0.35 carries more information than a hard label of 1 would.

In code, a Score question is a threshold on a number. TypeSafe's guardrails cookbook shows the shape: a severity scale of 0 to 3, with a review threshold of 0.35 and an action threshold of 0.70 under a strict policy or 0.85 under a permissive one [4]. Below the first threshold the message passes, between the two it goes to review, above the second the system acts. The levels are described in words in the question, so the scale means what you wrote it to mean.

One limit from TypeSafe's own list of weaknesses applies here: Score levels "are weak in numerical calibration" [5]. The ordering is reliable, so a 2.1 is above a 1.4. The exact distance between them is less reliable than the ordering, so a threshold should be set from your own labelled data rather than read off the scale as if the numbers were measurements. Where Jev is weak says more.

Noul: one probability that a statement is true

A Noul question is a statement, and the answer is one probability from 0 to 1 that the statement holds for the state. There is no separate confidence number [6]. That is by design: with two outcomes, the probability already says how sure the model is. A 0.95 is a confident yes, a 0.05 is a confident no, and a 0.5 means the model cannot tell.

In code, a Noul question is an if with a threshold. The statement should be written so that a yes means the same thing to the model, to the code that reads the answer, and to the person who reads the code six months later. "The response answers the question the user asked" is a good Noul. "The response is not off topic" is a worse one, because it contains a negation, and TypeSafe's jaggedness page says negations are read at face value and double negatives are answered less reliably [5].

Noul is the type we use most. Reveneau grades its eval suite with Jev, and most acceptance criteria are yes/no statements: the change adds the field the specification names, the diff touches only the files the plan listed, the generated text follows the instruction. Each is a Noul question with the criterion as the statement, and a probability below the threshold fails the change. How Reveneau uses Jev describes the thresholds, and evals with Jev is the how-to.

Which type for which decision

A short rule covers most cases. If the answer is one item from a list, use Choice. If the answer is a degree of something, use Score. If the answer is yes or no, use Noul.

The mistakes are predictable. Teams write a two-option Choice where a Noul would do, which works but gives up the simplicity of a single probability. Teams write a Score where the levels are categories rather than an ordered degree, which confuses the model because a Score assumes level 2 is more of something than level 1. And teams write one Choice with 40 options where four Nouls asked in parallel would be clearer, because each Noul gets a probability on its own terms rather than sharing one distribution across 40 options. TypeSafe's design guidance is to break a broad judgment into atomic questions asked in parallel and combine them in code [7], and many questions in one call covers the mechanics.

What every answer carries

Whatever the type, three things come back with the answer and all three are worth keeping.

The probabilities are the model's belief about each outcome. Log them. When you review the cases the model got wrong, the probabilities tell you whether it was confidently wrong, which is a question-writing problem, or uncertain, which is a threshold problem.

The shape is fixed. The docs describe the answer as always matching the schema of the question, which TypeSafe calls not hallucinating. What that means is that a Choice always returns one of your options, a Score always returns a value on your scale, and a Noul always returns a number between 0 and 1 [1]. The answer can still be wrong: DataCamp reports TypeSafe's own evaluation at 0 percent structured-output errors for Jev and 67.8 percent agreement with the reference answer [8], and the gap between those two numbers is why you need the probabilities.

The independence from other questions. TypeSafe's batching cookbook found that answers did not depend on what else was in the request, with a standard deviation of 0.0 across five repeats for 11 of 13 questions [9]. So you can add a question to a request without changing the answers to the others, which is what makes fan-out safe.

How to name the questions

The questions are a map, so each has a key, and the key is what your code reads. A few habits make the map readable.

Name the question as the thing you will do with the answer. "route_queue" for a Choice that picks a queue, "severity" for a Score, "in_scope" for a Noul. The name should tell a reader what branch, threshold or if the answer feeds.

Write the question text as one clause with no negation and no implied condition. TypeSafe's jaggedness page is explicit that the model "answers the question you wrote, not the one you meant" and that scoping words, negations and implied conditions are read literally [5]. "The diff modifies only files listed in the plan" is one clause. "The diff does not touch anything it should not" contains a negation and an implied standard, and it will be answered literally against whatever the model takes "should not" to mean.

Put the option list or the scale description in words the state uses. If your tickets say "refund" and your options say "reimbursement", you have added a translation step the model has to guess at.

Keep one decision per question. "Is this spam and is it urgent" is two questions wearing one name, and TypeSafe names "Is this spam?" on its own as too broad [7]. Ask four narrow Nouls about what the message contains and combine them in code.

A worked example

Take an incoming support message. One request, four questions. A Choice named route_queue with the options billing, technical, account and other. A Score named urgency on a scale of 0 to 3 described as: can wait a week, should be answered today, needs a reply within the hour, service is down for the customer. A Noul named contains_pii with the statement "The message contains a personal identifier such as a full name with an address, a card number, or a government ID number." A Noul named is_abusive with the statement "The message contains threats or abuse directed at a person."

The answers come back together in one response. Code reads route_queue and takes the branch if confidence is above the threshold for that queue, or sends it to triage if below. Code reads urgency and sets the priority flag from the score. Code reads contains_pii and, above 0.7, masks the message before it is stored. Code reads is_abusive and, above 0.85, routes to the safety team ahead of everything else. Four decisions, one request, and the response time is the same as for one question, because TypeSafe evaluates every question in parallel [10].

That is the whole shape of building with Jev: a state, a handful of well-named questions of the right type, and code that reads the probabilities and acts. The pillar, Jev and System One models, puts it in context, and how to write the state and the questions is the next page to read.