Calibrated probabilities and confidence, explained

The number that comes back with every Jev answer is worth more than the answer. An answer tells you what the model thinks. The probability tells you how much to trust it, and if the probability is calibrated, it tells you that in a form you can plan around. This page explains what calibration means, how TypeSafe's confidence number is computed, what the docs recommend for thresholds, and how we set ours.

What calibrated means

A probability is calibrated when it matches the frequency of being right. Take every answer where the model said 0.8. If the model is calibrated, 8 in 10 of those answers are correct. Take every answer where it said 0.55, and 55 in 100 are correct. The number is a promise about frequency, and it holds across many answers rather than for any single one.

That is different from a probability that is merely ordered. An ordered probability tells you that a 0.8 is more likely right than a 0.6, without telling you how often either is right. Ordering is enough to rank cases. Calibration is what you need to set a threshold, because a threshold is a decision about how many mistakes you will accept, and only a calibrated number lets you compute that.

TypeSafe says it trains Jev for calibration with a method it names Reinforcement Learning for Calibrated Decisions, or RLCD [1]. It has not disclosed the architecture, as MarkTechPost reported on 19 September 2026 [2], so what a team can verify is the output: whether the probabilities on its own data behave as frequencies. We say how to check that below.

What confidence is, and how it is computed

Confidence is a second number, separate from the probabilities, that TypeSafe returns for Choice and Score questions. The docs describe it as "a single number from 0 to 1" computed from how spread out the probabilities are [3]. If one option holds almost all of the probability, confidence is high. If the probability is shared across several options, confidence is low.

For three options the docs give the formula (3 x largest probability - 1) / 2 [3]. Work through it. If the largest probability is 0.9, confidence is (2.7 - 1) / 2, which is 0.85. If the largest is 0.5, confidence is (1.5 - 1) / 2, which is 0.25. If the largest is 0.34, which is barely above an even split across three options, confidence is (1.02 - 1) / 2, which is 0.01. The formula maps "no better than a guess" to 0 and "certain" to 1, and the shape generalises to other option counts.

The Score example in the docs shows what a middling confidence looks like. Probabilities of 0.0, 0.57 and 0.43 across three levels give a score of 1.43 and a confidence of 0.35 [4]. The model is sure the text is above level 0 and split between levels 1 and 2, and the confidence number reports that split as a single figure.

Why Noul has no confidence number

A Noul question returns one probability from 0 to 1 and no confidence number [5]. The reason is arithmetic. With two outcomes, the probability is the spread. A 0.95 is concentrated on yes, a 0.05 on no, and a 0.5 is the even split. Adding a confidence number would repeat what the probability already says.

So for Noul, the threshold goes on the probability directly. For Choice and Score, you have a choice of what to threshold: the probability of the chosen option, or the confidence number. In most cases they move together. The difference shows up when the second-best option is close to the best, which is exactly the situation confidence is designed to flag.

TypeSafe's recommended bands

The confidence docs give three bands. At 0.9 and above, "Act automatically". Below 0.5, "Route to a human, request clarification, or fall back to a different system". In between, "Proceed with caution" [3]. Those are starting points rather than rules, and the same page adds the line that matters most: "Different actions within the same system should be gated at different levels depending on the consequences" [3].

That line is the whole discipline of using a decision model. The band belongs to the action. A 0.8 is the same 0.8 whether the action is adding a tag to a ticket or issuing a refund, and the two should be gated differently because a wrong tag costs a click to fix and a wrong refund costs money. TypeSafe's guardrails cookbook shows a worked policy: on a 0 to 3 severity score, review at 0.35 and act at 0.70 under a strict policy or 0.85 under a permissive one, with the route precedence support, block, review, pass [6]. The two policies differ only in where the action threshold sits, and that is a product decision about how many false blocks the team will accept.

How to set a threshold for each consequence

We use a three-step method, and it works for any decision model that returns calibrated probabilities.

First, name the two mistakes. Every yes/no decision can be wrong in two directions: acting when you should not, and holding back when you should have acted. Write down what each costs in your product. For a spam filter, a false block hides a real message and a false pass shows a spam one. For an eval grader, a false pass releases a broken change and a false fail delays a good one.

Second, price the two mistakes against each other. If a false pass costs ten times what a false fail costs, you want a threshold that produces at most one false pass per ten false fails. That ratio, applied to calibrated probabilities on a labelled sample, gives you a threshold rather than a feeling. It is the same reasoning behind how to decide if a feature needs a human in the loop.

Third, set two thresholds rather than one. Above the upper threshold, act. Below the lower one, take the safe default. Between them, send the case to a person and keep that person's label. The middle band is where the model has told you it is unsure, and it is the cheapest labelled data you will ever collect, because the model has already sorted out the easy cases.

This is what Reveneau does in its eval suite. Checks with a deterministic answer never touch a model. For the checks that used to need a language model judge, each is a Noul or Score question to Jev with the rubric written into the criteria. A grade below the threshold fails the change. A grade in the uncertain band goes to a person, and that person's label is kept so we can re-check the grader later. On our own suite the run is ten times faster than it was with the previous grader. How Reveneau uses Jev has the detail, and evals with Jev turns it into a how-to.

How to check calibration on your own data

Calibration is a claim you can test, and you should, because the claim is TypeSafe's and the data is yours. Take a labelled sample of a few hundred cases. Send each through the question you plan to use. Sort the answers into bins by probability: 0.5 to 0.6, 0.6 to 0.7, and so on. For each bin, count how many were correct. If the 0.8 to 0.9 bin is right 85 percent of the time, the model is calibrated there. If it is right 60 percent of the time, it is overconfident on your data, and your threshold has to move up to compensate.

Two things make this test more useful. Run it per question, because calibration on one question says nothing about another. And re-run it when you change the question text or when TypeSafe releases a new model version, because either can move the curve. The current version is jev-1.13.0 and the jev-latest alias tracks whatever is newest [7], so a team that uses the alias should expect the curve to move without warning and should pin a version for anything with a tight threshold.

Two limits from TypeSafe's own jaggedness page bear on this. Score levels "are weak in numerical calibration", so a Score's between-level value is less reliable than its ordering [8]. And "Accuracy falls as the state grows with content unrelated to the decision" [8], which means a question that is calibrated on short, clean states may be less so on long, noisy ones. Test on the states you will actually send.

What confidence cannot tell you

A confidence number reports how concentrated the model's belief is, and the question itself can still be the wrong question. If you ask "Is this message spam?" and the model returns 0.96 for yes, that is a confident answer to a question TypeSafe itself names as too broad [9]. Confident and correct are different properties, and DataCamp's report of TypeSafe's own evaluation shows the gap: 0 percent structured-output errors and 67.8 percent agreement with the reference answer on the same workflows [10]. Every one of the wrong answers had a well-formed probability attached to it.

Nor does confidence protect against hostile input. TypeSafe documents that adversarial content in the state "can move the answer" [8]. A message written to look like a legitimate request will get a confident answer that it is one. Confidence gating reduces the damage from the model's uncertainty, and a separate check on the input is what catches certainty that was manufactured; where Jev is weak covers it.

The one rule to keep

Gate every action on its own consequences. The probabilities are the model's, and the thresholds are yours. A team that sets one threshold for everything has decided that a wrong tag and a wrong refund cost the same, and no team believes that once it is written down. The pillar page, Jev and System One models, puts this in context, and Jev in production covers what happens to thresholds once the system is live.