Amrut Joshi shows why an LLM's own confidence score cannot be trusted

Many agent designs route a reply to a human reviewer when the LLM says it is unsure of itself. In a post that reached the Hacker News front page with 107 points, Amrut Joshi argues the number the model returns in that step is a token continuation of the prompt, so it has none of the properties a confidence score needs. He gives a five-step rule for building one the team can trust.

The piece appeared on the rumple.github.io blog. Joshi writes that an LLM "produces text one token at a time", so when a model answers "0.85" to "how confident are you", that string is a plausible continuation of the chat and nothing more. It was never read from an internal gauge and was never checked against how often the model is correct.

What a real confidence score needs

Joshi defines the two properties a usable confidence has. First, it depends on the information: with the same facts and the same rule, two forecasters have to arrive at the same number. Second, it is calibrated against outcomes, which Joshi illustrates with a one-year weather record. On the 120 days the forecaster said 10 percent, it rained 11 percent of the time. On 90 days at 30 percent, 29 percent. On 80 days at 70 percent, 71 percent. On 75 days at 90 percent, 89 percent.

A number from an LLM has neither. The output is a function of the prompt and the prior tokens, so rewording the question changes the number. And the model was never scored on whether the confidence it writes down matches how often the task is right, only on what text comes next. Joshi says the number "will always look precise" and that an incorrect one looks the same as a correct one.

Joshi's five-step rule for a trusted score

For a customer-support agent that reads a help article and writes a reply, Joshi lays out five steps. One, name the things that can be observed about each reply without asking the LLM how it feels, including whether the search returned a matching article, how many, whether a question of this type has been answered before, and whether the agent gives the same reply when asked twice. Two, start every reply at the base rate, so a system where a reviewer approved 150 out of 200 past replies begins at 75 percent.

Three, record every reply with those observations, the confidence the rule gave it, and whether it turned out to be correct. Four, group past replies by their observations and set the rule's number for each group to that group's past accuracy, so a group that was right 95 percent of the time gets 95 percent and a group that was right 40 percent gets 40 percent. Five, run the same calibration check the forecaster does, every week or so, and update the rule when a confidence level is higher or lower than its actual rate.

Joshi says the rule produces a confidence that stays the same whenever the information stays the same, and that depends on the information itself. The score is calibrated against the team's own kind of question, because it was built from its own records. The caveat Joshi gives is that it works only if someone checks whether past replies were correct, so the record exists for the rule to learn from.

Source

Asking LLM for confidence by Amrut Joshi on the rumple.github.io blog.