How to read an AI eval report if you are not an engineer

A report arrives before a release meeting. Its first line says the support assistant passed 92 percent of its eval cases. That 92 percent can come from 46 passes out of 50 cases or from 920 passes out of 1,000, and those two results support different decisions.

This page teaches a reader who is not an engineer to read such a report. It gives six things to look for, the arithmetic behind the first of them, and a checklist to print. Every report figure on this page is an invented illustration unless a source is cited beside it. An eval is a repeatable test: a set of inputs, a written expectation for each one, and a way to score the output. What an AI eval is has the full definition.

Start with what the pass rate was measured on

A pass rate is the number of cases passed, divided by the number of cases run. Before you read the percentage, find four facts in the report:

• how many cases were run
• where the cases came from: real user questions, or questions the team wrote
• the date of the run
• the exact model version

NIST, the United States standards body, asks for this in its AI Risk Management Framework: "Test sets, metrics, and details about the tools used during TEVV are documented" [7]. TEVV is its short form for test, evaluation, verification and validation. A report without these four facts cannot be checked by anyone.

Why 92 percent of 50 cases is weaker evidence than 92 percent of 1,000

The cases in an eval are a sample of all the questions users could send. A different sample would give a different pass rate. The confidence interval says how different. In plain words, it is the range that is likely to contain the true pass rate.

The formula uses the standard error, a number that says how far the pass rate could move if a different set of cases had been used. It has two steps, and both are given in a 2024 paper by Evan Miller [1]:

1. Standard error = the square root of (pass rate × (1 minus pass rate) ÷ number of cases).
2. 95% confidence interval = pass rate plus and minus 1.96 × standard error [1] [2].

NIST's statistics handbook calls this form "the confidence expression most frequently used" [3]. Here is the arithmetic for 46 passes out of 50:

1. 0.92 × 0.08 = 0.0736
2. 0.0736 ÷ 50 = 0.001472
3. The square root of 0.001472 is 0.03837
4. 1.96 × 0.03837 = 0.0752, which is 7.5 percentage points
5. 92.0 minus 7.5 is 84.5, and 92.0 plus 7.5 is 99.5

Cases	Passed	Standard error	95% range
50	46	3.8 points	84.5% to 99.5%
200	184	1.9 points	88.2% to 95.8%
1,000	920	0.9 points	90.3% to 93.7%

The range for 50 cases is 15.0 points wide. The range for 1,000 cases is 3.4 points wide. With 50 cases the true rate could be 85 percent or 99 percent, and the report cannot say which. Four times the cases cuts the range in half: from 50 to 200 cases the margin goes from 7.5 points to 3.8.

Two cautions. NIST's handbook says this simple form "may not be accurate enough" when the sample or the number of failures is small, and it presents a second method, named after Wilson, first [3]. For 46 passes out of 50 the Wilson method gives 81.2% to 96.8%. Both methods lead to the same reading: 50 cases leave a wide range.

The second caution is about related cases. Anthropic reports from its own work that when questions come in related groups, the correct standard error on popular evals can be more than three times the simple one [2]. Fifty rewordings of five questions give less evidence than 50 separate cases.

How many cases are enough depends on the width you can accept. Rearranging the same formula for a 92 percent pass rate: a margin of 5 points needs 3.8416 × 0.0736 ÷ 0.0025 = 114 cases, and a margin of 2 points needs 3.8416 × 0.0736 ÷ 0.0004 = 707 cases, both rounded up. The method is in how many test cases an LLM eval needs, where LLM means large language model.

A lower score than last time may be no change

Suppose last month's report said 92 percent on 200 cases and this month's says 90 percent on the same 200. The first range is 88.2% to 95.8%. The second, computed the same way, is 85.8% to 94.2%. The two ranges share everything from 88.2% to 94.2%, and the difference is 4 cases out of 200. A fall of this size is within what chance produces on a set this small.

Miller's paper recommends comparing two runs question by question instead of comparing the two averages [1]. So ask one thing: which cases passed last time and fail now? Four new failures spread across topics suggest chance. Four new failures that all concern refunds are a finding, whatever the average did. What happens when the AI model changes covers one cause of a real fall.

An average can hide a failing group

The best known measured case comes from face analysis software. A 2018 study of three commercial systems found that "darker-skinned females are the most misclassified group (with error rates of up to 34.7%)", while "The maximum error rate for lighter-skinned males is 0.8%" [4]. Two existing test sets the authors examined were 79.6% and 86.2% lighter-skinned subjects [4], so a strong overall score on them gave little information about the smaller group. A 2019 paper on medical imaging describes the same pattern: "overall performance of a cancer detection model may be high, but the model still consistently misses a rare but aggressive cancer subtype" [5]. Both studies concern image systems. The arithmetic applies to any average.

Here is an invented illustration. A support assistant is run on 1,000 cases. It passes 855 of 900 ordinary questions (95 percent) and 65 of 100 refund questions (65 percent). The total is 920 of 1,000, which is 92 percent. The report's first line is the same as before, and 35 of every 100 refund answers are wrong. Ask for the pass rate per group, and for the number of cases in each group.

The failures decide what a good pass rate is

No single number is a good pass rate. NIST's framework "does not prescribe risk tolerance" [7], and the right figure depends on what the failures are. Ask for the list of failed cases, with the input and the output of each.

Then sort them. A clumsy sentence is one type of failure. A wrong refund rule or an invented source is another, and public AI failures and the tests that target each one shows what that type has cost. One failure of the second type can stop a release at a 99 percent pass rate. Using evals to decide a release explains how to set the pass line before the run.

Who or what did the grading, and how many times each case ran

A grader is the program or person that scores an output. Anthropic's engineering blog lists three types: "code-based, model-based, and human" [6]. When a model does the grading, ask how often it agrees with a person on a sample that people labelled first. Human review and reviewer agreement gives the method, and never let the model grade its own work explains why the grader must be separate from the product.

Then ask how many times each case ran. The same post says: "Because model outputs vary between runs, we run multiple trials to produce more consistent results" [6]. A report should state whether "pass" means the case passed once or passed every time. The two figures differ. In Anthropic's example, a product that succeeds on 75 percent of single runs passes all three of three runs 42 percent of the time, because 0.75 × 0.75 × 0.75 = 0.421875, which rounds to 42 percent [6]. Agent reliability across repeated runs covers this in detail.

Whether the set was run before, and whether the product was adjusted to fit it

Ask two things about the history of the set.

First, was the same set run on the previous version? Only then can two reports be compared.

Second, did the team adjust the product while looking at these same cases? If so, the score on them is higher than the score on new questions will be. The protection is a held-out set: a group of cases kept away from the people adjusting the product and used only for the final score. A 2024 paper by Kapoor and four co-authors reported on public tests of AI agents, which are AI programs that take steps on their own: "many agent benchmarks have inadequate holdout sets, and sometimes none at all" [8]. A benchmark is a public test set. Ask whether a held-out set exists, how many cases it has, and what the product scored on it. When evals give false confidence lists other ways a passing score can mislead.

Each number on the report and what to ask next

Number on the report	What it tells you	What to ask next
Pass rate	The share of cases passed	On how many cases, and where did they come from?
Number of cases	How wide the range is	Are the cases separate, or rewordings of a few?
95% range	The range likely to contain the true rate	Does the low end still meet the pass line?
Pass rate per group	Which type of question fails	How many cases are in the smallest group?
Runs per case	Whether a pass means once or every time	What is the rate when every run must pass?
Grader agreement with people	How far to trust the grader	On how many labelled cases was it measured?
Change since the last run	The direction of movement	Which cases changed from pass to fail?

A checklist to print

1. How many cases were run, and on what date?
2. Which exact model version and which prompt (the written instructions given to the model) were tested?
3. Where did the cases come from: real users or the team?
4. What is the 95% range around the pass rate?
5. Are the cases separate, or several wordings of the same question?
6. What is the pass rate for each group of cases?
7. Which cases failed, and what did the product answer?
8. Which failures would cost money or cause harm?
9. Who or what graded the outputs, and how was the grader checked?
10. How many times was each case run, and does "pass" mean every time?
11. Was this same set run on the previous version, and which cases changed?
12. Is there a held-out set, and what was the score on it?

The same questions work on a supplier's report. How to evaluate a vendor's eval suite gives more detail, and why AI evals matter gives the wider argument.

How Reveneau reports eval results

At Reveneau all code is written by AI, and every change must pass a large eval suite, written from the specification before the code, before it is released. An eval report from Reveneau states the items on this page's checklist: the number of cases and where they came from, the 95% range beside every pass rate, the pass rate per group, the full list of failed cases, the grader, the number of runs per case, and the model version. We write the pass line before the run.

The judged checks in our suite are graded with Jev, TypeSafe AI's decision model, and on our own suite the run is ten times faster than with our previous language-model grader. We check that grader against human labels, and the report says so.

Reveneau, as a company, takes responsibility for the whole project through production and after release. See AI development at Reveneau or contact us.