LLM evals: how to measure whether an AI product works

In 2023 three researchers, Lingjiao Chen, Matei Zaharia and James Zou, gave the same questions to GPT-4 in March and again in June. On one task, telling prime numbers from composite numbers, the March version was right 84 percent of the time and the June version 51 percent [1]. The service had the same name on both dates. Their conclusion was that a hosted model, one that a vendor runs and sells access to, can change its behaviour within a few months and has to be monitored continuously [1].

A product built on that model behaved differently in June, with no edit by the team that built it. The way to find that out is a test the team owns and can run again. That test is an eval. An LLM is a large language model, an AI model that produces text, and an LLM eval is a repeatable test of what a product built on one produces. OpenAI's documentation gives the reason ordinary software tests are insufficient: "Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures." [2]

This guide is the method, in the order the work is done, and each step has its own page. It covers the output of an AI product such as a support assistant or a document search. Five related subjects have their own guides: the business reasons in why AI evals matter, testing AI-written code in eval-driven development, grader design with a decision model in evals with Jev, products that take actions in AI agent evals, and public model scores in AI benchmarks vs your own evals.

The method in order

Step	The question it answers	Where it is explained
1. Know the three kinds of eval	Which check can decide this question at the lowest cost	The three kinds of LLM eval
2. Read real outputs	How this product fails, and how often	Error analysis
3. Build the test set	Which cases every version is run on	How to build an LLM eval dataset
4. Size the set	How far the pass rate can be trusted	How many test cases an LLM eval needs
5. Choose a score for each failure type	What is counted, and where that count misleads	LLM eval metrics
6. Split a search-based product in two	Whether the search step failed or the answer did	RAG evaluation
7. Check the human labels	Whether two reviewers agree	Human review and reviewer agreement
8. Keep measuring after release	What changed once real users arrived	Offline evals vs online evals

Each section below gives the main point of one step, with one fact from a named source. The linked pages hold the detail.

Three kinds of eval: code checks, human review and model graders

Every eval is one of three kinds. Anthropic's documentation names them code-based grading, human grading and LLM-based grading [3]. A code check is a short program that returns pass or fail. Human review is a person reading the output. A model grader, often called LLM-as-a-judge, is a second model that returns a verdict.

Each has its use. A code check costs only computer time and gives the same verdict on every run, so it suits any question with one right answer: the reply contains the order number, or the date matches the record. Human review can judge anything a competent reader can judge, and it is slow. A model grader is fast on questions of judgment, and its verdicts count only after they have been compared with human labels.

One measurement of model graders comes from Zheng and twelve co-authors in 2023. On 80 chat questions, GPT-4 as grader agreed with human labelers on 85 percent of the votes that were not ties, while the labelers agreed with each other on 81 percent [4]. The same paper found that when two answers swapped places, GPT-4 kept its verdict in 65.0 percent of cases [4]. A grader whose verdict depends on the order of the answers has to be checked on your own outputs. A related rule is argued in never let the model grade its own work.

The rule, in the words of Anthropic's documentation, is to "choose the fastest, most reliable, most scalable method" [3]. Use the cheapest kind that can decide the question. The three kinds of LLM eval compares them in a table and lists the measured faults of model graders.

Read the outputs before choosing a metric

The first piece of work is reading. Take a random sample of real outputs, write a short note on each one that failed, group the notes into failure types, and count each type. Only then decide what to measure. This step is called error analysis.

Hamel Husain and Shreya Shankar, who teach a course on evals, recommend reviewing at least 100 traces and stopping when new traces show no new failure type [5]. A trace is the full record of one request. The number is their own working guide, with no dataset behind it.

The research behind the step is a 2024 study by Shankar and four co-authors. In a qualitative study (one that observes what people do and reports it in words) with nine industry practitioners, they found that people changed their grading criteria as they graded more outputs [6]. They named this criteria drift, meaning the criteria change as the reading goes on: "users need criteria to grade outputs, but grading outputs helps users define criteria" [6]. Criteria written before any reading therefore miss failures that the reading would have shown.

A ready-made score skips this step and measures a property somebody else chose. OpenAI's documentation lists "Overly generic metrics" among common mistakes in eval design [2]. Error analysis: read the outputs before you choose a metric gives the four steps, with a worked table of failure types and the check each one leads to.

Build the test set from real usage

A test set is the list of cases that every version of the product is run on. Each case holds an input, the context the model was given, the expected behaviour, and tags that say which failure type it tests. A set whose expected answers a person has confirmed is often called a golden dataset.

Cases should come from what users do. OpenAI's documentation lists the possible sources: "Consider synthetic eval data, domain-specific eval data, purchased eval data, human-curated eval data, production data, and historical data." [2] Synthetic data means cases written by a model. They help cover rare requests, and Husain and Shankar state their limit: "Synthetic data cannot tell you how common a failure is in production." [5]

On size, no vendor page read for this guide on 30 September 2026 states a minimum. The numbers that exist are suggestions from named people and companies. Anthropic's engineering post of 9 January 2026 says 20 to 50 simple tasks drawn from real failures are enough to begin [7]. Husain and Shankar write: "A purpose-built eval set often grows to 100 or more examples." [5]

Two more rules apply. Include cases where a behaviour should happen and cases where it should stay absent [7]. Keep part of the set away from the people who write the prompts, the written instructions given to the model, so that each prompt is tested on cases its writer never saw. How to build an LLM eval dataset from real usage covers sources, synthetic cases, personal data and versioning.

Size the set for the difference you need to detect

A pass rate from a small set fits a wide range of true values. Evan Miller's 2024 paper on eval statistics gives the formula for pass or fail scores. The standard error measures how far the pass rate would move if the test were repeated on a fresh set of cases. It is the square root of p × (1 - p) / n, where p is the pass rate and n is the number of cases, and the 95 percent confidence interval is the pass rate plus and minus 1.96 standard errors [8]. A confidence interval is the range in which the true pass rate plausibly lies.

Cases	Standard error	95 percent interval for a 90 percent pass rate
100	0.0300	84.1 to 95.9 percent
1,000	0.0095	88.1 to 91.9 percent

The working for the first row: 0.9 × 0.1 / 100 = 0.0009, the square root of 0.0009 is 0.03, and 1.96 × 0.03 = 0.0588, which is 5.88 percentage points on each side of 90. The second row uses the same steps with n = 1,000.

A version that scores 92 percent on 100 cases is inside the interval for a 90 percent pass rate, so the 2 point difference could be chance. Miller recommends comparing two versions on the same cases and analysing the difference case by case, and answering each case several times when the output varies between runs [8]. The paper's own example arrives at 969 cases to detect a 3 point difference, under assumptions the paper itself calls fictional [8]. How many test cases an LLM eval needs has the full table and the paired comparison.

Choose a metric for each failure type

A metric is the rule that turns an output into a number. The main families are exact match and other checks in code, word-overlap scores such as BLEU, similarity scores that compare meaning, rubric scores given by a person or a model, and pairwise comparison, where a judge is shown two outputs and picks the better one. A rubric is a written list of criteria.

Each family was built for one purpose. BLEU was published in 2002 by four researchers at IBM to score machine translation against human reference translations, by counting shared word sequences [9]. A 2018 review by Ehud Reiter covered 284 correlations, each a measured match between BLEU and human ratings, reported in 34 papers. It concluded that the evidence "does not support using BLEU outside of MT, for evaluation of individual texts, or for scientific hypothesis testing" [10]. MT is machine translation. A support assistant's reply has no reference translation, so a BLEU score says little about whether the reply is correct.

For judged questions, Husain and Shankar prefer pass or fail to a 1 to 5 scale: "Binary evaluations force clearer thinking and more consistent labeling." [5]

Our position is that a metric is chosen for one failure type at a time. One overall score hides the failure types inside an average. In an invented illustration, a support assistant can average 4.3 out of 5 for helpfulness while one reply in ten gives a wrong delivery date. LLM eval metrics has a table of what each metric counts and when it fails.

Test the search step and the answer separately in a RAG system

RAG stands for retrieval-augmented generation: a search step fetches documents and the model writes its answer from them. The name comes from a 2020 paper by Patrick Lewis and eleven co-authors [11].

Two different things can fail. The search step can fetch the wrong passages, or the model can answer wrongly from the right ones. A single score on the final answer cannot tell you which of the two happened, so each step gets its own test.

For the answer, the Ragas paper of 2023 defines three measures that need no reference answer written by a person: faithfulness, answer relevance and context relevance [12]. Faithfulness asks whether the answer is supported by the fetched text. A model splits the answer into statements and checks each one against that text, and the score is the number of supported statements divided by the total number of statements [12]. An answer with 10 statements, 8 of them supported, scores 8 / 10 = 0.8.

The paper tested the measures on WikiEval, a set built from 50 Wikipedia pages and labelled by two people. Agreement with those human labels was 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance [12]. The authors make the Ragas tool, and they name context relevance as the hardest of the three to evaluate [12]. RAG evaluation explains each measure and maps what the user sees to the step that failed.

Human labels set the limit for every other measure

A model grader is checked against labels given by people, so the grader can only be as accurate as those labels. Good labelling has four parts: a written guideline, a pass or fail label with a written reason, two people labelling the same cases independently, and a measure of how often they agree.

Plain percent agreement overstates how well two reviewers agree, because two people who both pass most cases will often agree by chance. Cohen's kappa corrects for that. As Mary McHugh's 2012 review prints it, kappa = (Pr(a) - Pr(e)) / (1 - Pr(e)), where Pr(a) is the agreement observed and Pr(e) is the agreement expected by chance [13]. McHugh cites a study in which percent agreement was 94.2 percent and kappa was 0.555 on the same data [13].

The usual names for kappa values are the bands proposed by Landis and Koch in 1977, in which 0.41 to 0.60 is "Moderate" and 0.61 to 0.80 is "Substantial". We read those bands in a secondary source, a 1998 technical report by Khaled El Emam, which also records that Landis and Koch conceded their benchmark is arbitrary [14]. Treat the bands as a convention.

When two reviewers disagree, correct the guideline and label again. Husain and Shankar add a rule for the process: reviewers label the same examples independently before they discuss them [5]. Human review in LLM evals works through a kappa calculation and the number of labels needed to check a model grader.

Keep measuring after release

An offline eval runs a fixed set of cases before release. An online eval scores live traffic after release. LangChain's documentation for its LangSmith product defines the first this way: "Offline evaluations target examples from datasets: curated test cases with reference outputs that define what "good" looks like." [15] It defines the second as evaluation of "real production traces without reference outputs" [15].

Each one finds a different class of problem. The offline set shows whether a change made known cases better or worse, and it holds only the requests somebody put into it. Online measures reach the new requests, and they report a problem after users have already seen it. The change that Chen and co-authors measured in GPT-4 is the reason to rerun the offline set on a schedule, including in months when your own code stayed the same [1].

The two form a cycle. A failure found online becomes a new offline case, the offline run confirms the fix, and the online measure confirms it on live traffic [15]. Husain and Shankar suggest reading 100 or more fresh traces in each review cycle of 2 to 4 weeks [5].

Tools change as well. OpenAI's documentation, read on 30 September 2026, says of its hosted Evals platform: "Evals will become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026." [2] Keep your cases and labels in files you own, so that a tool closing takes none of them away. Offline evals vs online evals compares the two and lists the open-source eval tools with their licences.

How Reveneau applies this method

All of Reveneau's code is written by AI, and every change must pass a large eval suite before release. We write that suite from the specification, the written description of what the product must do, before the code exists. For an AI product, we build the suite by the method in this guide. We read real outputs with a person on the client's side who knows the correct answers, and we write the failure types into the specification. We score each failure type with the cheapest of the three kinds of eval that can decide it, and we report every pass rate with its interval and the number of cases behind it.

Reveneau grades the judged checks in its suite with Jev, TypeSafe AI's decision model, which returns a probability for a question written in advance. On our own suite the run is ten times faster than with our previous language-model grader. We use AI instead of hiring more engineers, so a build takes a small team, and that saving goes into the client's price. Reveneau, as a company, takes responsibility for the whole project through production and after release, and that includes the measuring after release. Our AI development service page describes the work.