AI benchmarks vs your own evals: how to read a model score

On 13 August 2024 OpenAI published SWE-bench Verified, a set of 500 coding tasks chosen after 93 software developers screened 1,699 samples by hand [1]. On 23 February 2026, 18 months later, OpenAI wrote that it had "stopped reporting SWE-bench Verified scores" and recommended "that other model developers do so too" [2]. Its stated reason was that higher scores on the test had come to reflect how much a model was exposed to the benchmark during training [2]. In the same post it recommended a replacement, SWE-bench Pro. On 8 July 2026 it withdrew that recommendation, after its own audit counted broken tasks in the replacement at 27.4% by one method and 34.1% by another [3]. Each of these statements is OpenAI's account of its own work, read from the Internet Archive copies of its posts.

A benchmark is a public, fixed set of questions or tasks that many models are scored on. The events above show what that means for a buyer: a benchmark score has an author, a date and a set of conditions, and all three can change. The score answers one question, "how did this model do on that test?". A buyer needs the answer to a second question, "how will it do on my work?". This guide explains the benchmarks you will see quoted, the three ways a score misleads, and how to run a small eval of your own to choose a model. An eval of your own is a test built from your product's real cases. Each section below gives the main point of one page and links to it.

The three ways a benchmark score misleads

Problem	What happens	One piece of evidence
The questions were in the model's training text	The model scores well on questions it has already seen	On 1,205 newly written arithmetic problems, some model families lost up to 8% accuracy against the public set they resemble [4]
Every strong model scores near the top	The gaps that remain are too small to rank models	As of early 2026 the 15 leading models on MMLU-Pro all scored above 87% [5]
The vendor chose the settings	One model gets different scores under different conditions	OpenAI reported GPT-4 at 2.7% and at 28.3% on one coding benchmark, depending on the program that ran the model [1]

The first four pages of the guide describe the tests. The next three take these problems one at a time, and the last page gives the method for deciding with your own data.

What an AI benchmark measures and what it leaves out

A benchmark has three parts: a fixed set of items chosen by its authors, a rule for marking each answer, and a public table of model scores. MMLU, a knowledge test from 2020, is a clear example. It has 15,908 multiple-choice questions across 57 subjects, each with four options, so guessing scores 25% [6]. A score of 90% on it is a true statement about those questions.

The difficulty starts when a reader treats the test score as a measure of the skill the test is named after. A review of 445 benchmarks for large language models (LLMs), carried out by 29 expert reviewers and presented at the NeurIPS 2025 conference, found that less than 10% used complete real-world tasks [7]. It also found that 16.0% used uncertainty estimates or statistical tests to compare results [7]. In plain words, most benchmarks test short questions in place of whole jobs, and most publish no margin of error, so a reader cannot tell whether a gap of one point between two models means anything.

A benchmark also leaves out everything particular to you: your documents, your users' wording, your rules for a correct answer, your limits on cost and response time. Your own eval contains exactly those things, and its cases stay private. What an AI benchmark measures and what it leaves out explains who makes benchmarks, how a score is produced, and how a public test differs from an eval you write yourself. If the word eval is new to you, start with what is an AI eval.

SWE-bench: what a coding benchmark score means

SWE-bench is the coding test quoted most often in model announcements. The original set has 2,294 tasks taken from real problem reports in 12 software projects written in the Python programming language, and the project site defines the score as "the percentage of task instances solved" [8]. By OpenAI's description, a task counts as solved when the model's change to the code makes the project's own tests pass [1].

The history of the Verified version shows how much the test itself can decide the score. When OpenAI and the SWE-bench team had people screen the tasks, 68.3% of the screened samples were removed, and 61.1% were flagged because their tests could mark a valid solution as wrong [1]. On the cleaned set of 500 tasks, OpenAI reported that its GPT-4o model scored 33.2%, against 16% on the original set [1]. That is 2.075 times the earlier score. Both figures describe the same model, so the difference comes from which tasks were in the test.

Sources also disagree on the current top score. OpenAI's February 2026 post gives 80.9% as the best result on Verified [2]. Stanford's AI Index 2026 gives 76.8% for the leader as of February 2026, a figure the report words as approximate [5]. Quote either one with its source and date.

A SWE-bench score counts tasks from other people's public Python projects that already have tests. It says little about private code, other programming languages, or a request that arrives without a clear description. SWE-bench explained covers every version, the grading method and the published critiques. For testing code written for you, see our guide to eval-driven development.

MMLU, GPQA and ARC-AGI: knowledge and reasoning benchmarks

Knowledge benchmarks ask exam questions. On MMLU, people recruited through Amazon Mechanical Turk, a website where people are paid to do small online tasks, with no special training, scored 34.5%, and the largest model tested in 2020 scored under 20 percentage points above guessing [6]. Once models score close to the maximum, the test's own mistakes matter. Researchers at the University of Edinburgh and other institutions re-checked 5,700 MMLU questions by hand and estimated that 6.49% of MMLU questions contain errors [9]. In the virology subject, 57% of the 100 questions they analysed contained errors [9]. The 6.49% figure is an estimate from a sample, and it limits what a score above 90% can tell you: part of the remaining gap comes from faulty questions.

Newer tests were built to stay hard for longer. GPQA uses science questions written by subject experts, ARC-AGI uses grid puzzles, and Humanity's Last Exam collects expert-written questions across dozens of subjects. Each has its own size and format. Scores on the newest tests rise fast: the AI Index reports that top accuracy on Humanity's Last Exam went "from under 10% to 38.3%" in a single year [5].

The point for a buyer is about fit. A multiple-choice knowledge score shows what a model can recall and reason about in an exam format. A product that answers customers from your documents depends on a different ability: finding the right passage in your text and staying inside it. Only a test on your documents measures that. MMLU, GPQA and ARC-AGI explained gives each benchmark's contents, scoring and known limits.

Agent benchmarks: GAIA, WebArena, tau-bench and METR time horizons

An agent is an AI system that takes actions through tools, such as searching a website or changing a record, over many steps. Agent benchmarks mark the end result. In tau-bench, a 2024 benchmark from the company Sierra, the agent talks to a simulated customer and uses tools under a written policy. The test then "compares the database state at the end of a conversation with the annotated goal state" [10]. In plain words, it checks whether the records ended up as they should. It has 165 tasks: 115 in a retail setting and 50 in an airline setting [10].

The tau-bench paper gives the fact that matters most for a buyer. Its authors report that agents built on the model gpt-4o succeeded on fewer than 50% of the tasks. When the same retail task had to succeed in 8 tries out of 8, the rate fell below 25% [10]. A single-run score does not show how often an agent fails when the same request comes in again, and a product receives the same request many times.

The other benchmarks in this group test different work. GAIA asks questions that need several tools to answer, WebArena sets tasks on working copies of websites, and METR's time horizon expresses a model's ability as the length of task, measured in the time a skilled person needs, that the model completes half the time. All of them run in a prepared test setting. Agent benchmarks explained gives the task counts, the human scores and the best model scores at publication. To test an agent of your own, use our guide to AI agent evals.

Benchmark contamination: the test questions were in the training data

A benchmark's questions and answers are public, so they can end up in the text a model is trained on. The model can then score well by repeating what it has seen. Researchers call this contamination.

The most direct way to measure it is to write new questions of the same kind and difficulty and compare. A team at Scale AI did that for a public set of school arithmetic problems. They wrote 1,205 new problems using human writers only and found "accuracy drops of up to 8%", with several model families showing the pattern across almost all model sizes [4]. The same paper reports that many models, including the leading ones, showed minimal signs of the problem [4]. Scale AI sells evaluation services, and its result is mixed.

Coding benchmarks have the same exposure, because their tasks come from public code. In its February 2026 post OpenAI wrote that every leading model it tested could reproduce the original human-written fix, or exact details of the problem statement, for certain SWE-bench Verified tasks [2]. The post names three models from three vendors and publishes no rate for any of them.

A buyer has one reliable protection. Cases taken from your own product have never been published, so no model has seen them. Benchmark contamination sets out each study, what was done and what was found, and describes the benchmarks that replace their questions on a schedule.

Benchmark saturation and Goodhart's law

A benchmark stops being useful when every strong model scores near the maximum. The AI Index calls this saturation, "where models reach scores so high that a test can no longer distinguish between them" [5]. The same report says that "Evaluations intended to be challenging for years are saturated in months" [5]. Its example is MMLU-Pro, a harder version of MMLU released in 2024. As of early 2026 the 15 leading models all scored above 87%, and the gap from first place to fifteenth was "just over 4 percentage points" [5]. A test can rank fifteen models inside a gap of that size only if its margin of error is smaller than the gap, and most benchmarks publish no margin of error [7].

The second half of the problem has a name from economics. Charles Goodhart was Chief Adviser to the Bank of England, and the anthropologist Marilyn Strathern restated Goodhart's idea in 1997 in the form most people quote: "When a measure becomes a target, it ceases to be a good measure" [11]. That wording is taken from a University of Cambridge page that cites both authors. A public benchmark is a target for every model vendor. Once a vendor's release is judged by a score, the vendor has a reason to tune the model toward that test, and the score rises faster than the skill.

The same law applies inside your own company. A team that adjusts its instructions against one fixed set of cases will end up fitting that set. Keep part of your cases unseen by the people who tune the product, and add new cases from real use. Benchmark saturation and Goodhart's law explains the signs that a benchmark has stopped separating models and how to protect your own set.

How to read a model vendor's benchmark claims

A benchmark number in an announcement depends on choices the vendor made: how many attempts the model had, which tools it could use, how long it could reason, which subset of tasks was run, and which program ran the model. That program has a name, scaffold: the software wrapped around a model that hands it tools and runs it step by step. OpenAI's 2024 post gives the size of the effect. GPT-4 scored between 2.7% and 28.3% on SWE-bench Lite depending on the scaffold [1]. The model was the same in both runs.

Public ranking sites have their own version of this problem. Arena is a site where people compare two unnamed models' answers and vote. A 2025 paper, The Leaderboard Illusion, reported that some providers test many private versions of a model before release, and it identified "27 private LLM variants tested by Meta in the lead-up to the Llama-4 release" [12]. Several of the paper's authors work at Cohere, which competes on the same ranking. Arena replied that the paper "contains several incorrect claims" and that its policy lets "any model provider" submit as many public and private variants as they would like [13]. Both sides agree on the practice that matters to a buyer: a vendor can test several versions privately and release the one that scored best.

How to read a model vendor's benchmark claims is a printable list of questions to ask about any published score. For claims about a supplier's private tests, see how to evaluate a vendor's eval suite.

How to choose an AI model with your own evals

Public rankings place the leading models close together. As of March 2026 the AI Index listed the top models of six companies between 1,424 and 1,503 rating points on Arena, the voting site described above, a spread of 79 points, and said the narrow gaps are "shifting competitive pressure toward cost, reliability, and domain-specific performance" [5]. Google Cloud's documentation says that results on your specific tasks give "valuable insights which cannot be derived from public leaderboards and general benchmarks" [14]. A leaderboard is a public table that ranks models by score, and that Google page describes a paid service.

Prices do separate the models. Take an invented task of 3,000 input tokens and 500 output tokens, where a token is a piece of a word and the unit vendors bill in. At the prices on OpenAI's own pricing page on 30 September 2026, that task costs $55.00 per 100,000 requests on the model gpt-6-luna and $5,500.00 on gpt-6-astra, a ratio of 100 [15]. Prices change, so date any price you quote. Whether the lower-priced model is good enough for your task is a question only your cases can answer.

The method has seven steps: write the task and what correct means, collect real cases, run every candidate the same way, score quality with cost per task and response time, read the failures, decide, and keep the set to run again when a model is replaced. Running 200 such cases on the higher-priced model costs $11.00 at the same published price [15]. How to choose an AI model with your own evals gives each step with the full price table. Building the case set is covered in our guide to LLM evals.

How Reveneau uses benchmarks and evals

At Reveneau all code is written by AI, and every change must pass a large eval suite, written from the specification before the code, before it is released. We treat model choice the same way. A public benchmark score helps us decide which models to try on a client's task. The choice itself comes from evals written for that client's product, with the client's definition of a correct answer, and the evals stay in the project so they can run again when a vendor replaces a model.

We grade the judged checks in our suite with Jev, TypeSafe AI's decision model, and on our own suite the run is ten times faster than with our previous language-model grader. We use AI instead of hiring more engineers, so a build takes a small team, and that saving goes into the client's price. Reveneau as a company takes responsibility for the whole project through production and after release. If you have a model decision to make and want it made on your own cases, our AI development service is where that work starts, or you can contact us.