SWE-bench explained: what a coding benchmark score means

In October 2023 the best model tested on SWE-bench fixed 1.96% of the tasks [1]. On 23 February 2026 OpenAI wrote that the top score on the most quoted version of the benchmark had reached 80.9%, and in the same post it said it had stopped reporting that score and recommended that other model developers stop too [6]. Both facts describe the same test. This page explains what the tasks are and what the percentage counts, from the benchmark's own paper and project site, then gives the published criticism with its exact figures. It belongs to our guide to AI benchmarks vs your own evals.

What SWE-bench contains

A benchmark is a fixed, public set of tasks that many AI models are scored on, so that their results can be listed in one table. SWE-bench is a benchmark for coding work. Its tasks come from GitHub, a website where software teams store their code. On GitHub a written bug report or request is called an issue, and a proposed change to the code is called a pull request.

The SWE-bench paper, first published in October 2023, describes 2,294 software engineering problems drawn from real issues and their matching pull requests across 12 popular repositories written in Python, a programming language [1]. A repository is one project's complete folder of code together with its history. The authors started from 90,000 pull requests and filtered them down to the 2,294 tasks [1].

For each task the model receives the project's code and the text of the issue. Its job, in the paper's words, is "editing the codebase to address the issue" [1].

How a SWE-bench task is graded

Software projects include tests: small programs that run the code and check that it behaves as expected. SWE-bench uses those tests to grade the model. The model's change is applied to the project, the tests for that task are run, and the task counts as solved when they pass. OpenAI's description of the benchmark calls these unit tests, says they are not shown to the model, and says they mark a solution as correct or incorrect [2].

The published number is called "% Resolved", which the project site defines as "the percentage of task instances solved" [3]. A score of 50% on a 500-task set means that 250 tasks had passing tests after the model's change. One task on a 500-task set is worth 0.2 percentage points (1 divided by 500, written as a percentage).

Grading by tests has two consequences, and both appear in the audits further down this page. A change that passes weak tests counts as correct even when a reviewer would reject it. A correct change fails when the tests demand one particular way of writing the fix.

The versions of SWE-bench and what each one changed

"SWE-bench" in an announcement can mean any of seven sets. Each size below is the one its source states, read on 30 September 2026.

Version	Size	Who made it	What changed
SWE-bench	2,294 tasks	Jimenez and co-authors, 2023 [1]	The original set, from 12 Python repositories
SWE-bench Lite	300 tasks	The SWE-bench team [3]	A smaller set, "curated for less costly evaluation"
SWE-bench Verified	500 tasks	OpenAI with the SWE-bench team, August 2024 [2][3]	Every task screened by human developers
Bash Only	500 tasks	The SWE-bench team [3]	The Verified tasks, with every model run in one fixed setup
SWE-bench Multilingual	300 tasks	The SWE-bench team [3]	42 repositories across 9 programming languages
SWE-bench Multimodal	617 tasks (paper), 480 (site)	Yang, Jimenez and co-authors, 2024 [10]	Tasks from 17 JavaScript libraries, each with an image
SWE-Bench Pro	1,865 tasks	Scale AI, September 2025 [8]	A separate benchmark built from 41 repositories

The Multimodal paper counts 617 task instances [10] and the project site lists 480 [3], so quote each figure with its source. SWE-Bench Pro is absent from the SWE-bench project site. Its authors split it into a public set from 11 repositories, a held-out set (tasks kept back from the public) from 12 repositories, and a commercial set from 18 privately owned repositories [8].

Why SWE-bench Verified was made

OpenAI published SWE-bench Verified on 13 August 2024. Everything in this section is OpenAI's own account of its own work [2].

OpenAI says it worked with 93 software developers experienced in Python, who screened 1,699 random samples from the SWE-bench test set by hand. Each sample was labelled three times by separate people, and the most severe of the three labels was kept. The screening flagged 38.3% of samples for a problem statement that left out information a developer would need, and 61.1% for unit tests that could mark a valid solution as incorrect. In total 68.3% of the screened samples were removed. The Verified set is 500 of the samples that passed [2].

OpenAI reports that its GPT-4o model reached 33.2% on Verified against 16% on the original SWE-bench on the best-performing scaffold [2]. A scaffold is the program wrapped around the model that gives it tools and runs its steps one after another. Dividing 33.2 by 16 gives 2.075, so the same model scored more than twice as high after the tasks were cleaned.

What a SWE-bench percentage depends on

Three things change the number besides the model's ability.

The version. A score on Lite, Verified or the original set is a score on a different list of tasks.

The scaffold. OpenAI's 2024 post says GPT-4 scored 2.7% on SWE-bench Lite with an early scaffold and 28.3% with one called CodeR [2]. That is the same model, with a result 10.48 times higher (28.3 divided by 2.7). The Bash Only view on the project site exists for this reason: it puts "every model in the same mini-SWE-agent environment" [3]. A leaderboard, the public ranking table of scores, ranks a model together with its scaffold.

The date. On 5 August 2024 the top agents, meaning a model together with its scaffold, scored 20% on SWE-bench and 43% on Lite, according to the leaderboard as OpenAI quoted it [2]. For February 2026 two sources give two figures for Verified. Stanford's AI Index 2026 gives 76.8% for the leader as an approximate figure, with several other models between 70% and 76% [11]. OpenAI's post of 23 February 2026 says the best published score moved from 74.9% to 80.9% in the six months before it [6].

The settings a vendor chooses for a published score are the subject of how to read a model vendor's benchmark claims.

What the published audits found

Three pieces of criticism give exact figures, and each has a limited scope.

SWE-Bench+, October 2024. The authors read by hand the patches that one agent, SWE-Agent with GPT-4, had passed. A patch is the set of code changes a model proposes. They report that 32.67% of the successful patches had the solution given in the issue report or its comments, and that 31.08% passed because the tests were too weak to check the patch. With those cases removed, the agent's score fell from 12.47% to 3.97%. They add that over 94% of the issues were created before the models' knowledge cut-off dates, the dates after which a model has no training text [4]. These are shares of one agent's successful patches, and this page relies on the paper's abstract.

The SWE-Bench Illusion, June 2025. Given only the issue text, models named the file that held the bug with up to 76% accuracy on SWE-bench tasks, against up to 53% on repositories outside the benchmark. Word-for-word overlap with the reference fix (the human-written change stored as the correct answer), measured in runs of five consecutive pieces of text, reached up to 35% on SWE-bench Verified and the full set, against up to 18% on other benchmarks [5]. The authors read the gap as "possible data contamination or memorization" [5]. Contamination means the test material was in the text a model learned from. Our page on benchmark contamination covers the wider evidence.

OpenAI's audit, 23 February 2026. OpenAI audited 138 Verified problems that its o3 model failed to solve consistently over 64 runs, with at least six experienced engineers reviewing each one. It found that 59.4% of the 138 had material faults in the tests or the description: 35.5% had tests that demand one particular way of writing the code, 18.8% had tests that check for behaviour the description never asked for, and 5.1% had other faults [6]. The 138 problems are 27.6% of the 500 and were chosen because a model failed them, so 59.4% is the fault rate of a hard subset.

OpenAI also says every leading model it tested could reproduce the original human-written fix or exact details of the task text for certain tasks, after probing GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview. Its conclusion is that improvements on Verified "increasingly reflect how much the model was exposed to the benchmark at training time" [6]. This is OpenAI's audit of a benchmark it helped build.

What happened to SWE-Bench Pro

In the February 2026 post OpenAI recommended that developers report SWE-Bench Pro until better benchmarks existed [6]. On 8 July 2026 it withdrew that advice [7].

OpenAI's July audit gives 30% as its approximate estimate of the share of SWE-Bench Pro tasks that are broken. Two review routes produced two counts on the 731-task public set: an automated analysis flagged 200 tasks (27.4%) and a human review, with five engineers per flagged task, identified 249 (34.1%). OpenAI also reports that leading models went from a 23.3% pass rate to 80.3% on that public set in eight months [7].

The authors of SWE-Bench Pro describe it as "a contamination-resistant testbed" [8]. A preprint (a paper posted before review by other researchers) dated 8 September 2026, read as an abstract, reports leaked reference solutions and task quality faults in SWE-Bench Pro and says some models score lower on its cleaned version [9]. The sources for this page include no reply from Scale AI to OpenAI's audit, so the July figures are one company's estimate of another company's benchmark.

Vendors still report the benchmark. Anthropic's system card for Claude Opus 5.5 is dated 22 September 2026, which is 76 days after OpenAI's withdrawal, and it reports a SWE-bench Pro result for Anthropic's own model [12].

What SWE-bench leaves out

Other languages. The original set and Verified are Python only [1]. When the Multimodal set, built from projects in the JavaScript programming language, was published in October 2024, the best system resolved 12% of them and the next best 6% [10].

Private code. Every task comes from a public repository. No task contains your codebase or its conventions.

Vague requests. Verified removed tasks whose description was incomplete. A real request often arrives incomplete, and working out what was meant is part of the job.

Work that has no test. A task can only be graded when tests exist for it. Choices about design, wording on a screen, or whether a feature should exist at all fall outside the score.

Large pieces of work. OpenAI labels 196 of the 500 Verified tasks, which is 39.2%, as fixes that take a person under 15 minutes, and 45 as taking over an hour [2]. A product is many such changes that have to agree with each other.

Does a high score mean a model can build your software

A SWE-bench score is evidence about one model, inside one scaffold, on bug fixes in public Python projects that have tests, on one date. It says little about your product, because none of your requirements were in the test.

The measurement that answers your question is an eval: a test you write for your own work. How to choose an AI model with your own evals gives the steps, and how to write your first eval suite covers the coding case. For a supplier's own suite, how to evaluate a vendor's eval suite lists what to ask.

How Reveneau applies this

All of Reveneau's code is written by AI, and every change must pass a large eval suite before release. We write that suite from the client's specification before any code exists. That order is the method described in our guide to eval-driven development.

SWE-bench tests a model on other people's repositories, against tests those projects happened to have. A Reveneau eval suite tests the change the client asked for, against checks written for that request. When a new model is announced with a higher SWE-bench score, the question we ask is whether it passes the suite for the client's project.

We grade the judged checks in that suite with Jev, TypeSafe AI's decision model, and on our own suite the run is ten times faster than with our previous language-model grader. Because AI does work that would otherwise need more engineers, a build takes a small team, and that saving goes into the client's price. Reveneau as a company takes responsibility for the whole project through production and after release. To see how this applies to your build, read about our AI development service.