Specific Labs releases Real-SWE, a benchmark of proprietary code, and the top-scoring agent solves 38.8 percent of tasks

Specific Labs, a Y Combinator-backed company, has released Real-SWE, a coding-agent benchmark built on private codebases licensed from real businesses. The New Stack's Amanda Caswell wrote up the results on 14 September. The top score, from Claude Fable 5.1 running under Claude Code, was 38.8 percent. Every other agent tested scored below that.

What Real-SWE tests

The benchmark uses proprietary code from real companies, including a consumer product with more than 200,000 users and a fintech platform that has processed more than 100,000 bank statements. Specific Labs writes that the code and its solutions are not publicly available, which makes it less likely the tasks appeared in a model's training data. The company also states that 99 percent of tokens in real-world enterprises are hidden from frontier models.

Real-SWE solutions touch a median of 11 files, close to double the six-file median that The New Stack cites for FrontierCode and DeepSWE. Each model gets eight tries at every task, and each is tested with its own coding tool: Fable 5.1 with Claude Code, GPT-6 Astra with Codex CLI, and Gemini 3.8 Flash with Gemini CLI. That means the scores reflect the full setup rather than the model alone.

Where the scores land

Fable 5.1 on Claude Code led at 38.8 percent. GPT-6 Astra on Codex CLI followed at 33.8 percent, and Gemini 3.8 Flash on Gemini CLI at 31.2 percent. After that, the scores fell off: GLM 5.3 at 28.8 percent, Grok 4.6 and Muse Spark 1.3 tied at 23.8 percent, Kimi K3 at 18.8 percent, and GPT-5.6 Sol at 16.2 percent.

Individual tasks are harder than the averages suggest. Six of the ten tasks had success rates below 15 percent across all agents. A billing schedule migration ran at 14.1 percent, API token metering at 12.5 percent, S3 storage tracking at 10.9 percent, a linearizable scan at 4.7 percent, and a tax jurisdiction bug fix at 3.1 percent. No model solved an analytics stream reducer across 64 combined attempts. On the other end, Astra and Gemini both went eight for eight on a multi-region sweep and Fable solved seven of eight, yet all three failed every attempt at the linearizable scan.

The failure modes Specific Labs recorded are also worth knowing. Fable 5.1 most often missed requirements (36.7 percent of failures) or ran into integration errors (34.7 percent). Astra's failures split evenly between integration errors and unverified assumptions, both at 34 percent. Nearly half of Gemini 3.8 Flash's failed runs contained integration errors, and GPT-5.6 Sol made unverified assumptions in 43.3 percent of its failures.

What the benchmark does not settle

Real-SWE runs on ten tasks, which is a small sample, and Specific Labs cannot guarantee that no model has ever seen any of the code. It also cannot fully separate the model from its scaffolding: Fable 5.1 through Cursor rather than Claude Code would post a different number, and past benchmarks have shown the harness matters.

What the numbers do show is that solving a coding problem is different from finding a way through an unfamiliar production codebase, and that a top score on public benchmarks does not carry over cleanly. A team choosing a coding agent for its own repository has more reason to trust an eval it wrote itself than a public leaderboard, and a small internal harness of the team's own hard tickets will predict production behaviour better than SWE-bench Verified.

Source

Reporting: AI's best coding agent fails 60% of the time, and the data backs it up, Amanda Caswell, The New Stack, 14 September 2026. Primary source: Real-SWE leaderboard, Specific Labs.