Specific Labs publishes Real-SWE, and every frontier model scores under 40 percent on private enterprise codebases

Specific Labs published Real-SWE this week, a coding benchmark built from tasks licensed from real companies' private production repositories. The Hacker News thread went up two hours before this item was written and carried 58 points and 43 comments. Every number below is Specific Labs' own reporting.

The lab tested eight frontier models. Fable 5.1 from Anthropic leads at 38.8 percent resolution, GPT-6 Astra from OpenAI is second at 33.8 percent, and Gemini 3.8 Flash from Google is third at 31.2 percent. GLM 5.3 scored 28.8 percent, Grok 4.6 and Meta's Muse Spark 1.3 tied at 23.8 percent, Kimi K3 reached 18.8 percent, and GPT-5.6 Sol came in last at 16.2 percent.

What the benchmark tests

Specific Labs writes on the page that every task "comes from a private production codebase that we licensed from a real-world company." The point of that framing is training data. The lab's own claim: "These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model."

The eight sample tasks the page walks through cover billing changes, tax handling, and customer management work that spans multiple services. Reference solutions edit a median of 11 files, against 6 for FrontierCode and DeepSWE. Prompt length is close to those two, at a median of 1,742 characters. Verifiers reuse or adapt each company's existing test suite. Real-SWE ships in Harbor format, and the page says sample tasks are available on request.

Where the models fail

Specific Labs groups the failures. The most common one across the leaderboard is what they call "missed requirement," which they define as "leaves out behavior the instruction requires." Rollout length also matters. The lab reports that 71.4 percent of runs shorter than 10 minutes failed, against 73.4 percent of longer ones. Estimated rollout cost across the tested models ranged from $2.50 to $6.96 per task.

The task-level range is wider than the leaderboard suggests. On "Multi-region sweep," models collectively resolved 67.2 percent of attempts. On "Analytics stream reducer," they resolved 0.0 percent.

What the numbers do not answer

The page does not state the total task count, so the leaderboard percentages come from a scope Specific Labs has not publicly quantified. It also does not say when the code was licensed, which matters for a benchmark whose whole pitch rests on out-of-distribution content. And an eight-model comparison from one lab is not an independent replication.

Public benchmarks that these same models report much higher scores on, including SWE-bench Verified and its private variants, are also selection biased: authors pick tasks with self-contained fixes and clear tests. Real-SWE takes the opposite selection, cross-file work with company-specific patterns, and finds numbers that are less flattering by a lot. A team pricing agent time against SWE-bench is pricing against the easier problem, and a team measuring review capacity against public numbers is doing the same. Real-SWE is a data point saying to widen the range and add slack.

The other useful reading, if the numbers hold up, is that the gap between the top and bottom model on Real-SWE is 22.6 percentage points. If private code separates models more than public code does, choosing the model matters more for real work than the public leaderboards suggest, and the sensible response is to run a paid pilot on one's own repository before committing to a subscription.

Source

• Real-SWE Benchmark, Specific Labs
• Hacker News discussion