Real-SWE: Coding Agents Score 38.8% on Private Code

Specific Labs published Real-SWE on September 12, 2026, a coding-agent benchmark assembled entirely from private production codebases that the models have almost certainly never seen. The top score on the board is 38.8%, posted by Claude Fable 5.1 running inside Claude Code. On SWE-bench Verified, the public benchmark the industry quotes in launch posts, the same Claude generation sits at 95%.

That gap is the story. It is not a rounding difference or a harness quirk. It is roughly 56 percentage points between what a coding agent scores on public GitHub issues and what it scores on the kind of code most people actually get paid to work on.

What Real-SWE actually measures

Real-SWE is built by Specific Labs, a Y Combinator F25 company, from codebases licensed directly from three operating businesses: a Luma and Partiful competitor with more than 200,000 users and a top-100 App Store ranking, a consumer fintech platform that processes over 100,000 bank statements, and enterprise AI sales tooling. Every task is lifted or adapted from a ticket an engineer on that team actually closed.

The methodology is stricter than the headline number suggests. Each model runs eight independent attempts per task, and the reported resolution rate is pass@1 averaged across those eight runs with 95% confidence intervals. Agents work inside an isolated sandbox. Verifiers are drawn from the repository's existing test suites, sometimes used verbatim, and are injected only at grading time so the agent cannot read the test and write to it.

The authors also insist on scoring model-and-harness combinations rather than models in isolation, on the grounds that this reflects how engineers actually work. That decision turns out to matter more than it sounds.

The full leaderboard

Note the fourth row. GLM 5.3, an open-weight model, scores 28.8% when it is driven by Claude Code, ahead of both Grok 4.6 in Grok Build and Muse Spark 1.3 in Muse Code. The harness is doing real work. A mid-tier model in a strong agent loop beats a stronger model in a weaker one.

Why the scores collapse

The authors give a blunt explanation: "99% of tokens in real-world enterprises are hidden away from the frontier models." Private code is, by definition, out of distribution. The conventions, the internal libraries, the accumulated business logic, none of it appears in pretraining data, and none of the fixes are sitting in a public pull request the way a resolved GitHub issue is.

This is also the cleanest answer available to the contamination question that hangs over every public leaderboard. When solutions to SWE-bench Verified tasks are indexed on the open internet, a high score is genuinely ambiguous: it could be reasoning, or it could be recall. Real-SWE removes the ambiguity by removing the corpus.

The difficulty is not that the instructions are unusually terse. The median Real-SWE instruction runs 1,742 characters, which puts it between DeepSWE at 1,975 and FrontierSWE v2 at 992, and nowhere near Terminal-Bench at 31,584. Agents are getting a normal amount of context. As the authors put it, the hard part is "triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns."

Where the agents actually fail

Specific Labs classified every failed run into five categories, and the spread across models is wide:

• Missed requirement (28.3% to 67.2% of failures): the agent omitted behavior the ticket required.
• Integration error (8.2% to 49.1%): the change did not connect correctly to the surrounding systems.
• Unverified assumption (10.9% to 43.3%): the agent built on a guess it never checked.
• Regression (0% to 10.9%): existing functionality broke.
• Wrong file (0% to 7.0%): the edit landed in a code path nothing calls.

Missed requirement dominating the taxonomy is the most useful finding in the whole report, because it is not a coding failure. The models can write the code. They are losing on comprehension: reading a terse ticket against an unfamiliar system and working out the full set of things that ticket implies. Regression and wrong-file errors, the failures people most often fear when handing work to an agent, are comparatively rare.

What this changes for picking a coding agent

Three things follow directly from the data.

First, treat public benchmark scores as a ceiling, not a forecast. A 95% SWE-bench Verified number does not predict 95% on your repository, and the honest planning assumption for unfamiliar proprietary code is closer to one task in three.

Second, evaluate the harness, not just the model. The GLM 5.3 result shows the agent loop is a first-class variable, so a bake-off that only swaps model names is measuring the wrong thing. If you are comparing options, run the same tickets through Claude Code, Codex CLI and Gemini CLI rather than through one harness with different models plugged into it.

Third, spend your review attention on requirements rather than syntax. Given that missed requirements cause up to 67.2% of failures, the highest-leverage habit is writing the acceptance criteria explicitly into the prompt and checking the diff against that list, instead of reading the code for correctness and assuming scope was understood.

The caveats that matter

The published sample is ten tasks, eight runs each, for 80 rollouts per model. That is small. Specific Labs reports 95% confidence intervals precisely because a ten-task sample carries wide ones, and the gap between the second and third place finishers is well inside the range where the ordering could move. Read the ranking as a rough tier list, not a precise ordering.

The full benchmark's total task count is not disclosed, only the sample. The company sells software engineering tooling, so this is vendor-published research rather than independent or peer-reviewed work, and it has not been replicated. Specific Labs offers sample task access on request, which is the only route to checking the tasks yourself.

None of that undermines the central finding. The direction and rough magnitude of the public-to-private gap are consistent across all eight models tested, and a sampling artifact that large would be a remarkable coincidence.

Frequently asked questions

What is Real-SWE?

Real-SWE is a coding-agent benchmark released by Specific Labs on September 12, 2026. Its tasks come from production codebases licensed from three private companies, so neither the code nor its fixes appear on the public internet.

Which coding agent scored highest on Real-SWE?

Claude Fable 5.1 running in Claude Code, at a 38.8% resolution rate. GPT-6 Astra in Codex CLI took second at 33.8%, and Gemini 3.8 Flash in Gemini CLI third at 31.2%.

Why are Real-SWE scores so much lower than SWE-bench scores?

Public benchmarks draw on open-source repositories whose issues and fixes are in the training data, so a high score may reflect recall as much as reasoning. Real-SWE uses private code the models have not seen, and adds business logic and cross-system triage that open-source issue tickets rarely require.

Does the harness matter or just the model?

The harness matters measurably. GLM 5.3 driven by Claude Code scored 28.8%, beating Grok 4.6 in Grok Build and Muse Spark 1.3 in Muse Code, both at 23.8%. Benchmark comparisons that hold the harness constant and swap only models are measuring an incomplete system.

What do coding agents get wrong most often?

Missed requirements, which account for 28.3% to 67.2% of failures depending on the model. Integration errors and unverified assumptions follow. Outright regressions and edits to the wrong file are rare, at 10.9% and 7.0% of failures at worst.

How reliable is the Real-SWE leaderboard?

Treat it as directional. The published sample covers ten tasks at eight runs per model, the full task count is undisclosed, and the research is vendor-published rather than independently replicated. The large gap versus public benchmarks is consistent across every model tested, but the exact ordering of adjacent models is not firm.