SWE-bench Is Contaminated. Here Is How to Read Coding Benchmarks Now

In May the SWE-bench authors published SWE-bench Live, a variant that draws issues from repositories after each model's training cutoff and rotates its problem set monthly. The results reframed a year of progress.

Models that score in the high 70s on SWE-bench Verified score in the mid 50s on Live. The gap is consistent across providers, which rules out the simplest explanations. It is not that one lab overfit. It is that the whole field has been climbing a benchmark that partially leaked into training data.

What Contamination Actually Looks Like

Not deliberate cheating. SWE-bench Verified draws from public GitHub repositories. Those repositories, their issues, their pull requests and their commit histories are in every large web crawl. A model trained on a snapshot that postdates a given issue has, in some meaningful sense, seen the answer.

The model is not retrieving a memorised patch verbatim. It is doing something subtler and harder to detect: it has strong priors about what the fix for this specific issue looks like, which is not the same skill as fixing an issue it has never encountered.

The Practical Consequence

Benchmark deltas between model versions are less informative than they look. When a new release claims a jump from 71% to 79% on SWE-bench Verified, some of that is capability and some is a more recent training cutoff covering more of the benchmark's source repositories. You cannot separate the two from the outside.

We publish SWE-bench Verified numbers on our benchmarks page because they remain the most widely comparable figures available, but they should be read as a rough tier indicator rather than a precise ranking.

What To Trust Instead

1. Rotating benchmarks. SWE-bench Live, LiveCodeBench and similar suites that draw problems postdating training cutoffs. Lower absolute scores, much better signal.
2. Your own eval set. Twenty to fifty tasks drawn from your actual codebase, scored by whatever you actually care about. This is boring to build and worth more than every public benchmark combined.
3. Long-horizon task completion. Not "did it produce a correct patch" but "did it complete a four hour task without human intervention." Much harder to game, much closer to what you are buying.
4. Human preference at scale. Arena-style pairwise comparison has its own problems, notably a bias toward verbose confident answers, but it is not contaminated in the same way.

Building Your Own Eval Set

The minimum viable version takes an afternoon. Pick twenty closed issues from your own repository from the last three months. Write down what a correct fix looks like. Run each candidate model against each issue with your actual tooling and prompts. Score pass, partial, fail.

Twenty tasks is not statistically rigorous. It is also dramatically more predictive of how a model will perform on your work than any leaderboard, because it tests your codebase, your conventions and your definition of correct.

Budget the token cost first with our cost calculator. A twenty-task eval across five models runs somewhere between two and twenty dollars depending on which tier you are testing.

The Uncomfortable Version

Some meaningful portion of the "models are improving rapidly at coding" narrative is measurement artifact. Models are improving. The rate is probably slower than the benchmark curves suggest. Anyone making capacity or hiring decisions off benchmark extrapolation should discount accordingly.

That is not a reason for pessimism. Mid-50s on genuinely unseen software engineering tasks is still remarkable. It is just a different number than the one on the slide.