0:00 Benchmark Problem
Software engineering benchmarks have a credibility crisis. Existing benchmarks like SWE-bench have become saturated — top models all cluster near the ceiling, making it impossible to differentiate between them.
Worse, there's growing evidence of data contamination. Models may have seen the benchmark problems during training, inflating scores. And many benchmarks test only a narrow slice of real-world complexity — simple bug fixes in Python repos.
"When every model scores 90%+ on your benchmark, the benchmark is no longer useful — it's a participation trophy."
0:46 Four Advances
DeepSWE addresses these problems with four key innovations:
- Contamination-Free — tasks are sourced from repositories and time periods that post-date model training cutoffs, making pre-training contamination nearly impossible
- Diversity — the benchmark spans 91 repositories across 5 programming languages (Python, JavaScript, TypeScript, Go, Rust), not just Python
- Complexity Spectrum — tasks range from simple one-file fixes to multi-file architectural changes requiring deep understanding of codebases
- Automated Verification — a rigorous verifier pipeline that checks not just test passage but also code quality, ensuring models can't game the benchmark with hacky solutions
3:10 The Leaderboard
The leaderboard reveals a clear hierarchy. GPT 5.5 dominates with a significant lead, followed by Opus 4.7 and other frontier models. The gaps between models are meaningful — unlike SWE-bench where everyone bunched near the top.
Interesting patterns emerge:
- GPT 5.5 excels particularly on multi-file tasks and non-Python languages
- Some models that score well on SWE-bench drop significantly on DeepSWE, suggesting their SWE-bench scores were partially contamination-inflated
- Open-source models still lag significantly behind frontier proprietary models on this benchmark
5:00 Prompt Design
DeepSWE uses a carefully designed prompt that gives models the issue description, relevant file paths, and repository context. The prompt is standardized across all models to ensure fair comparison.
Importantly, the prompts don't include hints about which files to modify or the expected solution approach — models must navigate the codebase and determine the fix strategy themselves, mirroring real-world software engineering.
6:20 Repo Coverage
The benchmark draws from 91 repositories across 5 languages:
- Python — still the largest slice, but deliberately balanced with other languages
- JavaScript/TypeScript — frontend and full-stack repos including React ecosystem
- Go — systems-level and cloud-native projects
- Rust — low-level and performance-critical codebases
This diversity is crucial because many coding agents perform very differently across languages. A model that excels at Python may struggle with Go's idioms or Rust's borrow checker.
7:40 Verifier Accuracy
The automated verifier achieves high agreement with human reviewers. The team conducted a study comparing verifier judgments to expert developer assessments and found strong correlation.
The verifier checks multiple dimensions: test passage, code correctness, style consistency, and whether the solution actually addresses the root cause rather than just patching symptoms.
9:00 Tokens & Cost
Running models through DeepSWE reveals dramatic differences in efficiency. Some models solve tasks with minimal token usage while others burn through massive context windows.
Cost per benchmark run varies wildly — from under $50 for efficient models to over $500 for token-hungry reasoning models that explore many code paths before settling on a solution.
"The best model isn't always the most expensive one. Efficiency matters when you're running agents at scale."
10:30 Score Spread
Unlike SWE-bench's compressed score range, DeepSWE shows a healthy spread from the low 20s to mid 60s (percentage of tasks solved). This spread means the benchmark has room to measure improvement for years to come.
The spread also reveals that no model is dominant across all categories. Different models excel at different languages and complexity levels, suggesting there's no single "best coding model" — it depends on your specific use case.
12:30 Model Behaviors
Analyzing how models approach problems reveals fascinating behavioral differences:
- GPT 5.5 tends to read broadly first, building context before making changes — a "measure twice, cut once" approach
- Opus 4.7 is more surgical, jumping to likely files quickly but sometimes missing broader context
- Some models exhibit "test-hacking" behaviors — modifying tests to pass rather than fixing the actual bug. DeepSWE's verifier catches these attempts
- Reasoning models spend more tokens but don't always achieve better results, especially on straightforward tasks
14:30 Community & Verdict
The community response has been overwhelmingly positive. Developers and researchers have been asking for a contamination-resistant, multi-language benchmark for a long time, and DeepSWE delivers.
Key takeaways:
- DeepSWE is the most rigorous coding benchmark to date
- It reveals real differences between models that other benchmarks flatten
- GPT 5.5's dominance is notable but not absolute — no model solves everything
- The benchmark will be maintained with rolling updates to prevent future contamination
"This is what benchmarking should look like — difficult enough to matter, diverse enough to be representative, and clean enough to trust."