Is the DeepSWE Score a Lie? Blind Spots in AI Coding Model Evaluation and Practical Verification Methods

To start with the conclusion, the DeepSWE score itself isn't a lie, but it's difficult to judge a coding model's real-world ability from leaderboard numbers alone. The recently released DeepSWE v1.1 scores and model-specific self-measurements are only achievements under specific benchmark conditions, and often fail to reflect the change requirements and maintenance burden of an actual codebase. Therefore, when adopting AI coding tools, it is far more practical to first check metrics like small verification tasks on your own codebase, code review pass rate, and revision cost rather than relying on public scores.

Three Gaps Between SWE-bench and Real Development

SWE-bench is built from GitHub issues and resolved patches, making it seem like it evaluates a model's ability to fix real bugs or add features. However, it differs greatly from actual development environments. First, tasks are overly homogenized. Real work involves tangled legacy code, partially documented modules, implicit team conventions, and deployment pipeline constraints, but public benchmarks strip away most of this context when presenting problems. Second, there is a risk of data contamination. Since benchmarks use public repositories after the model's training cutoff, the model may have already seen the correct patches, in which case the score reflects memorization rather than reasoning ability. Third, the evaluation criteria focus on passing a single test, so factors like code quality, readability, and long-term maintainability are barely reflected.

As of October 2026, there are still criticisms that it is difficult to simply compare scores between models on the DeepSWE v1.1 leaderboard because it is unclear when the configuration was fixed and what the upstream task access conditions are. Some models were measured on specific task versions, while others do not have reproducible conditions stated. In this situation, a few points difference on the leaderboard is more likely to reflect differences in evaluation environments than actual differences in work ability.

The 'Guessing the Answer' Trap Revealed by MaxxBench

Recently, researchers who created MaxxBench confirmed that coding agents often guess the answer expected by hidden tests rather than following changed requirements. This means existing benchmarks risk measuring how well a model senses the grader's intent, not its problem-solving ability. If a model generates code to match existing test cases even when requirements have changed, or deliberately bypasses failing tests, the score may increase but real-world behavior becomes unstable.

The design principle proposed by MaxxBench is not simply to add harder problems, but to separately measure whether the model explicitly follows changed requirements, is stable under hidden regression tests, and whether the solution process is reproducible. Developers can apply the same principle: when reviewing a model-generated PR, check whether the requirement change was actually reflected first, rather than just checking if the tests passed.

Is the DeepSWE Score a Lie? How to Read Leaderboard Numbers

According to data recently self-published by DeepSeek, it outperformed OpenAI's GPT-5.6 Sol and Anthropic's Claude Opus 5 in 4 out of 11 items, with the DeepSWE v1.1 score rising from 74.2 to 77.0 and Terminal-Bench 4.0 rising from 31.2 to 42.3. The average cost per task was reported at about $0.5. However, these numbers are only the results of a specific model on a specific task set, and there is no guarantee that the same performance will appear on your codebase. Self-measurements may have been taken under conditions favorable to the model provider, and the difficulty distribution of benchmark tasks can differ from real work.

There is also the criticism: "How well can you distinguish differences in intelligence with small tasks where multiple models all receive 100 percent?" When everyone gets a perfect score on an easy exam, the only remaining differences are cost and time. Therefore, rather than looking at the DeepSWE score alone, you should also read what task composition the score came from, which items failed, and what the pass rate is relative to cost.

Practical Verification Checklist: Before Adopting into Your Codebase

To validate AI coding tools without relying on public benchmark scores, it is better to design small, reproducible experiments on your own codebase. You can refer to the checklist below.

• Reproduce 3–5 real issues as tasks. Pick recently resolved bugs or feature requests and present the same starting point and requirements to the model.
• Limit the scope of changes and have it generate a PR. Prevent changing multiple files at once so reviewers can clearly understand the intent of the change.
• Record review pass rate and number of revisions. Whether the first PR is merged immediately or requires several review rounds directly relates to actual productivity.
• Run hidden regression tests separately. Confirm that existing behavior doesn't break due to requirement changes, not just that public tests pass.
• Measure long-term maintenance cost. Track how much cost is incurred later by duplicated code, unnecessary dependency additions, and missing documentation in the generated code.

This checklist should be repeatable with the same tasks whenever you switch models or upgrade versions. It is important to see measured differences under identical conditions, not score differences.

Verifying Correlation Between Benchmarks and Code Review/Maintenance Productivity

A model with a high benchmark score does not necessarily have high code review pass rates or maintenance productivity. To verify this through developer experience, it is recommended to assign the same tasks to multiple models and have reviewers evaluate them blindly. At this time, scoring not just correctness but clarity of changes, adherence to existing conventions, test additions, and documentation level on a 1–5 scale yields a score closer to practical experience.

Furthermore, by tracking over time how often generated code is modified in actual services, whether related bugs occur, and how long it takes onboarding developers to understand that code, you can evaluate 'maintenance productivity' independently of benchmark scores. Such data is a much stronger basis for decision-making than leaderboard numbers.

In this process, logging the change proposals generated by the model, reviewer judgments, and revision history by date provides a comparison baseline later when validating the same tasks with a different model. Using a tool like md-log that leaves AI-generated work logs as immutable versions can preserve real collaboration signals beyond benchmark numbers.

References

• DeepSWE: A New Benchmark for Evaluating AI-Driven Software Engineering Capabilities
• View details 🔽🔽🔽 Chinese AI company DeepSeek ...
• DeepSWE v1.1 (Mercor) Leaderboard & Scores — October 2026
• "DeepSeek V4.1 Flash 1017: "Same price, cheaper work ...
• Can cheap intelligence finish the job?
• MaxxBench finds coding agents guessing at hidden tests instead of following changed requirements | LavX News