- Services
Long-horizon, enterprise-style coding tasks that take human engineers hours, not minutes.
SWE-Pro raises the bar past SWE-Verified. The tasks are larger, the codebases are bigger, the changes span multiple files, and the test suites are deeper. A successful run looks like a full pull request rather than a small patch. SWE-Pro measures whether a model and its agent harness can act like a junior engineer working on a feature for an afternoon.
Tasks ship with a repository snapshot, a description, and a hidden test suite. The agent makes changes, runs tests, and iterates. Scoring is the fraction of tasks that pass all hidden tests within a fixed step or token budget.
Benchmark data sourced from: Hugging Face, Arena.ai, Artificial Analysis.
| # | Model | Lab | Source | Score |
|---|---|---|---|---|
| 01 | Gemini 3 Flash | Closed | 71.2 | |
| 02 | Beam 1.0 501B A23B | Reflection | Open | 66.1 |
| 03 | GLM-5.2 | Z.ai | Open | 62.1 |
| 04 | Atria-Dawn-Preview | Shanghai AI Laboratory | Open | 59.6 |
| 05 | LongCat-2.0 | Meituan | Open | 59.5 |
| 06 | MiniMax M3 | MiniMax | Open | 59.0 |
| 07 | Kimi K2.6 | Moonshot AI | Open | 58.6 |
| 08 | GLM-5.1 | Z.ai | Open | 58.4 |
Ad Tools for AI BuildersAll the hardware and tools you need to run private AI at your office or at home.Shop Now | ||||
| 09 | Hunyuan Hy3 | Tencent | Open | 57.9 |
| 10 | Hy3 | Tencent | Open | 57.9 |
| 11 | GPT-5.4 | OpenAI | Closed | 57.7 |
| 12 | DeepSeek-V4-Pro | DeepSeek | Open | 55.4 |
| 13 | minimax-m2.5 | MiniMax | Open | 55.4 |
| 14 | Ring-2.6-1T | inclusionAI | Open | 53.8 |
| 15 | Qwen3.6-27B | Alibaba | Open | 53.5 |
4 model(s) with undisclosed parameter counts not shown. Most closed-source labs do not publish model size.
The tasks are longer and the codebases are bigger. A SWE-Verified issue typically needs a 5–50 line patch. SWE-Pro tasks routinely need hundreds of lines across several files, with careful test wiring.
For senior-engineer-level autonomy, yes. For day-to-day code completion, no. Pair it with a fast feedback benchmark like HumanEval++ or a real evaluation in your own repo.
Based on score correlations across our database.
