Evaluation
SWE-bench
A benchmark that measures whether a model can fix real GitHub issues in open-source projects, verified by the project's own tests.
SWE-bench Verified, a human-checked subset, is the most cited test of practical coding ability. High scores require reading unfamiliar code and producing a working patch.
Frequently asked questions
What does SWE-bench mean?
A benchmark that measures whether a model can fix real GitHub issues in open-source projects, verified by the project's own tests. SWE-bench Verified, a human-checked subset, is the most cited test of practical coding ability. High scores require reading unfamiliar code and producing a working patch.