# Why we no longer evaluate SWE-bench Verified

DevFeed: [Why we no longer evaluate SWE-bench Verified](<https://devfeed.tech/articles/why-we-no-longer-evaluate-swe-bench-verified-6722.md>)

Original publisher: [Read original article](<https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified>)

Published: 2026-02-23T11:00:00Z

Content type: article

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bug](<https://devfeed.tech/tags/bug.md>), [coding](<https://devfeed.tech/tags/coding.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [model](<https://devfeed.tech/tags/model.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [openai](<https://devfeed.tech/tags/openai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [research](<https://devfeed.tech/tags/research.md>), [software](<https://devfeed.tech/tags/software.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

OpenAI says SWE-bench Verified no longer reliably measures frontier autonomous software-engineering capability because many remaining tasks have flawed tests and the benchmark is contaminated by training exposure.

## Source excerpt

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.