What does it mean that SWE-Bench Pro tasks are broken?

It means the tasks can't be graded fairly, not that the benchmark is fake. Each task is scored by a hidden test suite, and OpenAI's audit found that up to 34.1% of the 731 public tasks have flawed grading: overly-strict tests that reject correct patches, low-coverage tests that pass wrong ones, and underspecified or misleading prompts. The headline resolve rate counts those tasks anyway, so the number mixes real signal with noise.

Why does benchmark task validity matter?

Because a benchmark's number drives real decisions — deployment, safety claims, and model-release comparisons. If a third of the tasks are invalid, the score is a false capability signal: you can't tell how much of a jump is the model improving versus grading that was never valid. Validity is the property that lets the number mean what people assume it means.

How do you check if a coding benchmark's tasks are valid?

OpenAI's method is a good template: flag likely-flawed tasks with an automated data-quality pipeline, investigate each with agents that read the instructions, tests, and failure traces, and run a separate human annotation pass that labels the flaws by type. As a general practice at smaller scale, you sample tasks by hand, re-run known-good and known-bad solutions to see whether the graders agree, and treat a benchmark like a measurement instrument you calibrate before trusting.

OpenAI finds about 30% of SWE-Bench Pro tasks broken — Benchmark task validity

TL;DR

What is it: OpenAI's audit "Separating signal from noise in coding evaluations" checks SWE-Bench Pro — a benchmark for longer-horizon coding agents — and surfaces the idea of benchmark task validity: whether each task actually measures what it claims, or is quietly broken.
Why it’s needed: A benchmark's headline number steers deployment decisions, safety claims, and model-release comparisons, so tasks that can't be graded fairly turn that number into a false capability signal — the failure the Evals & Diagnostics and Production Evals lessons exist to catch.
vs previous: Where the usual read trusts the pass rate — one percentage over all tasks — and sorts models by it, this audit checks the tasks themselves first and finds up to 34.1% with broken grading: overly-strict tests, underspecified prompts, or tests too weak to catch a wrong fix.

Jargon

SWE-Bench Pro: A benchmark of real software-engineering tasks for coding agents: given a repository and an issue, the agent must produce a patch that is graded by running a held-out test suite. OpenAI's audit covers its 731-task public split.
Resolve rate: The headline metric — the share of tasks the agent solves, judged by whether the hidden tests pass. It treats every task as if its grading were correct.
Hidden test: The test suite that decides pass or fail, which the agent never sees — the benchmark's answer key. See AI Agents → Evals & Diagnostics → Pass/Fail vs Score.
Overly-strict test: A hidden test that rejects a genuinely correct patch — for example, it demands one exact string or code path when several are valid. The result is a false fail that understates the model's ability.
Low-coverage test: A hidden test so weak it passes a wrong patch — it never exercises the case that would expose the bug. The result is a false pass that overstates the model's ability.
Underspecified prompt: A task whose description omits a requirement the hidden test silently enforces, so no agent could reliably know what "correct" means. It fails agents for reading the prompt as written.
Task (construct) validity: The measurement-theory question underneath all of this: does an item measure what it claims to? A related but distinct idea to predictive validity, which asks whether a whole ranking transfers; here the concern is whether each individual task is even gradeable.

The news. On July 8, 2026, OpenAI published Separating signal from noise in coding evaluations, an audit of SWE-Bench Pro, a benchmark built to test longer-horizon agentic coding. Combining an automated data-quality pipeline, Codex-based investigator agents, and a human annotation campaign with experienced engineers, they now estimate that a large share of the 731-task public split is broken: the automated pipeline flagged 200 tasks (27.4%), and human annotation identified 249 tasks (34.1%). Over the same benchmark, frontier resolve rates had climbed from 23.3% to 80.3% in eight months. Read the audit →

Picture handing an agent 731 coding problems and grading each one against a sealed answer key — a hidden test suite the agent never sees. On paper you now have a clean score: solve this many, fail that many, print the percentage. But a pass rate is only ever as trustworthy as the answer key behind it, and OpenAI's audit is the moment someone opens the keys and checks them. When they do, roughly one in three keys is wrong. Some reject a correct fix; some wave a wrong one through. The score was never lying about arithmetic — it faithfully counted passes. It just counted them against tasks that could not be graded fairly in the first place.

That gap is what benchmark task validity names. A benchmark is not a scoreboard handed down from nowhere; it is a pile of individual measurement instruments, one per task, and each is only as good as its hidden test. When a task's test is wrong, a correct or incorrect solution gets silently mislabeled — an overly-strict test fails a valid patch, a low-coverage test passes a broken one, and an underspecified prompt punishes the agent for something the description never asked for. None of these show up in the headline number. They are exactly the ways an eval quietly breaks while still producing a confident-looking percentage.

The climb came first. The frontier pass rate's jump from 23.3% to 80.3% in eight months, reported in the same audit, is large enough to check the tasks, not only the models. So instead of trusting the rise, OpenAI turned the tools on the benchmark. The audit re-graded the tasks themselves — an automated data-quality pipeline and Codex investigator agents to flag likely-flawed tasks, plus a separate human annotation campaign — reading task instructions, model attempts, metadata, tests, and failure traces, and labeling the flaws by type. It is the eval discipline of building a trusted set of known-good cases, applied in reverse: proving how many of the existing cases can't be trusted.

Here is why a third is such a heavy number. Hold the reported figures fixed: 731 public tasks, a frontier resolve rate of 80.3%, and a human-identified broken rate of 34.1% (OpenAI's audit). An 80.3% resolve rate on 731 tasks is about 587 counted as solved (arithmetic on the two source numbers). But that 80.3% is computed over the whole 731-task set, and 249 of those tasks (34.1%) can't be graded fairly — so the percentage blends real passes with wrong fixes a low-coverage test waved through and correct fixes an overly-strict test rejected. You cannot read 80.3% as 587 clean solves when a third of the set's grading is invalid. The number looks precise; the set it measures over is one-third rubble.

Broken-task type	What the grader does wrong	Which way the score lies
Overly-strict test	rejects a genuinely correct patch (demands one exact form)	false fail → understates the model
Low-coverage test	passes a wrong patch it never exercises	false pass → overstates the model
Underspecified prompt	hides a requirement the test silently enforces	unfair fail → punishes reading the prompt as written
Misleading prompt	points the agent toward the wrong solution	unfair fail → penalizes the intended behavior

The practical lesson is not "SWE-Bench Pro is bad." It is that you validate the tasks before you trust the score — the same reason you would never gate a production rollout on an eval you hadn't sampled and re-graded by hand. A benchmark that silently loses a third of its validity is worse than a smaller, clean one, because it hands you a precise-looking number you have no way to discount.

Goes deeper in: AI Agents → Evals & Diagnostics → The 4 Eval Failure Modes

Related explainers

This explainer stands alone from its news item (one concept), so the closest neighbors are other results on how a single evaluation number can quietly mislead:

Agent leaderboards mislead under distribution shift — Predictive validity — the sibling failure one level up: predictive validity asks whether a whole ranking survives a shift; task validity asks whether each individual task is even gradeable
WeaveBench — trajectory-aware vs outcome-only grading — how a single run's grade can be inflated when you only check the final answer, not the path
The Verification Horizon — co-evolving verifiers — why checking a coding agent is often harder than the coding itself, which is exactly why hidden tests go wrong

Continue in trackEvals & Diagnostics: the four ways an eval quietly breaks

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based