Audit RL verifiers and trace 93% of failures to punctuation — Metamorphic verifier testing — What does it mean?

The news. On September 1, 2026, an arXiv paper called Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR published the first per-category breakdown of where automatic graders go wrong. The authors generated meaning-preserving rewrites of correct answers and ran 307,420 verdicts across four widely used verifiers. Self-validation ranged from 53.8% to 95.2% on identical inputs, two configurations of the same library disagreed on half the pairs, and whitespace and punctuation — not the LaTeX parsing usually blamed — accounted for almost all of the remaining error. Read the paper →

Put a bag of flour on a kitchen scale and it reads 500 g. Turn the bag on its side and it reads 500 g again — obviously, because turning a bag does not change how much flour is in it. Now imagine it reads 500 g upright and nothing on its side. You have learned something without ever knowing the true weight: you do not need ground truth to prove an instrument is broken, you only need a change that should not have mattered. That is the whole idea behind metamorphic testing, and it is what this paper points at the verifier instead of the model.

The bag of flour here is a correct answer, and turning it sideways is rewriting 3.5 as 3.50, or \frac{1}{2} as 0.5, or adding the trailing newline a chat model naturally emits. These are certified equivalent variants: the rewrite preserves the mathematical value by construction, so the verifier has no legitimate reason to change its verdict. When it does, there is nothing to argue about and no human adjudication needed — the rejection is a provable false negative, and the fault is in the scale.

This matters more in training than in benchmarking. A benchmark with a leaky grader reports a slightly wrong number once. In RLVR the verifier's verdict is the reward, so a false negative does not merely mis-score the run — it is fed back as a gradient. The model produced a correct answer, got a 0, and learns to avoid whatever it just did. Because the rejected forms are systematic rather than random, the model is being steered away from an entire formatting style, run after run.

The audit's contribution is the decomposition. Prior work had already reported that one harness accepts only about 94% of its own ground-truth answers and pinned it on LaTeX parsing, but an aggregate cannot tell you which answer forms consume the error budget — and it turns out the blame was misplaced. The failures are not spread thinly across parsing categories; they are concentrated almost entirely in whitespace and punctuation. A trailing period or a newline, the paper reports, dominates the budget. The three findings below are what the normalization step is actually doing when nobody is auditing it.

The fourth row is the one worth walking, because it fails in the opposite direction — not rejecting right answers but accepting wrong ones. Hold the error fixed at exactly one: the model is off by 1, every time. Vary only the size of the number. A reference answer of 3,000 against a reply of 3,001 is a relative error of 1/3,000 ≈ 3.3 × 10⁻⁴. The same off-by-one against a reference of 300,000 is 1/300,000 ≈ 3.3 × 10⁻⁶ — a hundred times smaller, from an identical mistake. Now put a fixed relative threshold of 10⁻⁵ (illustrative — the paper reports the resulting step, not the constant) in front of that: the first is rejected, the second sails through. The paper measures exactly this step in a reference numeric cascade: off-by-one answers are accepted 0% of the time below 10⁴ and 100% of the time at or above it. Same wrong answer, opposite verdict, and the only thing that changed was the magnitude — the scale's tolerance was a percentage, so it grew with the number it was checking.

The practical move is cheap and the paper hands it to you: before trusting a verifier, feed it its own ground-truth answers and count how many it accepts. Anything under 100% is the grader failing a test it wrote itself, and the gap is your floor on false negatives. From there, metamorphic variants tell you which categories to fix, in priority order — and on this evidence, stripping trailing whitespace and punctuation before the comparison buys back most of the budget for a few lines of code.

Goes deeper in: AI Agents → Evals & Diagnostics → The 4 Eval Failure Modes — where the same problem is named validator validation failure, here measured on deterministic graders rather than LLM judges.