The Resolution Hypothesis paper — Evaluation noise vs true improvement — What does it mean?
The news. On November 17, 2026, Yuanhao Zeng, Kan Ren and nine co-authors posted "The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements" (arXiv 2611.23068). They ran seven evolution methods — GEPA, SkillOpt-Lite, Trace2Skill, ReasoningBank, HarnessLens, Meta-Harness and AHE — on six models and four agent benchmarks, repeating each run up to 16 times and re-scoring each result up to 16 times: 1,030 evolution runs, over 2.1 million test trajectories and more than 10 million LLM calls. The authors say the reproducible materials and trajectory data will be released when the paper is accepted. Read the paper →
Evaluation noise is the random wobble in an agent's score from one run to the next; a true improvement is the part of a score change that survives a rerun. The paper's claim is that self-improving agents stall when the true improvement on offer is smaller than the noise — the selector then keeps lucky readings, not better artifacts.
Picture a hiker climbing in fog. Each round they try two steps and keep whichever one the altimeter says is higher. On a steep hill this works: the real rise beats the needle's jitter. On a gentle slope the jitter wins. The altimeter reads +5 for a step that actually goes down and −1 for the one that goes up, and the hiker walks downhill with full confidence.
The paper splits the wobble into two parts. Evaluation noise (σ) is how far one artifact's score moves between reruns: 4.5 percentage points (pp) on Terminal-Bench, and 1.0 to 3.7 pp on SpreadsheetBench, OfficeQA and ALFWorld. Run-to-run variation (τ) is the standard deviation of true quality between the final artifacts of separate runs of one method — from 0 up to 12.3 pp. The two limit different questions. Judging one candidate against the current artifact is limited by σ alone; judging one method against another is limited by σ and τ together, because every run produces a different artifact. In hiker terms, σ is the altimeter's jitter, and τ is how far apart two hikers who started at the same spot end up after a day of jittery choices. This is the noise floor the eval failure modes step warns about for small samples — except an evolution loop hits it on every round.
Is the problem that the proposer cannot find a better path? The paper says no. In a Proposal Pool of 100 candidate skills, written from 10 Terminal-Bench training tasks and scored with 16 rollouts on each of 10 different test tasks under a 2-hour budget, the best shared skill solved 57.5% of attempts, against a pool mean of 49.2% and a weakest skill at 41.9%. Better steps exist. The failure is in choosing them: in a second pool of 100 skills across 11 tasks, selection by execution score, seeing one rollout per candidate, picked the better of two skills 49.7% of the time — a coin flip. Regressions were 30.5% of that pool but 62.1% of the skills it accepted (a wide interval: the paper cannot say this is worse than random acceptance), and the rank correlation between one score and true quality was 0.054. LLM judges that read the whole trajectory did no better; the authors suggest a judge's own reading errors add to the evaluation noise instead of cancelling it.
| Selector (Selection Pool: 100 skills, 11 Terminal-Bench tasks) | Rollouts seen per skill | Picks the better of two | Accepted skills that are actually worse | Source |
|---|---|---|---|---|
| Random choice | — | 50.0% | 30.5% | Table 1 |
| Execution score | 1 | 49.7% | 62.1% (±42.9, wide interval) | Table 1 |
| SkillOpt-Lite native selection | 2 | 48.6% | 59.4% (±46.2, wide interval) | Table 1 |
| GEPA native selection | 2 | 48.6% | 31.6% | Table 1 |
| LLM judge, pointwise | 1 | 45.4% | 39.9% | Table 1 |
| LLM judge, pairwise | 1 | 49.0% | 57.6% | Table 1 |
| LLM judge (ACE) | 1 | 46.9% | 39.0% | Table 1 |
Now repeat that coin flip every round. Because the accept rule compares a candidate's noisy score with the current one, it keeps draws from the lucky upper tail of the noise, round after round — so the logbook climbs while the hiker stands still. The authors re-scored every intermediate artifact from eight SkillOpt-Lite runs and eight GEPA runs on Terminal-Bench, with eight rollouts per task where the loop had used one. In-loop gains had grown to 6.9 to 9.0 pp; re-measured, the same artifacts sat between +0.4 and −3.2 pp. The random choices also push runs apart: every run started from the same artifact, yet by the last round the noise-corrected run-to-run variance reached 4.2 to 24.5 squared pp. It is the evaluator-optimizer loop with an evaluator whose verdict is mostly noise, and the optimizer obeys it faithfully.
The same arithmetic breaks the rankings between methods. Across 171 results in nine agent-evolution papers, the new method led its strongest rival by a median of 3.4 pp; none reported a variance estimate, and only 6 averaged more than one evolution run. For over 60% of method pairs on the same model and benchmark, the gap between their long-run averages was smaller than one standard deviation of a single-run comparison. On Terminal-Bench, a results table with one run per method named the truly best of four text-artifact methods only 33% of the time, against 25% for a random guess. The sample-size rule from the A/B harness step applies unchanged: to detect a 3.4 pp margin at 80% power, the paper needs about 5 runs per method on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld and 23 on Terminal-Bench.
Here is why one reading cannot see a small step. Hold three things fixed: Terminal-Bench's measured noise of σ = 4.5 pp, a candidate that is truly 2 pp better (illustrative), and one evaluation per candidate. The difference between two noisy scores has a standard deviation of √2 × 4.5 ≈ 6.4 pp — more than three times the real 2 pp gap. A 2 pp lead is only 0.31 of that standard deviation, so, treating the noise as normal and independent, the better candidate scores higher in about 62% of comparisons. For that 2 pp step to win 95% of comparisons, you need 2 ÷ (6.4 ÷ √n) ≥ 1.645, which works out to about 28 independent evaluations of each candidate, every round, under the same approximation. The loops in the study scored each candidate once or twice, and the selectors it measured did worse than this 62%, consistent with true gaps smaller than 2 pp.
So how do some papers report gains of 10 to 40 pp? The largest published gains came from evaluation shortcuts — one behaviour the grader rewards on almost every task, big enough to rise above any noise. Auditing evolved artifacts instruction by instruction, the authors added only the single most responsible rule to the unevolved baseline; across the 27 model–benchmark pairs with a reported gain above 3 pp, that one sentence recovered a median of 90% of the gain. Such rules typically matched a grader that credits only one answer format, or a benchmark version that departs from the official evaluation. After the graders were repaired, the median gain fell from 17.6 to 0.8 pp.
The fixes the paper proposes raise resolution, not proposal quality: checks of intermediate behaviour — such as whether verification commands show unmasked failure diagnostics — had 13× to 57× lower noise than pass/fail verdicts in a preliminary analysis, though triggering more checks does not by itself mean solving more tasks; the proposer and the selector are evaluated separately, with Quality@K curves and selector curves on a fixed pool; and every result reports σ and τ next to its score. The study covers only loops that keep model weights frozen.
Goes deeper in: AI Agents → Evals & Diagnostics → The 4 Eval Failure Modes
Related explainers
- LangChain's Jev — evaluator variance vs the generative LLM judge — noise one level down: the same frozen input graded differently on a rerun
- The Verification Horizon — co-evolving verifiers — when the verifier is not noisy but too weak to tell a good solution from a gamed one
- TestGRAD — differential test loss for patch selection — a selector that only counts tests that split the candidates
- Scaling Discovery Requires Forgetting — another search loop that stalls as it accumulates history