EarlyEval halts agent eval runs mid-trajectory — Calibrated early stopping — What does it mean?

The news. On September 2, 2026, a paper introduced EarlyEval, a framework that predicts an agent evaluation's outcome from a partial trace and stops the run as soon as the prediction is confident enough. Tested on SWE-bench Verified, TerminalBench and Toolathlon, it removed 13–26% of agent steps, up to 44.1% of input tokens and 29.4% of output tokens, while moving reported resolve rates by only one to two percentage points on average. The code is open. Read the paper →

Picture an arbiter standing over a chess game with an evaluation bar beside the board. Thirty moves in, the bar has pinned hard to one side and stayed there. The arbiter does not need to watch mate get delivered; the result is already established, and the remaining forty moves would only confirm it. So the game is called, and those moves are never played.

That is the move EarlyEval makes, and the reason it is allowed to is easy to miss. When you run an agent for a user, you run it for the work it produces, so you have to let it finish. When you run an agent inside an eval suite, you are not running it for the work — you are running it for the score. Once the score is known, every further step is a bill with nothing on the other side of it. Picture a 60-step run that has been thrashing on the same import error since step 12: it is not going to resolve the task, and a reader of the trace could tell by step 20.

The bar in the picture has one detail that matters more than it looks. It has two marks on it, not one — a line high up for this run will pass and a separate line low down for this run will fail — and the game ends when the bar crosses either of them. That is not decoration; it is the mechanism, and it is where the design gets interesting.

Underneath the metaphor the machinery is deliberately plain. After each agent step, EarlyEval extracts three families of features from the trace so far: behavioural ones (how many steps have run, whether the agent is repeating itself, how tool calls are landing), textual ones (what the agent is actually saying and emitting), and reference-solution ones — signals from comparing the partial run against the known-good answer, which an offline suite has on file by construction and a live deployment does not.

Those features feed two separate LightGBM classifiers — one trained to recognise runs that will succeed, one trained to recognise runs that will fail — and each carries its own calibrated confidence threshold. Execution stops the moment either one crosses. Training two models instead of one is the load-bearing choice: success and failure plausibly do not look alike in a trace — a run heading for a pass might show narrowing, converging behaviour, while a run heading for a failure might show a loop, a stall, or an error the agent keeps re-encountering. Two specialists reading two different shapes is a better bet than one model asked to span both, though the paper reports the design rather than an ablation proving it. Splitting them also lets you set the two bars independently, which matters when a wrong fail call and a wrong pass call cost you different things.

The classifiers are gradient-boosted trees, not another language model, so the paper reports the per-step overhead as negligible — the predictor has to be orders of magnitude cheaper than the step it might cancel, or the accounting collapses. That constraint is also the argument against the obvious alternative — asking a model to judge the trace — although the paper does not report trying it.

Put a bill on it. Take an offline suite of 500 tasks where a run averages 120,000 input tokens across its steps, billed at $2.50 per million — all three numbers illustrative, chosen to hold the arithmetic still while only the stopping rule changes. One full pass over that suite reads 500 × 120,000 = 60 million input tokens, or $150. Hold the suite and the token profile fixed and switch on early stopping at the paper's best reported saving of 44.1%, and the same pass reads 33.5 million tokens: $150 → $84 per pass.

The saving is not really the $66. It is what $66 per pass does to the cadence: a suite you could afford nightly, you can now afford on every merge. That is the same argument the cost profile of an agent step makes about production traffic, applied to the eval pipeline instead — and eval spend is the half of the bill that tends to go unmeasured, because nobody is paying for it per user.

Now the part that is easy to skip past. Early stopping does not make the eval cheaper for free — it trades a small amount of faithfulness for the saving, and the paper is direct about it: reported resolve rates moved by one to two percentage points on average. At 89–97% prediction accuracy, three to eleven calls in a hundred are wrong, and a wrong call is a task scored on a guess rather than a result.

One to two points is nothing when you are asking is this build broadly worse than the last one and everything when you are asking did this change move us from 41.2% to 42.0%. So the rule that falls out is a scoping rule, not a switch: run the fast, early-stopped suite on every merge, and keep a full un-truncated pass for the numbers you publish or gate a release on. That is the same two-tier shape the online vs offline evals step draws — a cheap signal you can run constantly, and an expensive one you trust — and it is worth noticing that the choice only exists because the eval reports a rate rather than a verdict. Where the harness collapses a run to pass or fail, a confident early call and a completed run produce the identical bit.

Goes deeper in: Agent Engineering → Production Evals and Shadow Mode → Online vs Offline Evals

Related explainers

• Agentic abstention — when to stop — the mirror image: the agent deciding to stop itself, rather than the harness deciding for it.
• Long-Horizon-Terminal-Bench's partial-reward threshold — what a run's score even is once you stop treating it as one bit.
• AutoLab's iterative experiment-loop evaluation — why long-horizon agent evals got expensive enough to be worth truncating.