What is slice evaluation?

It is a cheaper evaluation setting that exercises repeated control over a selected portion of a task instead of driving the whole task from its original state. The worker still executes, so decisions are still validated by real runs — there is just less execution behind each one.

Does a cheaper evaluation give the same answer?

It gives a very similar ordering, not the same numbers. LoopArena reports a Spearman rank correlation of 0.9747 between the slice ranking and the full-task ranking under its Core criterion, with estimated inference cost down an average of 64.4%. That can support model selection, but it neither guarantees the same top model nor reports absolute quality.

When should you not use a slice evaluation?

When you need the level rather than the ranking — a headline pass rate, a number you will hold a team to, or anything a reader will quote. It also stops being a valid proxy once nobody re-checks the rank correlation against full runs as your model set drifts.

How is rank correlation different from accuracy?

Accuracy asks whether the cheap measurement reproduces the expensive measurement's value; rank correlation asks only whether it puts the same items in the same order. Most model-selection decisions need the weaker claim, which is why a proxy eval can be far cheaper and still be sound.

Agent· 2026-09-07

LoopArena benchmarks the model that supervises a coding agent — Slice evaluation as a rank-preserving proxy — What does it mean?

Jargon

Task slice: A selected portion of a full task. Slice evaluation runs repeated control over that portion rather than over the whole task, so less execution sits behind each scored decision.
Type I / II / III: LoopArena's three evaluation scopes in increasing cost: Type I runs no worker at all, Type II runs a slice, Type III runs the complete paired task.
Spearman's ρ: A rank-correlation score from -1 to 1. It asks only whether two methods put the same items in the same order, and ignores how far apart their scores are. LoopArena reports ρ = 0.9747 between the slice ranking and the full-task ranking, which means very similar orderings, not identical ones.
Core criterion: The main scoring rule the settings are compared under. The rank agreement between the slice setting and full tasks is reported under this criterion specifically, not as a claim about every criterion.
Paired reduction: The paper reports the saving as a paired reduction — measured against the matched full task rather than as a difference of two averages. Across controllers it averages 64.4%.
Strict Success Rate: The pass metric on full paired tasks; the best controller reached 24.69%. It comes out of the full setting, which is why the slice cannot report it. See controller-worker separation.

The news. On August 28, 2026, the LoopArena team published a benchmark for models that supervise a coding agent, and measured what a cheaper version of that benchmark costs you. Running control over a task slice rather than the whole task cut estimated inference cost by an average of 64.4% across controllers, and produced a similar ordering of those controllers under the benchmark's main criterion, at a rank correlation of 0.9747 — similar, not identical. Read the paper →

You have three pots of soup and one question: which is best? Eating all three to the bottom answers it and costs you the afternoon. A spoonful from each answers it too, as long as you stir first and take every spoon from the same depth. The spoonful will not tell you how good the winning soup is on any absolute scale, and a very close ranking is still not a promise about the top spot — but it narrows the field cheaply, which is usually the decision you actually have.

A task slice is that spoonful. Rather than driving a controller through a whole task from its original state, the benchmark exercises repeated control over a selected portion of one. The worker still runs, so every decision is still validated by real execution; there is simply less execution sitting behind each scored decision. That is where the saving comes from, and the paper reports it as a paired reduction — measured against the matched full task rather than as a difference of two averages — averaging 64.4% across controllers.

The catch is the one the metaphor already warns you about: a spoonful is only evidence while somebody still occasionally eats a whole bowl. A cheap proxy stays trustworthy exactly as long as someone keeps checking it against the expensive measurement it stands in for — which is what the rank correlation is, and why the paper reports it under one named criterion rather than as a blanket claim. This is the same tension as online versus offline evals: the cheaper, more controlled measurement is only worth having while you can still show it tracks the expensive one.

And the way you show it is a rank correlation, not an accuracy check. A proxy eval does not have to reproduce the full run's scores; it only has to order the candidates closely enough to support the choice you are making, which is a strictly weaker and much more achievable claim. It is the same distinction as scoring a run pass/fail versus grading it: choose the weakest output your decision actually needs, and the measurement gets cheaper for free.

Setting	Does the worker run?	Cost	What it tells you
Next-step choice (Type I)	no	lowest	Whether the controller picks the right Loop Contract, scored with execution-validated questions.
Task slice (Type II)	yes, over a slice	~64.4% cheaper on average than the matched full task (source)	A ranking of controllers very close to the full-task ordering — ρ = 0.9747 under the Core criterion.
Full paired task (Type III)	yes, end to end	highest — this is the baseline the other two are priced against	The headline number. Best observed Strict Success Rate: 24.69% (source).

Put a price on it. Say a full paired run of one task costs $1.00 of inference across the controller and the worker, and you want to rank 8 candidate controllers over 50 tasks. Both of those numbers are illustrative; the ratio the paper measured is not.

The full sweep is 8 × 50 = 400 paired runs, so $400. Apply the paper's 64.4% average paired reduction as a flat discount — illustrative, since it is an average across controllers and not a per-run guarantee — and a slice run costs $0.356, so the same 400 cells come to 400 × $0.356 = $142.40. You spend $257.60 less and, at a rank correlation of 0.9747, you get back a very close — not identical — ordering of the eight models.

What you do not get back is the level. The slice will never tell you that your winner clears only 24.69% of full tasks strictly, because it never runs a full task. That is the whole trade: slice evaluation is for narrowing the field, not for reporting how good the survivor is — so you still pay for the full run, once, on the model you picked.

Goes deeper in: Agent Engineering → Production Evals & Shadow Mode → Online vs Offline Evals

Related explainers

Controller-worker separation — why the worker is held fixed in the first place, and what that buys you when a run fails

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based