A 176-setting study ablates the coding-agent harness — Harness component ablation — What does it mean?
The news. On 17 September 2026, researchers published An Empirical Study of Harness Design for Coding Agents, which takes a coding-agent harness apart instead of benchmarking it whole. The execution loop stays fixed while three components move one at a time — planning, action space, and context management — across 176 matched settings: four models, two benchmarks, five context-management tiers, four context-window budgets, and targeted ablations of planning and the tool set. Read the paper →
Picture the desk with every channel up. The mix sounds good, and you have learned exactly one thing: the mix sounds good. You cannot tell whether the bass is carrying it or merely sitting there, because nothing ever played without the bass. That is whole-harness benchmarking — one number for the finished agent, and no way to attribute it to a part. So you mute one channel and replay the same take. Same take, same room, one fader moved, so whatever the meter does next belongs to that channel and nothing else.
Mute-and-replay is the entire method here. The study pins down the agent's execution loop — the think-act-observe cycle, the stuck detector that gives up after eight identical failing calls, the 300-step cap — and moves exactly three faders: whether the agent plans, what its action vocabulary is, and how it compresses history when the window fills. Each of the 22 settings is then replayed on all four models against both benchmarks — 176 runs over the same 500 SWE-Bench Verified issues and the same 89 Terminal-Bench 2.1 tasks — so inside a matched pair, a gap in success rate or in dollars per task has one candidate cause.
The first thing the ablation shows is that one fader only matters when the room is small. Context management is worth 35.7 success-rate points at a 32k window and 2.7 points at 128k — averaged across the four models on SWE-Bench Verified, comparing the managed tiers against no management at all. The reason is blunt rather than subtle: with nothing managing history, 78.7% of runs at 32k die of context overflow, against 8.7% at 128k, while every managed tier overflows on exactly zero tasks at every budget. Most of what context management buys is not a smarter agent. It is runs that do not die. That makes context as a scarce resource literally true: the tighter the window, the more the machinery earns.
The machinery itself is three composable mechanisms — cheap elision, reversible recall, and expensive summarization — switched on in five tiers.
| Tier | Mechanisms on | What happens to old history | Success rate — Nemotron-3 30B, SWE-Bench Verified, 32k window |
|---|---|---|---|
| T0 | none | It grows until the window overflows and the run dies | 9.4% (Table 3) |
| T1 | elision | Stale observation bodies become short stubs; the detail is gone | 20.6% (Table 3) |
| T2 | elision + recall | Same stubs, but the originals sit on disk behind a recall_event tool | 20.8% (Table 3) |
| T3 | summarization | The middle of the history folds into a running summary, one model call each time | 23.8% (Table 3) |
| T4 | all three | Elide at a soft threshold, summarize only if it is still over a hard one | 21.2% (Table 3) |
Read down that last column and the managed tiers look interchangeable — a few points apart, no clear winner. That is the finding, not a disappointment: once history is managed at all, which way you manage it barely moves accuracy, so the tiers separate on cost instead. Averaged across the eight model–benchmark pairs, T4 has the lowest mean cost per task at all four window budgets, and the reason is the ordering rather than the ingredients. Staging cheap elision ahead of expensive summarization is what makes the full tier the cost winner — early elision disposes of enough history that the model-priced summarization call fires less often than it does under summarization alone.
The reversible half of the design is the half that does not pay. Models almost never call recall_event to pull an elided observation back: 36 of the 64 context-sweep settings that offer it never call it once, the median invocation rate is zero, and the mean falls from 0.540 calls per task at 32k to 0.007 at 128k. Adding recall on top of elision moves accuracy by −0.36 percentage points as an equal-weight mean over 32 matched comparisons — machinery built, nothing bought. That is a general lesson for context engineering: an escape hatch the model has to choose to use is worth only what the model actually uses it for.
Here is where the ablation pays for itself. Hold everything fixed except the model — Tier 4 context management, a 128k window, the full tool set, SWE-Bench Verified — and flip planning on and off. Scale the study's per-task means to 1,000 tasks (the scaling to 1,000 is illustrative; the per-task figures are the study's).
On Nemotron-3 550B, planning on solves 65.8% at $2.33 a task: 658 tasks for ~$2,330. Planning off solves 67.8% at $3.31 a task: 678 tasks for ~$3,310. Switching planning off buys 20 more solved tasks for ~$980, or roughly ~$49 per extra solved task.
On Nemotron-3 30B, the same switch points the other way. Planning off solves 13.6% at $0.02 a task: 136 tasks for ~$20. Planning on solves 25.2% at $0.09 a task: 252 tasks for ~$90. Here planning buys 116 more solved tasks for ~$70, or roughly ~$0.60 per extra solved task.
Same fader, same benchmark, same everything else — and on these illustrative numbers the last solved task costs about 80× more on the big model than on the small one, with the switch pointing opposite ways. On the 550B, planning is not an accuracy scaffold at all; it is a ~30% cut to the cost profile of the agent, bought for two points of success rate. On the 30B it is the difference between an agent that works and one that mostly does not, and spending more tokens clearly earns its keep there.
The third fader, the action space, splits the same way and adds a failure mode worth knowing. Back at Tier 4 with a 128k window and planning on, giving the 550B a bare shell instead of the predefined tool set raises success by 3.6 points on SWE-Bench Verified while cutting cost 53% — bash-only trajectories there issue 32% fewer calls, consistent with the strong model composing denser shell commands, so the tool set had been charging it overhead rather than scaffolding it. Hand the same bare shell to the 30B and success on that benchmark drops 15.0 points. What goes wrong is clearest on the other benchmark: on Terminal-Bench 2.1, 66% of the 30B's bash-only runs end by emitting tool calls the bash registry cannot resolve, cutting the average run from 71 turns to 15. The model keeps reaching for an action vocabulary it learned in training, and the harness has no way to answer. That is a tool-design result rather than a model-capability one: the right interface is the one your model already speaks.
Put the three faders together and the real output is not a recommended harness. It is a warning about transfer. Planning and the action space each helped one model and taxed another, so a harness benchmark you read somewhere is evidence about their pairing, not yours. The portable part is the method — fix the loop, move one thing, replay the same tasks, read success rate and cost together — which is exactly what an A/B harness does in production, with your model and your budget supplying the only answers that apply to you.
Goes deeper in: Agent Engineering → Production Evals → A/B Harness
Related explainers
- E3 cuts coding-agent scope before expanding context — one concrete context policy, measured the way this study says to measure one.
- Harness Handbook localizes agent behavior before edits — once you know which component to change, this is how you find the code that implements it.
- HarnessBridge — learned agent harness vs hand-engineered — the opposite bet: let the controller learn the components instead of ablating them by hand.