Compact-Bench — measuring compaction damage with paired forks — What does it mean?

The news. On October 22, 2026, researchers at the University of Science and Technology of China released Compact-Bench. It covers 35 SWE-bench repairs, 146 agentic sessions and 106 production compaction chains, and it compares 14 summarizer backbones and 5 production compaction prompts from coding-agent harnesses. Read the paper →

Picture a builder who must leave in the middle of a shift. Two things can go to the replacement: the whole diary of the shift, or a one-page note. Whether the note was good is only visible if both replacements start from the same half-built wall and the same inspector judges the result. That is the design of Compact-Bench. At the compaction boundary it freezes the agent's history H and a snapshot D of the repository. The PRE copy continues from H. The POST copy continues from the summary S = C(H). Both restore D, use the same tools and the same eight-step budget, and are graded by the repository's FAIL_TO_PASS tests.

Each repair then lands in one of four cells: preserved (both pass), damaged (PRE passes, POST fails), rescued (PRE fails, POST passes) or unsolved (both fail). The damage rate is the share of PRE-pass repairs that fail after compaction. Keeping damage and rescue apart stops gains on some repairs from cancelling losses on others, which a single success rate would hide. The Context Engineering module lists how context goes wrong; this design measures one such loss directly.

The spread between summarizers is large even when everything else is held fixed. Take the 22 repairs that the agent solved with full history (the paper evaluated 24 and dropped two where PRE failed on re-run). With the Claude Code prompt and a Kimi-K3 continuation agent, Claude Opus 4.8 as summarizer damaged 1 of the 22, which is 1 ÷ 22 ≈ 4.5%, reported as 5%. DeepSeek-Flash damaged 14 of the 22, which is 14 ÷ 22 ≈ 63.6%, reported as 64%. Same history, same repository, same agent, and 13 more repairs lost: a 59-point gap that comes from the summarizer alone. Repeated runs confirmed 120 of the 126 first-round damaged pairs, so the gap is not sampling noise.

The prompt matters too. With Kimi-K3 fixed as summarizer, the Kimi Code prompt passed 19 of 22 (86%) and the Claude Code prompt 18 of 22 (82%). Yet on the separate 146-session retention set, the same two prompts retained 8.2% and 15.0% of recurring details. The ranking flips: the prompt that keeps more strings is not the one that finishes more repairs. The paper notes these are configuration-level comparisons on different datasets, and that the Claude Code prompt writes summaries of about 10,000 characters against 1,600–5,200 for the others, so length and structure differ along with the instructions.

Keeping the right strings does not keep the progress. The paper audits four stably damaged cases. In every one the summary still names the target file. On pytest-7571 all 5 audited gold-patch tokens survive in the summary, and the continuation still fails. On sphinx-9230 the summary keeps strings from the fix but drops the reasoning that split(None, 1) must become rsplit. In that trace the full-history copy edits the call in 8 tool actions, while the summary-only copy reads tests and reproduction code through 13 actions inside the same eight-step budget and never makes the edit. The builder has the right bricks and has lost the plan for the next row.

On the retention side the paper reports that the best LLM summary keeps at most 16.0% of recurring details, while the mechanical anchor_index baseline keeps 79.1%, a 63.1-point gap. Read this with its scope: the reference set is, by construction, strings that appear in both the history and the next 3 to 8 request payloads, matching is exact substring, and the baseline reads at most 400,000 characters and keeps at most 40 values per type. It shows how lossy generated summaries are for literal values; it does not show that copying strings completes tasks. Across production chains the retention was 15.7% at the first boundary (106 sessions) and 19.9% at the second (27 sessions); later boundaries have only 6 and 1 sessions, so those numbers are too thin to read. The authors also list the limits themselves: results depend on the tested repairs, continuation agent and budget.

Goes deeper in: AI Agents → Context Engineering → The 4 Fixes

Related explainers

• CliffCompaction — a compaction rule that avoids summary-of-summary drift
• CarryOn — carrying task state in revisable records instead of summaries