Cache Engineering for Agents — Drop and reposition vs re-prefill — What does it mean?

The news. On October 20, 2026, a Tsinghua University team posted the paper Cache Engineering for Agents to arXiv. It adds two agent-callable operations on the live KV cache, drop and repos, and ships an open-source serving engine called AKV that makes them cheap. Read the paper →

Picture an agent's whole trajectory as a binder of numbered index cards. Each card was written while reading the cards before it, so its wording already reflects them. That is exactly what a KV entry is: the key and value vectors one token produced while attending to everything earlier. After a few hundred tool calls the binder is enormous, and the paper measured that reading the cache back made up 82.4% of the total API cost in its cost breakdown. So the agent needs to shrink the pile it reads from.

The usual way is to edit the text: trim old tool outputs or summarize them. In binder terms, you pull card 3 and every later card is now wrong, because each was written with card 3 in view. The server must rewrite every card after the edit, which is a re-prefill, and the earlier the edit, the more it rewrites. This is the same rule that makes prefix caching fail when one early token changes. Harnesses therefore wait for a size threshold and then cut most of the history at once. The authors tested this on 100 BrowseComp tasks with GPT-OSS-120B: keeping only the latest 12 tool rounds cost 1.85× the full-context baseline, and summarizing at a 48k-token threshold saved only about 7% while losing 5 accuracy points.

Cache engineering keeps the cards and changes which ones are on the reading pile. drop removes chosen KV entries from all future attention, and leaves the text, the other entries and their position numbers untouched. Nothing is recomputed. The retained cards were written while the dropped one was in view, so they can still carry some of what it said. drop alone has a catch: position indices keep growing with the full trajectory, so a small cache can still hit the model's position limit. The second operation, repos, renumbers the survivors from 0 and resets the next position. With RoPE, moving a key from position p to q is one rotation, K ← R(q) R(p)⁻¹ K, and the values are left as they are, so no hidden state is recomputed. In the authors' ablation on BrowseComp, adding repos to tool-response dropping raised the pass rate from 55% to 62%.

Their main policy is a tool-response rolling drop: keep every reasoning step and the most recent tool outputs, and drop older tool outputs as the run goes on. Attention measurements in the paper showed older tokens and tool outputs get less attention than recent tokens and assistant messages, which is why tool outputs are the first thing to drop. What you drop matters: with repos on, also dropping the model's own reasoning gave lower pass rates than dropping tool outputs alone.

There is a serving problem, though. In normal serving, the token prefix fully determines the cache, so a radix tree can match requests on tokens alone. Here it does not. The paper's example has three trajectories that all end up holding B and C: one appends C and then drops A, one drops A before appending C, and one starts from B alone. C's KV entry differs in all three, because C either saw A or did not. Identical text can sit on different caches depending on the order of append, drop and repos, so reusing a cache on token match alone would be silently wrong.

The AKV engine handles this with a stateless interface and stateful internals. Each request carries the full messages plus a cache field that lists the drops and repositions, so the server can rebuild any state it already evicted. Inside, AKV extends the radix tree with drop nodes so that matching checks token and operation history together. Its eviction frees a dropped entry once every request using it has dropped it, and it prefers evicting dropped cache over cache from finished requests, which is likelier to be reused next turn. To rebuild missing state in one pass, segment-batch attention splits the queries at the drop points so each segment sees only its own active cache, while the feed-forward layers still run over the whole sequence as one batch, which avoids re-reading the FFN weights per segment.

Here is the arithmetic in one concrete case (illustrative numbers, not from the paper). An agent is 100k tokens in, and the oldest 20k are stale tool output. Its next turn adds 2k new tokens. With a text edit, the server must re-prefill the 80k retained tokens plus the 2k new ones, 82k tokens prefilled. With drop, it prefills only the 2k new tokens: 2k tokens prefilled, about 41× less prefill work for that turn. Each decode step also attends to 80k entries instead of 100k, so it reads 20% fewer KV entries. The paper attributes its decode speedup mainly to this shorter attention, and reports that the gap widens at high concurrency, where the baseline's re-prefill crowds out decoding.

On the measured results: across three LLMs and four agent benchmarks, AKV's average score was 1.3 to 2.1 points above the full-context baseline for each model (1.7 on average), and average cost fell 27.7% against that baseline and 21.9% against text-level engineering. For serving, on a SWE-bench Verified subset with GPT-OSS-120B, each method at its own peak-throughput concurrency (baseline at 12, AKV at 14), AKV reached 1.51× lower P95 latency, 1.53× the input throughput and 1.18× the output throughput. The score and cost figures come from three LLMs and four benchmarks, while the latency and throughput figures come from a single GPT-OSS-120B serving setup, so none of them should be read as general. The paper also reports that the rolling-drop policy was hand-designed, and a policy found by agent self-improvement cut a further 7.1% of cost on held-out tasks, so the best policy is likely task-dependent.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → Why Full-Prompt Hashing Fails

Related explainers

• KV-Repurpose — Position-adjusted segment KV reuse — another way to move cached keys to new positions instead of recomputing them
• Tangram — Per-head KV cache budgets — a different answer to which KV entries deserve to stay
• HEAR — Workflow-aware KV caching — how an agent harness and a serving engine can coordinate on the cache