Bursts That Outlast Themselves — Prefix-Cache Metastability — What does it mean?

The news. On November 10, 2026, the paper Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving (arXiv 2611.13412, Microsoft and an independent co-author) ran paired vLLM experiments in which two servers saw identical sessions, except that one received 5 minutes of extra arrivals. In 9 of 12 pairs only the burst run fell into a state with a hit ratio near 0.1, under half the throughput, and a backlog that kept growing for the rest of the 90-minute run; no pair went the other way. The authors do not claim the feedback loop as new (systems such as CONCUR and KAIROS observed it); their contribution is that the loop persists and that leaving it needs a different control than avoiding it. Read the paper →

Picture the reading room on a normal afternoon. A handful of readers each have a desk, and each leaves a stack of books on the desk while they step out to check something. When they come back, the stack is still there, so they open it and add one page. That is a cache hit: an agent turn whose previous context is still resident only has to prefill the roughly thousand new tokens from the last tool call. On a cold miss, it prefills the whole 10–40K-token context again. In the paper's measurement, a mid-session turn costs 0.15 s of prefill on a hit and 2 s on a miss.

Now a bus unloads a crowd for five minutes. Desks run out, so the librarian reshelves the stacks touched longest ago. Each returning reader whose stack is gone has to fetch every book again, which takes longer, so readers stay in the room longer. (The paper infers this eviction step from its measurements; vLLM does not export per-block eviction events.) This is the step that closes the loop: agents are closed-loop clients, so by Little's law slower turns mean more sessions in the system at once. More sessions need more desks, more stacks get reshelved, and turns get slower again. The bus left long ago, but the room never empties.

The paper names three properties that close this loop. Backlog is working set: every session in the system needs its prefix resident to hit. The miss penalty grows with session age, because a miss recomputes a context that grows every turn. And running requests and cached prefixes share one pool, so the busier the engine, the less room is left for idle sessions' stacks. In the degraded runs, 99.6–100% of completed turns had less than half their prompt cached, and turn throughput matched the same engine with prefix caching switched off.

Here is the arithmetic with the paper's main setup held fixed: Qwen3-8B on one AMD MI300X, a 400K-token KV pool, and a mid-session turn of 21.6K tokens. Dividing gives room for at most about 18 such prefixes (400K ÷ 21.6K ≈ 18.5, our arithmetic; the pool also holds running requests, so the real number is lower). A healthy run kept 10 or fewer sessions in the system, so returning turns usually found their stack and paid 0.15 s. After the burst, the same server carried roughly 110 to 160 sessions, six to nine times what the pool can hold, so nearly every turn missed and paid 2 s, about 13× more prefill per turn. Throughput fell from up to 1.27 turns/s to 0.35–0.46 turns/s. The paper's workload-balance condition puts the break-even arrival rate near 0.010–0.014 sessions/s, while the collapse happened at 0.035: by our arithmetic, once collapsed, arrivals would have to fall to roughly a third of the original load for the backlog to drain by itself.

The uncomfortable part for capacity planning is where this happened. The usual rule is to run at 70–80% of saturation to leave headroom for spikes. This server ran at about 70% of its cache-warm capacity and still collapsed, because the capacity of a prefix-cached engine is not one number: it is high while the cache is warm and close to the no-cache figure once it is cold. A burst is not the only trigger, either. A cache flush with preemption, which is what a replica restart or weight update does, left 1 of 5 runs persistently degraded with no extra load at all.

Getting out is where the obvious fix fails. Closing the door to new readers does nothing for the crowd already inside: each of them keeps cycling through desks that cannot hold them, and must finish all remaining turns, all misses, before the room empties. What works is asking some readers who are already inside to wait in the lobby. At its next turn boundary, a parked session sends no turns and its prefix is allowed to be evicted, so the remaining sessions start hitting again, throughput returns, and the queue drains. A re-admitted session usually pays one miss, since its stack was likely reshelved while it waited.

A fixed session cap works on this homogeneous workload, but only because someone measured the per-session demand first. The budgeter counts tokens instead of sessions: it adds up each admitted session's current context plus its output budget, admits sessions while that total is below 75% of the pool, and parks a session at its next turn boundary when the total passes 95%. The pool size is reported by the engine at startup, so nothing needs per-workload tuning, and moving the band to 65/85% or 85/98% changed neither the outcome nor the timing. That matters when sessions differ: on Qwen3-32B replaying SWE-bench agent trace shapes, a session cap set from a lighter load recovered 0 of 3 collapsed runs and the budgeter recovered 3 of 3. Across all its recovery experiments the budgeter recovered 17 of 17 collapsed runs, and with it on from the start, no run collapsed under any trigger tried.

Two limits keep this honest. A CPU offload tier (128 GiB) only delayed the collapse, from minute 20–25 to minute 40–60, because the growing crowd eventually outgrows the host tier too. And the budgeter restores the engine's efficiency but cannot create capacity: on one 32B trace mix, arrivals were slightly faster than even the cache-warm engine could serve, so the hit ratio recovered while the queue still grew slowly. Parking also has a cost users can see: sessions arriving after control began took a median of 11.6 minutes to complete, against 3.8 minutes on a server that never collapsed.

Goes deeper in: LLM Serving → Prefix Caching → When It Helps, When It Fails

Related explainers

• Prefix-cache nondeterminism in agent runs — the same cache, judged on whether your measurements repeat instead of on throughput
• TAO: tool-aware joint session scheduling — deciding which waiting agent sessions get a GPU node while their tools run
• CacheReforge: stale KV after a LoRA update — another way a prefix cache stops paying off