What is the prefill reserve in an LLM serving engine?

It is GPU memory an engine holds back so that the largest permitted prefill chunk always has room to run, no matter what else is in flight. Because it is sized for a worst case that most scheduling steps never reach, it sits idle much of the time — and every byte of it is a byte unavailable to the KV cache, which is what limits how many requests a GPU can serve at once. The paper measures that reserve at 16% of KV capacity at TP1, falling to 2.7% at TP4.

Why did chunked prefill beat elastic KV-cache reclamation?

Elastic reclamation lends the reserve to decode and takes it back before the next prefill, so the memory is only available during decode windows and the scheduler must anticipate recommit, which the paper reports takes tens of milliseconds against a decommit of a few. Lowering the chunked-prefill token limit instead shrinks the reserve you must hold at all, returning that memory on every step with no timing to get right. The paper's live-load tests found the smaller chunk limit recovers more KV capacity, at about 1% difference in median TTFT between 8,192- and 32,768-token chunks.

How does this relate to PagedAttention and CUDA graphs?

The allocator borrows PagedAttention's central idea — keep the virtual address a kernel uses fixed and change the physical memory behind it — but applies it to the prefill reserve rather than to the KV cache. That indirection is what makes the scheme compatible with CUDA graphs, since a captured graph replays the exact addresses it was recorded with and would break if the memory moved. The paper reports both CUDA graphs and prefix caching surviving the mechanism, which is why its defeat is a result about value rather than about feasibility.

Show chunked prefill beats elastic KV-cache reclamation — Reclaiming the prefill reserve

TL;DR

What is it: The news anchor is a paper that builds a working elastic KV-cache allocator for LLM serving; the concept it makes concrete is reclaiming the prefill reserve — the GPU memory an engine holds back so the next prefill chunk always has room, and which therefore cannot be used as KV cache.
Why it’s needed: KV cache capacity is what caps how many requests a GPU can serve at once, so memory that is reserved and left idle is not an accounting detail — it is concurrency you paid for and did not get.
vs previous: Instead of a static reserve that is set aside at startup and never touched, the allocator remaps physical GPU pages behind a fixed virtual address so the reserve can be lent to decode and taken back in time — yet the paper’s own measurements show that simply lowering the chunked-prefill token limit recovers more memory, with median TTFT differing by about 1%.

Jargon

Prefill and decode: The two phases of serving a request: prefill reads the whole prompt in one or a few large passes, while decode then produces output one token at a time. They stress the GPU differently, which is why an engine schedules them differently. See the two phases side by side →
KV cache: The stored keys and values for every token already processed, so attention never recomputes them. It grows with every token of every live request, so the size of the KV pool is effectively the ceiling on concurrency. Work through the memory cost →
Chunked prefill: Splitting a long prompt into bounded pieces and interleaving them with decode steps, instead of letting one huge prefill stall everyone else. The token limit per chunk is the knob, and it is the knob this paper’s measurements end up favouring. See what chunking fixes →
The prefill reserve: GPU memory an engine sets aside so that the largest permitted prefill chunk is always guaranteed room. It is sized for the worst case and idle the rest of the time — the memory this paper tries to win back.
CUDA VMM (virtual memory management): The CUDA API that lets a program keep one unchanging virtual address range while swapping which physical memory sits behind it — releasing that memory is a decommit (the paper: a few milliseconds), attaching it again a recommit (tens of milliseconds). It is the same separation of address from storage that PagedAttention uses for the KV cache, applied here to the reserve instead.
Tensor parallelism (TP1, TP4): Splitting each layer’s weights across several GPUs so they compute one forward pass together; TP4 means four of them. It matters here because sharding the weights leaves more room per GPU for KV cache, which is what shrinks the reserve’s share from 16% to 2.7%.
CUDA graphs: A captured, replayable recording of a fixed sequence of GPU kernel launches, used to cut launch overhead during decode. A captured graph pins the addresses it was recorded with, which is exactly why the allocator has to move physical pages without moving the virtual address. See capture and replay →
TTFT (time to first token): How long a user waits before the first output token appears — the latency prefill dominates. It is the number a smaller prefill chunk is usually assumed to move, and the paper measures that difference at about 1% at the median. Read latency as a distribution →

The news. On August 24, 2026, a paper (arXiv 2608.23658) reported building a working elastic KV-cache allocator: it lends the idle prefill reserve to decode and returns it before the next prefill, without changing the attention kernels. A one-step-ahead scheduler toggles two physical CUDA memory handles behind one contiguous virtual address range per layer, decommitting the reserve during decode and recommitting it before prefill, which preserves CUDA graphs and prefix caching and avoids static overcommit. It works — no out-of-memory event occurred with the dynamic allocator, decommit completes in a few milliseconds and recommit in tens of milliseconds. Then the controlled live-load tests deliver the twist: simply lowering the chunked-prefill token limit recovers more KV capacity at almost the same latency, with median TTFT differing by about 1% between 8,192- and 32,768-token chunks, while the reclaimable reserve itself falls from 16% of KV at TP1 to 2.7% at TP4. Read the paper →

Picture an office floor. Most of it is open desks, and every conversation currently in progress needs a seat there — that is the whole business of the place. Off to one side is a meeting room, kept permanently empty. It is not wasted by accident: it exists so that whenever a large group arrives, there is somewhere to put them immediately, without asking anyone at a desk to stand up. The room is sized for the biggest group you allow through the door, and it sits empty every hour that no such group arrives.

That floor is a GPU, the desks are the KV cache, and the meeting room is the prefill reserve. An engine that runs chunked prefill promises that a prompt will be read in pieces no larger than some token limit, and it has to keep enough memory free to honour that promise at any moment. The reserve is therefore sized by a worst case that most steps never hit, and every byte of it is a byte the KV cache does not get — which, since KV capacity is what caps the number of live requests, converts directly into users you cannot serve.

The paper's answer is the folding partition wall. During a stretch of pure decode, when no prefill chunk is scheduled, the wall opens and the meeting room becomes desk space; before the next group arrives, it closes again. Doing this on a GPU is harder than it sounds, because the decode step has usually been captured as a CUDA graph, and a captured graph is welded to the exact addresses it was recorded with — move the memory and the recording breaks. So the allocator moves the storage without moving the address: it keeps one contiguous virtual address range per layer and toggles two physical CUDA memory handles behind it, decommitting the reserve during decode and recommitting it before prefill. This is the same trick PagedAttention plays on the KV cache — separate the address a kernel uses from the physical memory that backs it — pointed at the reserve instead, and the paper's result is that it works: attention kernels untouched, prefix caching intact, and no out-of-memory event in testing.

And then the measurements ruin it. The room can be reclaimed, but there was a second door all along: you can shrink the meeting room by lowering the largest meeting anyone is allowed to book. Halve the chunked-prefill token limit and the reserve you must hold shrinks with it — permanently, on every step, with no wall to fold and nothing to time. The paper reports that lowering the limit recovers more KV capacity than elastic reclamation does, and measures the latency difference at roughly 1% of median TTFT between an 8,192-token and a 32,768-token chunk. A scheduler constant, changed once at startup, beat a live memory-remapping mechanism at the mechanism's own job.

Worse for the clever version, the prize shrinks exactly where the deployments are. The paper reports the reclaimable reserve falling from 16% of KV at TP1 to 2.7% at TP4 — and the arithmetic behind that is not mysterious. The reserve is sized by one prefill chunk, a quantity fixed in tokens that does not grow when you add GPUs, while the KV pool on each GPU does grow once the model weights are split across four of them. In the office: four floors, each with its own small meeting room, each floor now mostly desks. Meanwhile the cost of the wall does not shrink — recommit still takes tens of milliseconds against a decommit of a few, so the scheduler must still look a step ahead and still be right about it, to win back a fraction that has quietly become rounding error.

Hold one number fixed and the two doors can be priced against each other. Take a GPU with a 40 GB KV pool (illustrative — the paper reports shares, not absolute pool sizes). At TP1, the reserve elastic reclamation can hand back is 16% of that, about 6.4 GB — but only while decode is running, and it must be surrendered again before the next prefill, with tens of milliseconds of recommit standing between the decision and the memory being usable. At TP4 the same mechanism, on the same 40 GB pool, returns 2.7%, about 1.1 GB — while the recommit bill stays exactly where it was. Now price the other door on the identical pool: cutting the chunk limit from 32,768 to 8,192 tokens shrinks the reserve you must hold in the first place, hands the difference to KV on every step rather than only during decode windows, needs no lookahead and no timing, and moves median TTFT by about 1%. What the paper measured is that the mechanism's prize shrinks as the model is sharded — 16% of KV at TP1 down to 2.7% at TP4 — while the chunk limit returns its memory on every step, and recovered more of it in their tests.

	Elastic KV-cache reclamation	Smaller chunked-prefill limit
What it changes	two physical CUDA handles behind one virtual range per layer	one scheduler constant: tokens per prefill chunk
When the memory is available	only during decode windows, then returned	on every step, permanently
Memory recovered	16% of KV at TP1, 2.7% at TP4	more than elastic reclamation, per the paper’s live-load tests
Latency effect	decommit a few ms; recommit tens of ms, on the scheduler’s critical path	~1% difference in median TTFT, 8,192 vs 32,768-token chunks
What it must not break	CUDA graphs and prefix caching — both preserved by the allocator; untouched by the chunk limit
Failure mode if mistimed	recommit arrives late, prefill waits	none of this kind; the reserve is simply smaller

A negative result is worth more than it looks here, because the mechanism was not a strawman: it was built, it held, and it respected two constraints — CUDA graphs and prefix caching — that usually kill this class of idea outright. It lost anyway, to a constant. The transferable lesson is an ordering rule: before adding a mechanism that manages a resource dynamically, check what happens if you simply ask for less of that resource — and check it at the parallelism you actually deploy at, because a share measured at TP1 may be most of the argument, and most of the argument may not survive TP4.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Chunked Prefill

Related explainers

vLLM 0.25 — Retiring PagedAttention — what happens to the virtual-memory idea when the KV cache itself stops needing it
WorldKV — Evict-and-reinsert KV memory — the other way to win KV capacity: decide what to throw away rather than what to reserve
AMD Atom — Prefill/decode disaggregation — the alternative to chunking: stop interleaving the two phases and give each its own hardware

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based