vLLM 0.29 — Mamba prefill checkpoints — What does it mean?
The news. On September 9, 2026, the vLLM project released v0.29.0 — 594 commits from 277 contributors. Among them, PR #52789 enables prefix caching for Mamba layers by storing internal prefill checkpoints, which the release notes credit with a 9%–25% improvement in time-to-first-token. A companion change (#52216) promotes the
prefix_cache_retention_intervalsetting from an environment variable to a proper CLI argument, and flips its default for SSM and sliding-window models from dense retention to0. Read the release →
Keep the stock pot in mind, because the whole difficulty is that a pot has no slices.
When an attention layer reads a prompt it writes down two vectors — a key and a value — for every single token, and it keeps them. Ask it later what the prompt looked like at token 2,000 and it can answer: those entries are still there, untouched by everything that came after. That property is the entire basis of prefix caching. vLLM hashes the prompt in fixed-size blocks and, when a new request opens with the same blocks, hands it the already-computed keys and values instead of recomputing them. The cache is a stack of receipts, and any prefix of a stack of receipts is still a stack of receipts.
A Mamba layer keeps no such record. It carries one fixed-size recurrent state, and each arriving token stirs itself into that state and is gone. After 8,000 tokens you hold one state — not 8,000 of anything — and it is not a sum you can take back apart. There is no entry for token 2,000 to look up, no boundary to cut at, and no general way to run the last 6,000 updates backwards. That constant memory is the reason to use an SSM in the first place; it is also the reason its prefill could not be cached.
So a hybrid model could only ever half-use prefix caching. The attention layers hit the cache and skipped their share of the work, while the Mamba layers in the same request and re-read the prompt from an empty pot, every time.
vLLM 0.29's answer is to manufacture the cut points the architecture does not provide. Partway through prefill, at intervals, the engine copies the recurrent state out and keeps it. (The release notes name the mechanism and its effect; they do not specify where the checkpoints land or how a restore is stitched together.) That is the jar ladled off mid-simmer. A later request sharing the first several thousand tokens of the prompt no longer starts cold: it loads the nearest checkpoint it can match, and replays only the tokens after it.
The cost is blunt, and it is memory. A checkpoint is a full copy of the state for every Mamba layer, so the more often you snapshot, the more GPU memory you spend to make restarts cheaper — memory that would otherwise hold KV blocks for other requests. The retention interval is the dial. Note where 0.29 set it: the default for SSM and sliding-window models moved from dense retention to 0, which means you now choose how much to retain instead of getting dense retention for free — unless you run a hybrid model with EAGLE or MTP speculative decoding (a small draft model proposes several tokens and the big one verifies them in a single pass), where dense retention is restored automatically (#55760, #55861).
| Layer type | What it stores | Grows with | Can a prefix be reused? |
|---|---|---|---|
| Attention | a key and a value per token, per head, per layer | sequence length | Yes — hash fixed-size blocks, hand back the ones that match |
| Mamba / SSM | one fixed-size recurrent state per layer | nothing; the size is constant | Not directly — the state has no per-token parts to slice |
| Mamba with prefill checkpoints (vLLM 0.29) | that state, plus a saved copy taken partway through prefill | how many checkpoints you retain | Yes — restore the nearest checkpoint, replay the tail |
Put numbers on it, holding one thing fixed: a 6,000-token shared system prompt that every request in a support workload begins with, on a hybrid model whose Mamba layers checkpoint every 1,024 tokens (illustrative — the release notes give neither an interval nor where along the prompt a checkpoint lands, so treat the spacing below as a worked shape, not a spec).
Request one is a cold miss and pushes all 6,000 tokens through prefill. Request two arrives with the same prefix. Its attention layers hit the block-hash cache, as they always did. Its Mamba layers now match the checkpoint taken at token 5,120 — the highest multiple of 1,024 that fits inside 6,000 — and replay only the remaining 880 tokens. That is 880 of 6,000 tokens re-read, roughly 15% of the prefill work, where it used to be all of it.
The end-to-end effect is smaller than that ratio, and it should be: the attention layers were already hitting the cache, and prefill is not the whole of time-to-first-token. vLLM's own measurement across its benchmarks is a 9%–25% TTFT improvement (#52789) — a real number, on a mechanism that used to be simply unavailable.
Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → Block-Hash Chain (vLLM APC)
Related explainers
- vLLM 0.29 — Batch-sharded sampling — What does it mean? — the other memory change in the same release: why every tensor-parallel rank was holding its own copy of the whole batch's logits, and what happens when they stop.