vLLM 0.34 — Extensible KV cache sizing — What does it mean?
The news. On November 16, 2026, the vLLM project released v0.34.0, a 1,061-commit release whose headline change turns the extensible KV cache on by default on CUDA and on ROCm, AMD's GPU software stack (PR #56492). The release notes give one example of the effect, MiniMax-M3 on H200 growing from 433K to 672K KV-cache tokens, and state the trade-off openly: per-token decode latency can rise 6–18% on some H200 models at concurrency 128. The same release makes Model Runner V2 the only GPU runner and adds speculative-decoder plugins; this page covers the KV sizing change only. Read the release →
Picture a new street where the city must print a delivery route before any house is built. The route lists house numbers, so the numbers have to be fixed on day one, yet nobody knows how much land the construction trailers will need. The old answer was to guess the trailers' footprint, build houses on the rest, and leave a strip of lots empty in case the guess was wrong. The extensible KV cache answers differently: assign every house number on day one, leave the lots empty, let the trailers do their work, measure the land they really used, and only then build houses on everything that is left. The route still works, because every house sits at the number the route already lists.
In GPU terms, the delivery route is a CUDA graph. Capture records each kernel's memory addresses and replay reuses them, so the KV cache must already sit at its final address before vLLM captures any graph. That is why the pre-0.34 startup sized the KV cache first and captured graphs second: a profiling run measured peak activation memory, the engine allocated the cache from what was left, and everything allocated after that (CUDA graph memory, attention workspaces, sampler buffers, speculative-decoding drafts) had to fit inside a slack. PR #56492 describes that slack as the gpu_memory_utilization=0.92 default: 8% of the device, about 15 GiB of a 192 GiB GPU, left idle to absorb estimation error.
The extensible cache removes the ordering problem by separating an address from the memory behind it, using the CUDA driver's virtual memory management API:
- Reserve. Before capture, vLLM reserves the cache's full capacity as a range of virtual addresses, with no physical memory behind it.
- Back only what is touched. Capture and warmup run against that range, and only the blocks they actually touch get physical pages.
- Measure. After warmup, each worker measures the memory that is really free and the transient peaks it really saw.
- Commit. The engine picks the final block count in one step and maps more physical memory into the same reserved range.
Because the addresses never move, every layer's view of the cache and every already-captured CUDA graph stay valid while the cache grows underneath them. The default can therefore use the whole device, keeping a measured reserve for transients plus a margin instead of a fixed 8% guess. A side effect is faster startup: there is no throwaway capture pass to estimate graph memory, and the PR measures engine initialization, excluding weight loading, falling 21–34% across its eight test models.
If this sounds like operating-system virtual memory, it is the same idea. Each program in the diagram below sees a contiguous range of addresses, while the physical pages behind it sit wherever the OS placed them. vLLM borrows the GPU driver's version of that mapping for one reason: so an address can be handed out before the memory behind it is decided. Keep this apart from PagedAttention's block table, which maps each request's logical blocks onto KV blocks inside the cache. The extensible cache changes how big the cache is, not how blocks inside it are assigned to requests.
Hold three things fixed: Qwen3-32B, one GB200 with 189 GiB, tensor parallelism 1, the PR's own benchmark. With the old sizing the KV cache came out at 103.1 GiB; with the extensible cache it is 113.0 GiB, a gain of 9.9 GiB (+9.5% tokens). Now set that gain against the slack it replaced: 8% of 189 GiB is about 15.1 GiB. The engine handed back 9.9 GiB of it, so roughly 5.2 GiB of the old slack was real cost: CUDA graphs, warmup transients and the margin the new path still keeps (our arithmetic from the PR's table). The slack was not all waste; the point is that only measurement can tell which part was waste. With a BF16 cache, Qwen3-32B stores about 256 KiB of keys and values per token (64 layers × 8 KV heads × 128 dims × 2 tensors × 2 bytes, our arithmetic), so the extra 9.9 GiB holds roughly 40,000 more tokens, about ten more 4,000-token conversations resident at once.
| Aspect | Pre-0.34 default (estimate) | 0.34 default (extensible) |
|---|---|---|
| When the KV cache is sized | Before CUDA graph capture, from a profiling run | After capture and warmup, from measured free memory |
| How graph memory is counted | Ignored, or estimated with a throwaway capture pass | Measured directly after capture |
Default gpu_memory_utilization | 0.92, 8% of the device idle (PR #56492) | 1.0 when unset; acts as a cap (release notes) |
| KV capacity at default settings | Baseline | +4–10% tokens on eight GB200 models (PR #56492); MiniMax-M3 on H200, 433K → 672K tokens (release notes) |
| Engine init time, excluding weight loading | Baseline | 21–34% faster on the PR's eight GB200 models (PR #56492) |
| Per-token decode latency | Baseline | TPOT +6–18% on some H200 models at concurrency 128 (release notes) |
Three limits come with the change. First, a bigger cache can cost per-token latency. The release notes report TPOT rising 6–18% on some H200 models at concurrency 128 but do not explain the cause; one possible explanation is that more resident requests create larger decode batches, which take longer per step even when total throughput improves. Operators who care more about per-token latency can opt out with --no-enable-extensible-kv-cache, or cap memory with an explicit --gpu-memory-utilization (the release notes suggest 0.9). Second, the gain depends on how much of the old slack was already in use. DeepSeek-V4-Flash gains only 4.1%, because its CUDA graphs and workspaces need about 15 GiB per GPU after warmup, nearly the whole slack. At an explicit gpu_memory_utilization=0.92, the new path even produces a cache 1–4% smaller than before, because it now subtracts graph and warmup memory the old path left to the slack. Third, it is not universal: where a configuration cannot use the virtual-memory path, vLLM falls back to estimate-based sizing automatically, and the release lists follow-up fixes for startup out-of-memory errors and allocator fragmentation.
Goes deeper in: Inside vLLM → Compilation & CUDA Graphs → What Startup Buys You
Related explainers
- TensorRT-LLM KV cache manager V2 — Distributed KV pool rebalancing — the opposite direction: resizing KV pools at runtime, which forces captured CUDA graphs to pause.
- vLLM 0.25 — Retiring PagedAttention — Model Runner V2, the runner the extensible cache is enabled with.