TensorRT-LLM makes KV cache manager V2 the default — Distributed KV pool rebalancing — What does it mean?

The news. On August 31, 2026, NVIDIA released TensorRT-LLM v1.3.0rc25 and made KV cache manager V2 the default for the DeepSeek, GLM, GPT-OSS, Mistral, Kimi, MiniMax, Nemotron, Qwen and Gemma families. The notes say V1 will be deprecated, existing models will migrate gradually, and new models go straight to V2. The release notes list what V2 gained in this cycle: DSA support, zero-copy token passing, cold-page codecs (compressing blocks that have gone cold), and distributed pool rebalancing — plus fixes that say a lot about what rebalancing costs, including suspending CUDA-graph padding dummies before a rebalance and synchronising host-tier quotas across ranks. Read the release →

Picture the car park. On opening day you painted a line down the middle: compact cars on the left, vans on the right. You had to paint it before anyone arrived, because you cannot hand out a bay you have not marked out yet. Then the day happens. The compact side fills by ten, cars start being turned away, and the van side sits three-quarters empty until closing. Nothing is broken. You are simply running a smaller car park than you built, because a line you painted in the dark is holding capacity on the wrong side of it.

A serving engine does not hold one KV cache — it holds several pools, and the split between them is a guess made at startup. The blocks are all the same GPU memory, but they are not interchangeable: ordinary attention keys and values, a sparse-attention indexer's key cache, a state-space model's convolution state and a speculative decoder's mirrored draft cache all have different shapes and different lifetimes, so each gets its own pool with its own block layout. TensorRT-LLM decides the carve-up from a pool_ratio setting at startup, which is the moment the engine knows least about the traffic it is going to see. A pool that runs dry starts evicting cached prefixes and recomputing work it had already done, while the neighbouring pool holds blocks nobody is asking for.

Rebalancing repaints that line while the park is open: capacity moves from the idle pool to the pressured one, and the startup guess stops being permanent. That is the whole idea, and stated like that it sounds easy — it is an allocator moving a boundary. The reason it arrived as a multi-pull-request effort rather than a one-line change is that two parts of a modern serving engine are built on the assumption that this memory never moves.

The first is the shuttle driver. Decode is a sequence of tiny kernels, and launching each one from the CPU can cost more than running it, so engines record the whole sequence once and replay the recording. A captured CUDA graph has its buffer addresses baked in, so you cannot move a pool out from under a graph that is mid-replay — the replay would read whatever now sits at the address it memorised. The release notes do not spell out the failure mode they were fixing; what they record is the fix — TensorRT-LLM suspends the CUDA-graph padding dummies before the rebalance adjusts the pools (PR #16157) — and that is exactly what you would do to stop a replay reading an address that moved underneath it. Wave the shuttle to a stop, repaint, let it go again.

The second is the attendants. A model spread over eight GPUs runs them in step: the ranks meet at collective operations, and a collective is only correct if every participant agrees on the shapes involved. If one rank rebalances and another does not, the next collective is talking about blocks that no longer exist — the kind of mismatch that tends to surface as a hang rather than a clean error. The release makes the rebalance safe under tensor, context and pipeline parallelism and under attention data parallelism (PR #17391), and separately synchronises the automatic host-tier quota and its fallback decisions across ranks (PR #17717). The notes give the scope of those fixes, not the protocol behind them. Every attendant repaints at the same moment, or none of them does.

Worth being straight about what the source does and does not say: these are release notes, not a design document. They record that V2 is the default for the listed model families, that rebalancing exists, and which surrounding cases had to be hardened — not whether rebalancing itself is on by default, nor what triggers it. The policy — which pool gives up blocks, by how much, and on what trigger — is not described in the release, so treat any specific account of the heuristic as unverified until NVIDIA documents it.

Put numbers on the stranded bays, because the size of the mistake is the reason anyone bothered. All the values below are illustrative — the release publishes no capacity measurements — but the arithmetic composes the way the diagram above does.

Hold the card fixed: one 80 GB GPU, with weights and activation workspace taking 20 GB, leaves 60 GB for the KV budget. Hold the split fixed too: a pool_ratio that gives 85% to the attention KV pool and 15% to the indexer's key cache carves that into 51 GB and 9 GB. Now let the traffic disagree with the guess — the day's requests only ever need about 4 GB of indexer cache, so 5 GB of the second pool is never touched. Using the per-token formula in the breakdown above, a 32-layer model with 8 key-value heads of 128 dimensions at FP16 stores 2 × 32 × 8 × 128 × 2 bytes ≈ 131 KB of cache per token. That leaves ~5 GB stranded — raw capacity for roughly 40,000 tokens of cache you have already paid for, about twenty 2,000-token prefixes before allocator and block overhead. Rebalancing is the mechanism that hands those bays back, and every hardening fix above is the price of handing them back without stopping the car park.

Goes deeper in: Inside vLLM → KV Cache Manager → Where Blocks Come From

Related explainers

• vLLM 0.28 ships disk-backed tiered KV offload — the other direction the same budget can stretch: moving KV down a tier instead of sideways between pools.
• Tangram's per-head KV cache budgets — the same stranded-capacity problem one level finer, inside a single pool, per attention head.