What is a partial secondary-tier load in vLLM?

It is a load from a storage tier below the GPU — host memory or, as of vLLM 0.28, a filesystem — that comes back carrying only some of the KV blocks it was asked for. vLLM 0.28 lists the addition as “partial secondary-tier load results (#50321)” and does not describe the interface, so the point to take is the consequence rather than the API: because a KV cache prefix is only usable up to its first gap, a result that can express how far the transfer got lets the engine keep the blocks that arrived and prefill just the remaining tail, instead of discarding the transfer and recomputing everything.

Why does tiering the KV cache to disk need this at all?

Because a tier below the GPU introduces a step the GPU pool does not have. Blocks already resident in the pool are either there or not, and the engine knows which without moving anything. Blocks in host memory or on disk have to be transferred first, and that transfer can come back with less than was asked for — a second way to fall short, independent of whether the cache held the data. An engine that can only record the transfer as success or failure scores every such case as a total loss, spending the fetch and keeping none of it.

How does this relate to prefix caching and eviction?

Prefix caching is what asks the question — it walks a prompt’s block-hash chain looking for the longest run already cached. Eviction is what decides which blocks leave a tier when it fills; the release notes a fix removing quadratic cost from an ARC batch-eviction path on the offload tier (#50992). Tiered offload sits between the two: instead of evicting a prefix out of existence, it demotes it to a slower tier, and the partial-load result is what determines how much of that demoted prefix is still worth something when it is asked for again.

vLLM 0.28 ships disk-backed tiered KV offload — Partial loads from a lower cache tier

TL;DR

What is it: The news anchor is vLLM 0.28.0, which adds disk offloading underneath the engine’s KV-cache offload connector; the concept it makes concrete is the partial secondary-tier load — a fetch from that tier which, rather than reporting one verdict, comes back carrying only part of the cached prefix it was asked for.
Why it’s needed: KV cache capacity is what caps how many requests a GPU can serve at once, so a storage tier below the GPU decides whether a finished prefix is thrown away or merely put somewhere slower — and whether a later request pays for it again.
vs previous: Instead of a transfer that either delivers the whole matched prefix or counts as a miss, a partial result hands the scheduler a shorter prefix it can still use and lets it recompute only the tail, turning an all-or-nothing move into a salvage.

Jargon

KV cache: The stored keys and values for every token a model has already read, so attention never recomputes them. It grows with every token of every live request, which makes the size of the KV pool the practical ceiling on concurrency. Work through the memory cost →
KV block: The unit the pool is managed in: the keys and values for a fixed number of consecutive tokens (16 is a common choice), stored contiguously. Blocks are the reason a cache can be shared and reused at all, and the reason order matters — block 29 of a sequence is meaningless without blocks 1 through 28. See how blocks are allocated →
Prefix caching: Reusing the KV blocks of a prompt prefix that some earlier request already computed, instead of prefilling it again. The engine hashes the prefix block by block and looks for the longest chain it already holds. Follow the block-hash chain →
KVConnector / OffloadingConnector: vLLM’s pluggable interface for moving KV data in and out of the GPU pool. The OffloadingConnector is the implementation that copies blocks to host memory; vLLM’s documentation describes it as configured with a block size in tokens and a byte budget for the host side.
Secondary tier manager: The component that owns a storage level below the GPU pool — CPU memory, and as of this release a filesystem. vLLM 0.28 lets you supply your own via a module_path, so a storage backend can be added without forking the engine (release note: out-of-tree secondary tier managers, #51007).
Partial load result: A load from a lower tier that comes back with only some of the blocks it was asked for. The release notes the addition as partial secondary-tier load results without describing the interface, so read what follows as the consequence of a result that is no longer a single verdict: the caller can keep the usable prefix rather than discard the whole transfer.
ARC (Adaptive Replacement Cache): A classic eviction policy that balances recently used against frequently used entries and adjusts the split as the workload shifts. It is what decides which blocks leave the CPU tier; this release notes a fix that removed quadratic cost from its batch eviction path (#50992).
Recompute: Re-running prefill over tokens whose KV blocks are not available, to rebuild them. It is the fallback price of every cache miss, and the quantity a partial load is trying to shrink rather than eliminate.

The news. On August 26, 2026 the vLLM project released v0.28.0 — 584 commits from 270 contributors, 76 of them new. Under Tiered KV cache offloading the release lists disk offloading support (#49644), out-of-tree secondary tier managers via module_path (#51007), partial secondary-tier load results (#50321), tiering metrics (#48798), and a canonical CPU layout for parallelism-agnostic offload (#48414); a separate note records that quadratic ARC batch eviction was avoided (#50992). Elsewhere the release adds an adaptive budget for speculative scheduled input tokens (#51725), reported as ~60% better DSpark time-to-first-token, plus Decode Context Parallel and fused FlashKDA kernels for Kimi-K3, optional shared-expert sharding saving ~17 GiB per GPU, Model Runner V2 encoder/prefill/decode disaggregation, and ROCm support for Kimi-K3 and DeepSeek V4.

Picture the library. The open shelves at the reading desk are small and expensive, and everything anyone is actually working on has to be there. Behind the desk is the closed stack, larger and slower, where a librarian can fetch from in a minute. Somewhere across town is the depository — enormous, cheap, and reached by a van that runs on its own schedule. Nothing in this arrangement is unusual; what is unusual is that the sets in this library only make sense read in order, so a request is never for one volume but for volumes 1 through n.

That library is a GPU serving an LLM. The open shelves are the KV cache block pool; the closed stack is host memory; the depository, new in this release, is a filesystem. A KV block holds the keys and values for a fixed run of consecutive tokens, and a conversation's cache is a chain of them. When a new request arrives sharing a prompt prefix with an old one, prefix caching walks that chain hash by hash and asks for the longest run it already has. The lookup is therefore not "do you have this?" but "how far along this chain can you take me?" — and the answer has always been a number, not a yes.

Block Table: Logical → Physical

→

16 tokens

→

16 tokens

→

P14

16 tokens

→

16 tokens

Contiguous logical blocks → scattered physical blocks in GPU memory

The block table above is why the number is what matters. Every logical position in a sequence points at some physical block, and the engine can only start the model where that mapping is unbroken. A gap is not a small loss; it is a hard stop — you can use everything before it and nothing after. In the library: volumes 1 to 28 plus volume 40 is worth exactly as much as volumes 1 to 28, because you cannot read 40 without 29 through 39. This is the constraint that makes a tier below the GPU awkward: a tier you have to fetch from is a tier whose answer can arrive incomplete, and an incomplete chain is worth only its unbroken head.

Which is the gap vLLM 0.28 closes. The release note is one line — partial secondary-tier load results (#50321) — and it does not describe the interface, so the interesting part is what such a result is for. A result that can express a length rather than a verdict is one the caller can act on: the van returns with volumes 1 to 28, the desk keeps them, and the reader re-derives 29 onward from what arrived — prefill over the tail only, rather than prefill over everything. A transfer that could previously only be spent or wasted can now be spent partway, and how far it got is the whole of what it is worth. The other tiering entries in the release read as the supporting work for that: a canonical CPU layout for parallelism-agnostic offload (#48414), a fix removing quadratic cost from ARC batch eviction (#50992), and tiering metrics (#48798).

Hold one prefix fixed and price the two answers against each other. Take a shared prompt prefix of 40 blocks at 16 tokens each — 640 tokens (illustrative; block size is configurable and the release states no workload numbers). The GPU pool no longer holds it, the disk tier is asked for it, and the fetch comes back with 28 blocks — the release does not say why a load would return short, so take the cause as unspecified and the arithmetic as the point. Under an all-or-nothing result the engine records a miss and prefills the whole prefix: 640 tokens of recompute. Under a partial result it keeps the 28 blocks that arrived — 448 tokens — and prefills only the 12-block tail: 192 tokens of recompute. Same fetch, same failure, same hardware: the work saved goes from 0 to 448 of 640 tokens, about 70%. Note what the change did not do — it did not make the disk faster, and it did not raise the hit rate. It changed how a partial success is reported, and a partial success stopped being scored as a failure.

Change in vLLM 0.28	Release note	What it makes possible
Disk offloading for the CPU offload connector	disk offloading support (#49644)	A third tier below CPU memory, so a prefix that leaves host memory is not automatically gone
Partial results from a lower tier	partial secondary-tier load results (#50321)	A load can return a usable prefix of the match instead of one success-or-failure verdict
Pluggable tier managers	out-of-tree secondary tier managers via `module_path` (#51007)	A storage backend can be attached without forking the engine
Canonical CPU block layout	a canonical CPU layout for parallelism-agnostic offload (#48414)	Offload that is described as parallelism-agnostic — the note states the layout, not what it is read back under
Eviction cost on the CPU tier	quadratic ARC batch eviction avoided (#50992)	Batch eviction stops scaling quadratically as the tier fills
Per-tier visibility	tiering metrics (#48798)	Measurement attributed to a tier rather than to the cache as a whole; the note does not enumerate which metrics

The transferable idea is about the shape of a cache's answer, not about vLLM. A cache that reports hit or miss is describing itself as a single object; a cache built out of ordered pieces is really being asked a different question — how much of this can you give me — and it can only answer that question if the layer beneath it is allowed to answer it too. Blocks sitting in the GPU pool are simply there or not, and the engine knows which without moving anything; blocks in a tier below it have to be fetched, and a fetch is a second thing that can come up short independently of whether the cache held the data. Adding a tier therefore adds a failure mode that a single-tier cache does not have — and the value of the tier depends on whether the engine can bank an incomplete answer or has to throw it away.

Goes deeper in: Inside vLLM → KV Cache Manager → The Longest Match

Related explainers

Chunked prefill vs elastic KV-cache reclamation — the other way to widen the KV pool: win memory back inside the GPU rather than adding a tier beneath it
WorldKV — Evict-and-reinsert KV memory — what an eviction-and-return scheme looks like when the index is semantic rather than a prefix hash
vLLM 0.25 — Retiring PagedAttention — the earlier release that reshaped how the same block pool is addressed

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based