Make NVMe KV-cache loads 2x faster with scheduler-aware preloading — External KV-cache break-even — What does it mean?
The news. On September 10, 2026, a paper introduced py-kvcache, a vLLM KV Offload connector that characterises external KV caching across GPU, CPU and NVMe tiers using synthetic workloads, long-context benchmarks and production traces. It reports 2.0x faster disk loading than LMCache at 80K tokens, a 1.34x contribution from scheduler-aware preloading alone, 1.23x with GPU, CPU and disk caching combined, and performance within about 4% of native vLLM KV Offload. Its more interesting result is negative: loading a cached prefix is not automatically faster than recomputing it, especially for short prefixes or fast H100-class GPUs. Read the paper →
Picture the kitchen. A cook needs a litre of the same base sauce on almost every ticket, and there is a frozen batch of it in the walk-in freezer at the end of the hall. The obvious move is to fetch it — the sauce is already made, and making it again is work. But the hall is long. For a small ladleful, the round trip to the freezer takes longer than simply cooking it, and the cook who walks anyway has made the order slower by being clever.
An external cache hit is not free, and that is the whole point. In a serving engine the frozen batch is the KV cache of a prompt prefix, written out to NVMe because GPU memory could not hold it any longer. Cooking it again is prefill — one forward pass over the prefix that rebuilds every Key and Value the model needs. Fetching it is a disk read. Both produce exactly the same bytes. Only the cost differs, and which one costs less is not fixed.
So the question a serving engine should ask is never "is there a hit?" but "is fetching this hit cheaper than making it again?" — and that reframing is what the py-kvcache paper contributes. A cache that treats every hit as a win has only answered the first question. This one answers the second, and reports the conditions under which the honest answer is no: short prefixes, where the fixed cost of setting up a read swamps a small payload, and fast GPUs, where recomputing is cheap enough that the drive cannot keep up. It is the same compute-versus-bandwidth question the roofline model asks, applied one level down the memory hierarchy — to a cache tier instead of a kernel.
The reason a break-even exists at all is that the two sides scale differently. The walk to the freezer gets longer in proportion to how much sauce you carry: the bytes moved grow linearly with the prefix length. Cooking does not scale that politely. Prefill has to run attention over the prefix, and each new token attends to every token before it, so the compute grows faster than the token count does. A cost that grows in proportion and a cost that grows faster than proportion will cross somewhere, and past the crossing the cheaper option flips. Left of it the stove wins; right of it the freezer does.
Put numbers on the two lines, holding three things fixed: one model, whose KV cache costs about 0.14 MB per token (illustrative); one drive, reading at 5 GB/s after roughly 10 ms of fixed setup (illustrative); and one GPU, which prefills 4,000 tokens in 0.08 s (illustrative).
At a 4,000-token prefix, the saved cache is 560 MB. The drive delivers it in 560 / 5,000 = 0.112 s, plus 0.010 s of setup, so 0.12 s to fetch against 0.08 s to cook. The freezer loses. At an 80,000-token prefix the payload is 20x larger — 11.2 GB, or 2.24 s plus the same 10 ms, so 2.25 s to fetch. Recompute does not merely scale 20x with it: attention over a prefix 20 times longer costs disproportionately more, and at this length the same GPU needs about 3.4 s (illustrative). The freezer wins by more than a second, and somewhere between 4,000 and 80,000 tokens the two lines crossed.
Now change one variable and watch the crossing move. Swap in a GPU twice as fast and the fetch is untouched at 2.25 s while recompute drops to 1.7 s — the same 80,000-token hit that was a clear win is now a loss, because a faster cook pushes the break-even to the right. That is the paper's H100 caveat stated as arithmetic, and it is why the break-even is a property of your deployment rather than of the cache.
KV cache size (Llama 3.1 70B)
Transfer time (0.67 GB KV cache)
NVLink is ~28× faster than PCIe — same-node transfers are near-free.
| Where the prefix comes from | What you pay | When it wins |
|---|---|---|
| GPU memory | nothing moves | whenever the prefix is still resident — nothing has to be transferred |
| CPU memory | one host-to-device copy | the prefix was evicted from the GPU but is still on the node |
| NVMe disk | a disk read, plus fixed per-read setup | long prefixes, where the read finishes before prefill would have |
| Recompute | a full prefill pass | short prefixes, or a GPU fast enough to out-run the drive |
Which leaves the second half of the mechanism: if the break-even is where it is because the walk takes time, send the runner earlier. That is scheduler-aware preloading, and it is the part of py-kvcache that is not about disks at all. A request does not reach the GPU the instant it arrives — it sits in the scheduler's queue while other work finishes. The connector starts its asynchronous direct I/O read during that wait, into a bounded staging buffer, so the bytes are already on their way before the request is admitted.
Preloading does not make the drive faster; it moves most of the read off the critical path, which is a different and better trick. The 2.25 s transfer in the worked example does not shrink — but if 1.5 s of it happened while the request was queued (illustrative), only 0.75 s is visible as latency, and the break-even slides left to include prefixes that were losing before. The paper measures that lever directly: scheduler-aware preloading contributes a 1.34x speedup on its own, and the full disk path lands 2.0x faster than LMCache at 80K tokens while staying within about 4% of native vLLM KV Offload, which puts it close to the built-in path.
The practical reading for anyone running a serving stack: treat external KV reuse as an admission decision, not a cache lookup. A hit rate is not a benefit until you know the prefix lengths behind it, the drive under it, and how fast the GPU it is competing with actually is. A cache tier that pays for itself on long prompts can be a loss on short ones, or on a faster GPU, with the hit rate unchanged and looking healthy the whole time — which is exactly the when-it-helps question that prefix caching itself already forces, now one tier further down.
Goes deeper in: LLM Serving → Prefix Caching → When It Helps, When It Fails
Related explainers
- Partial loads from a lower cache tier — what happens when only part of a saved prefix is still worth fetching
- Predictive prefetching for SSD-backed memory — the same hide-the-disk-latency idea applied to an external memory table instead of the KV cache