KV-Repurpose — Position-adjusted segment KV reuse — What does it mean?

The news. On October 9, 2026, Taiga Ikeda, Daisuke Miyashita and Jun Deguchi of Kioxia posted KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse. It extends vLLM v0.20.0 with under 900 lines of Python, released under Apache-2.0, and tests the idea on GPT-OSS-120B on one H200 GPU with the cache on NVMe SSD. The authors say they are not proposing a new quality-recovery method; they want a simple, reproducible baseline to measure what naive non-prefix reuse is worth. Read the paper →

Picture the print shop. A customer orders a course reader: a new cover page, then chapters A and B, then a question sheet. The shop already printed chapters A and B last week for a different reader. The prefix-caching shop refuses to reuse them because the cover page is different, and reprints everything. That rule is not stupidity. In a transformer, every token's key and value are computed while attending to every token before it, so a document's KV after cover page X is genuinely not the same numbers as its KV after cover page Y. Serving engines therefore hash the prompt block by block from the start, and the first differing block ends the match. Why full-prompt hashing fails walks through the hash chain; one changed token at the front misses everything after it.

KV-Repurpose takes the other side of that trade. The prompt is split by delimiter tokens into segments, and each segment is looked up in the cache on its own, so a miss on the cover page no longer prevents a hit on chapter A or chapter B. The paper contrasts this with LMCache, which checks chunks from the first one onward and stops at the first miss.

Reusing a chapter in a new binder raises two problems, and the paper fixes only the first.

Problem one: the page numbers are wrong. Position enters attention through RoPE, which rotates each key by an angle set by its position. A document cached right after an 80-token system prompt and reused after a 2,016-token one (illustrative positions) would carry the wrong rotation. KV-Repurpose stores each segment's keys unrotated and applies RoPE for the current position at reuse time — the page stamp goes on at binding, not at printing.

Problem two: the chapter never read the new cover page. The paper states that this adjustment "does not recompute attention with the preceding prefix." The reused KV is exactly what the document computed while reading its original front pages, so it carries no information about the new system prompt or about other documents placed before it. Methods such as CacheBlend and EPIC (position-independent caching) try to repair this by recomputing a few tokens; KV-Repurpose deliberately leaves it unrepaired to measure the naive baseline.

The missing cross-notes have a mirror image: notes from the old cover page that the chapter did keep. In a side experiment the authors put "My name is {name}" before a document, cached only the document segment, then reused that cache without the name. Across three trials per name, the model produced the name at least once for 35 of 100 names; none leaked when the cache was built without the name. That is why the testbed builds every document cache in a dedicated request with a neutral system prompt, never from a real user's request: a cached segment can leak what came before it.

The table is the quality side of the trade. Single-document question answering and summarization barely move, while multi-hop questions that combine facts from several documents lose a large share of their score. The paper reports this as a limitation without isolating the cause; an oracle test that kept only the relevant documents recovered part of the gap but not all of it. The reading that fits the mechanism is the print-shop one: chapters bound side by side never wrote notes about each other.

Now the speed side. Hold two things fixed: one 64K-token document, requested behind 100 different prefixes (different system prompts, chat histories or timestamps). Exact-prefix caching may end up storing up to 100 × 64K = 6.4M tokens of KV for that document, one prefix-dependent copy per prefix, and it prefills the document again each time it meets a new prefix. Segment reuse stores one 64K-token copy, a 100× smaller footprint for the same document, and prefills only the prefix and the question. The paper's simulated RAG workload shows what that does to a fixed cache: 2,500 requests drawn with Zipfian popularity, where only 20% share a fixed date prefix. At 30 GB of cache, prefix caching hits 3.3% of tokens and KV-Repurpose hits 63.9%; even at 1 TB, prefix caching reaches only 17.9%. With that hit rate, the paper reports lower p50 and p95 time to first token, with the latency advantage appearing once documents pass roughly 16K tokens.

Where does this leave a serving engineer? Segment reuse is a capacity and latency win for RAG traffic where each answer mostly comes from one long document, and a quality risk where the answer has to connect several. Exact-prefix caching remains the safe default because it never changes the numbers the model sees; when it helps and when it fails is exactly the question of how often your prompts really share an opening. If they rarely do, as in the retrieve-then-generate pattern with request-specific timestamps, the cache is mostly empty, and repositioned reuse is the lever — measured first on your own multi-document tasks.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → When It Helps, When It Fails

Related explainers

• Request Order Matters — History-dependent selective KV reuse — what happens when you do recompute a few tokens to repair repositioned KV, and why the result can depend on which requests ran before.
• CacheWeaver — Prefix-cache-aware evidence reordering — the opposite approach: keep exact-prefix caching and reorder retrieved chunks so more prompts share an opening.