Non-prefix KV reuse study — Targeted KV repair — What does it mean?
The news. On November 4, 2026, a paper titled "Where Should a KV-Cache Repair Budget Go?" reported a controlled study of non-prefix KV reuse across three 7B–8B models and four datasets spanning dialogue memory, open-domain QA and API documentation. Its finding: the damage from stitching independent caches is spread over the other chunks, so the repair budget is better spent on the target. Read the paper →
Picture a company that records every meeting separately. Each note-taker writes tidy minutes of only the meeting they attended. Before a big decision, an assistant staples the relevant minutes into one briefing, in whatever order the question needs. That is non-prefix KV reuse: each document chunk is run through the model once, alone, its Key/Value state is cached, and later requests concatenate those caches instead of re-reading the text. It avoids re-running the prefill work for those chunks, which is why serving systems want it.
The staple is the problem. A token's KV state depends on every token before it, so minutes written in isolation never recorded what the meeting before them decided. The page that says "we agreed to the second option" is fine on its own and wrong in the binder, because nobody told it which option was second. Normal prefix caching avoids this by reusing only an exact shared beginning, where the context is identical by construction; the block-hash chain breaks at the first token that differs.
The study asks a practical question: if you can afford to recompute only a small slice, which pages of the binder should the reviewer fix? The authors vary the distractor chunks around a fixed answer-bearing chunk and compare cached reuse, recomputation, inserting the target's full-prefill KV, and reordering. Two results shape the answer. Reuse loses 19–100% of the context-dependent questions that full prefill answers, and it loses more when the other chunks are of the same kind as the target, such as several similar API documents. And because the recoverable damage is spread thinly over all the other chunks, restoring a few of them recovers little.
| Strategy | What it does | What the study reports |
|---|---|---|
| Full prefill | Reads the whole prompt in order, no reuse | The reference: the answers every other row is measured against (paper) |
| Plain non-prefix reuse | Concatenates independently encoded chunk caches | Loses 19–100% of context-dependent questions that full prefill answers (paper) |
| Insert the target's full-prefill KV | Pastes in the target's ideal KV | Recomputing the target against the reused cache beat this in all seven settings tested (paper) |
| Spread repair over distractors | Spends the budget on the other chunks | Recovers little, since the damage is diffuse (paper) |
| Reorder (InfoFlow-style) | Moves likely-relevant chunks near the question | Beaten by target recomputation (retriever-picked chunk, moved next to the question) at the same 5% budget, in six of nine open-domain QA and API-documentation settings (paper) |
| Recompute the target | A retriever picks the top chunk, moves it next to the question, recomputes it against the reused cache | Best use of the budget in the open-domain QA and API-documentation settings above; not significantly better than a token-level selector on dialogue memory (paper) |
Why does recomputing the target work even though it does not reproduce the ideal state? The paper's explanation is contextual compatibility: the recomputed target is read against the very cache it will sit beside, so it fits that cache better than the ideal stand-alone KV does, even though the recomputed KV stays far from the full-prefill KV. In meeting terms, the reviewer does not rewrite the page into what a full attendee would have written. They reread the key page with the other minutes in front of them so it stops contradicting its neighbours.
Here is a worked example with illustrative numbers (the paper's exact budget accounting may differ). A prompt holds 20 retrieved chunks of 200 tokens each, so 4,000 tokens in total, and the repair budget is 5%, which is 200 tokens. Spread evenly, that is only 10 tokens of fix-up per chunk, a thin sliver on every page, and the study says the damage is too diffuse for slivers to help. Concentrated, it is one whole chunk recomputed, and that chunk is the one the question reads. The risk is the choice: if the pick is wrong, the 200 tokens fix a page nobody needed. The paper reports that question attention over the reused caches selects the target in at most 18.8% of development questions, while a retriever finds it more often, so finding the target is the hard part.
Read the limits before copying the recipe. The target-recomputation win holds for open-domain QA and API documentation; on dialogue memory, recomputing chunks never significantly beat the token-level selector chosen for each setting. The study covers three 7B–8B models, a fixed 5% budget in the headline comparison, and controlled distractor sets, so it does not say how the result scales to much larger models or to messier retrieval. It tells you where to look first, not that non-prefix reuse is safe by default.
Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → Why Full-Prompt Hashing Fails