What is selective KV recomputation?

It is the practice of re-running only some of a RAG prompt's retrieved chunks through the model in their real, concatenated context, while reusing the precomputed keys and values for the rest. The chunks you re-run regain the cross-chunk attention that a per-chunk cache never had; the chunks you skip cost only the work of loading and joining their caches.

Why does concatenating precomputed KV caches hurt answer quality?

Each chunk's cache was computed while that chunk was alone, so it holds no attention between that chunk and any other. Pasting the caches end to end gives the model a context that looks long but was never read as one document, and the damage grows with how much of the answer's evidence is spread across chunks.

How much faster is it?

The abstract reports two results separately: a RULER score 9.7 points above a baseline that recomputes caches without the concat-aware fine-tuning, measured on a 124,000-token input, and a time to first token 80% below full attention, for which it does not state the input length. It does not disclose the recompute fraction behind those numbers in the abstract.

How does this relate to prefix caching?

Prefix caching reuses a KV cache across requests that share the same opening text, which is safe because the shared part really was computed together. Reusing arbitrary retrieved chunks is the harder case: the pieces were never computed together, so the reuse is an approximation rather than an identity.

LLM· 2026-09-11

Cut RAG TTFT 80% with selectively recomputed KV-cache chunks — Selective KV recomputation — What does it mean?

TL;DR

What is it: A September 2026 arXiv paper pairs concat-aware fine-tuning with selective KV recomputation: when a RAG prompt is assembled by gluing together the precomputed KV caches of many retrieved chunks, re-run only a chosen subset of those chunks through the model instead of all of them or none.
Why it’s needed: Prefill is the part of a long RAG prompt the user actually waits through, and the paper reports 80% lower time to first token than full attention (without stating the input length behind that figure) — the difference between an answer that starts typing and one that sits there.
vs previous: The previous approach was to reuse every cached chunk as-is, which is fast but throws away the cross-chunk attention that makes a long context cohere; against the recompute-only baseline this method scores 9.7 points higher on RULER.

Jargon

RAG: Retrieval-Augmented Generation. Before answering, the system searches a document store, pulls back the most relevant chunks, and pastes them into the prompt. Walk the pipeline
Prefill: The first pass, where the model reads the whole prompt at once and builds its internal state for it. Long prompts make prefill the expensive half. See prefill vs decode
Full attention: Prefilling the whole assembled prompt from scratch, reusing nothing, so every token attends to every earlier token. It is the quality reference point and the slow end of this tradeoff.
KV cache: The keys and values the model computed for every token it has already read, kept so it never has to read them again. How it works
TTFT: Time to first token — how long the user stares at nothing before the first word appears. Prefill is most of it. Serving metrics
Cross-chunk attention: Attention that runs between two retrieved chunks, so the model can connect a fact in chunk 3 to a fact in chunk 17. It is exactly what a cache computed per-chunk never contains.
Prefix caching: The production practice of reusing a KV cache across requests that share an opening prefix. Reuse across arbitrary chunks, as here, is the harder cousin. Where it helps and fails
RULER: A long-context benchmark that scores whether a model can actually use text far back in its window, rather than just accept it. The paper's headline number is a RULER score at a 124,000-token input.

The news. On September 9, 2026, Fumihiko Tachibana, Daisuke Miyashita and Jun Deguchi posted Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both? to arXiv. The paper takes a RAG setting where many retrieved chunks are concatenated into one long prompt, notes that reusing each chunk's precomputed KV cache cuts prefill cost but leaves it unclear whether answer quality survives at very long inputs, and proposes doing two things at once: fine-tune the model with concatenated caches present, and recompute only a subset of the caches at inference. On a 124,000-token RULER input the combination scores 9.7 points above recomputing alone. Separately, the abstract reports TTFT 80% below full attention, without stating the input length behind that second figure. Read the paper →

Picture the kitchen at the start of service. Every dish on the menu has already been cooked, each in its own pot, each kept warm. An order comes in for five of them, and the plates go out in seconds — nothing had to be cooked while the customer waited. The speed is real, and so is the thing it costs: no pot ever simmered with any other, so what lands on the table is five separate dishes sharing a plate rather than a meal.

That is what happens when a RAG prompt is built from precomputed KV caches. Each retrieved chunk was prefilled once, on its own, and its keys and values were stored. At query time the system pastes those caches end to end and skips straight to generating. Most of the waiting is gone, because prefill is most of what the user waits through — loading and joining the stored caches still costs something.

But attention is the step where every token gets to look back over every earlier token, and a cache computed for chunk 3 alone contains no trace of chunk 17. Concatenating the pots does not make them taste of each other: the cross-chunk attention was never computed, and pasting caches together cannot compute it after the fact. At short inputs this barely shows. At 124,000 tokens — dozens of chunks, most of the answer's evidence spread across them — it shows.

So the paper reaches for two knobs rather than one.

The first is to put a few pots back on the heat together. Selective KV recomputation re-runs a chosen subset of the cached chunks through the model and reuses the stored keys and values for the rest. The abstract says a subset of the caches is recomputed; it does not say how that subset is picked. The chunks you re-run regain attention to the context around them; the chunks you skip cost only what it takes to load and join their caches. The fraction you recompute is the dial between the two ends — recompute nothing and you have the fast, disjointed plate; recompute everything and you are back to full attention, paying the full prefill.

The second knob is stranger, and it is the one the title argues for. Rather than only patching the context at inference, the model is fine-tuned with concatenated caches already present, so it learns to work with the altered attention pattern instead of being surprised by it. The chef is trained on pre-cooked parts. Cache reuse itself is not new — the abstract points back at previous works that do it. What this paper proposes is the combination, because the two target different halves of the same damage: the fine-tuning adapts the model to what concatenation does, and the recomputation undoes some of it.

Retrieve chunkstop-k documents for this query

Load cached KVeach chunk was prefilled once, alone

Pick a subsetwhich chunks are worth re-running

Recompute + concatenatecross-chunk attention restored where chosen

Decodefirst token appears — TTFT stops here

Approach	Prefill work	Cross-chunk attention	Reported result
Full attention	every token, every request	complete	the TTFT reference point
Reuse every cached chunk	none	none between chunks	not reported in the paper's abstract
Recompute a subset only	a fraction of the chunks	restored for the chunks re-run	the paper's baseline
Concat-aware fine-tuning + selective recompute	the same fraction	the same, plus a model trained for it	+9.7 RULER vs recompute-only at 124K tokens; separately, 80% lower TTFT vs full attention, input length not stated (paper)

Hold the input at the paper's 124,000 tokens and say, illustratively, that it arrives as 20 retrieved chunks of 6,200 tokens each. Full attention pushes all 124,000 through prefill. Reuse every cached chunk and you push none of them — the fast, disjointed end. Recompute 4 of the 20 and you push 4 × 6,200 = 24,800 tokens back through the model, one fifth of the text. That fraction is the whole dial. The paper does not state which fraction it lands on; it reports the outcome: TTFT 80% below full attention, with the concat-aware fine-tuning worth a further 9.7 RULER points on top of recomputing alone. One caution on the arithmetic: the fraction of chunks you re-run is not the same number as the fraction of time you save, because a recomputed chunk still has to attend across everything that precedes it in the assembled prompt, which is far more context than it saw when it was cached alone — and the 80% is a measured outcome the abstract does not tie to a stated input length, not a dial setting you can read off the chunk count (illustrative chunk split; only the 124,000-token input, the 9.7 points and the 80% come from the paper).

The honest limit is that this is one paper, and its abstract reports one benchmark at one context length. RULER measures whether a model can use text far back in its window, which is precisely the ability concatenation damages, so it is the right benchmark — and also the one where the fix has the most room to look good. What travels regardless of the numbers is the shape of the tradeoff: KV reuse is not free quality, it is bought attention you did not compute, and you can buy some of it back by the chunk.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → When It Helps, When It Fails

Related explainers

FastTPS — Reloading-free KV-cache concatenation — the other kind of KV concatenation cost: the memory copy you pay to append a token, not the attention you lose by joining chunks.
Chunked prefill vs elastic KV reclamation — what a serving engine does about prefill cost when it cannot reuse anything at all.
MiniMax M3 — block-sparse attention — the other route to a cheap long context: keep computing attention, but over fewer blocks.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based