Translate KV states across model families to cut prefill 67% — Cross-model KV translation — What does it mean?
The news. On August 31, 2026, a paper (arXiv:2608.30963) proposed translating a source model's KV state into a representation a different target model can consume, across model families, tokenizers and attention configurations. The reported results: a Qwen2.5-1.5B to Gemma-2-2B handoff cuts target prefill cost by up to 67.05% at 4K context, and a Llama3.1-70B to Qwen2.5-7B handoff cuts measured latency from 899 ms to 138 ms while scoring 44.0% against 45.7% for native inference. Read the paper →
Picture a court stenographer. While the trial runs she fills a notebook with shorthand — dense, fast, and written in a system that is entirely her own. In an LLM that notebook is the KV cache: the keys and values the model computed while reading your prompt, so it never has to read those tokens again. Now a colleague takes over the transcript. He opens her notebook and it is unreadable. His shorthand uses different symbols, different spacing, a different rhythm. So he does the only thing he can, and starts the recording again from the beginning to take the whole thing down himself. A KV cache is written in one model's private coordinate system, so no other model can read it, which is why handing a conversation from one model to another today means re-reading the entire prompt from scratch. The paper's proposal is to stop making him do that: a learned layer that rewrites the first model's notes into the second model's shorthand, which the authors call a context-reuse layer.
That re-reading is prefill, and it is not a rounding error: prefill reads every token of the prompt in one pass and its cost climbs with prompt length, so a long conversation handed to a second model pays for that whole conversation twice. Existing prefix caching cannot help here, because it is built on exactly the assumption that breaks: it reuses a cache when the same model meets the same prefix again. Switch models and every cached block is dead weight. The mismatch is not cosmetic either. Two families disagree on hidden size, on layer count, on how many query heads share a key/value head, and even on how the text was split into tokens in the first place.
The paper's answer is to hire the transcriber. The context-reuse layer is a learned map from the producer model's KV space into the consumer model's KV space, so the second model resumes from translated state instead of recomputing the prompt. She does not re-attend the trial. She reads the first stenographer's notebook and rewrites it, page by page, into her colleague's system. The authors describe one universal layer that absorbs the differences in scale, architecture, attention configuration, tokenizer and family, and they name the abstraction it creates context mobility: context that travels, rather than context pinned to whichever model happened to produce it. Their framing is that "KV states can serve as transferable computational representations rather than strictly model-local caches".
Where it earns its keep
The clearest reported case is a big-to-small handoff: Llama3.1-70B has read the conversation, and Qwen2.5-7B is asked to continue it. Hold that single handoff fixed and price it two ways. Done the ordinary way, the 7B model prefills the entire conversation itself, and the paper measures 899 ms of latency. Done with translation, the 7B model consumes translated KV instead, and the same handoff measures 138 ms — a saving of 761 ms on one handoff. An agent that bounces between a strong planner and a cheap executor twenty times in a session would, on those figures, save roughly 20 × 761 ms, about 15 seconds of measured handoff latency (illustrative: the paper measures one handoff, not a multi-step session).
The translation is not free: that cross-family handoff scores 44.0% where native Qwen2.5-7B inference scores 45.7%, a 1.7-point gap standing alongside the latency it saves. The retyped notes are close to the original, not identical, which is the honest price of a learned mapping between two different representation spaces and the number a serving team would weigh against the speed it buys. The trade does not always run that way, though. On LongBench2, a Qwen2.5-7B to Qwen2.5-1.5B handoff moved the small model from 27.59% to 34.48% — the paper reports that result without explaining what produced it.
| Handoff | What was measured | Reported result | Source |
|---|---|---|---|
| Qwen2.5-1.5B → Gemma-2-2B | Target prefill cost at 4K context | Up to 67.05% lower | paper |
| Llama3.1-70B → Qwen2.5-7B | Measured latency | 899 ms → 138 ms | paper |
| Llama3.1-70B → Qwen2.5-7B | Accuracy vs native Qwen2.5-7B | 44.0% vs 45.7% | paper |
| Qwen2.5-7B → Qwen2.5-1.5B | LongBench2 score | 27.59% → 34.48% | paper |
If KV states are portable, the cache stops being a per-model scratchpad and becomes a piece of context a router can move between models. That is a different shape of serving system from the one most stacks assume. Today a KV cache transfer moves a cache between machines running the same model; a portable cache would let a router move it between models, which is what a cost-tiered agent stack wants and cannot currently have. The paper reports the mapping and the measurements above. It does not claim the layer is free to train or that it holds at every scale, and the training cost and the limits of the mapping are the parts a serving team would want to see before betting a stack on it.
KV cache size (Llama 3.1 70B)
Transfer time (0.67 GB KV cache)
NVLink is ~28× faster than PCIe — same-node transfers are near-free.
Goes deeper in: LLM Internals → KV Cache → The KV Cache Solution
Related explainers
- ReCache — Composition-invariant KV blocks — reusing cached blocks in a new composition, for one model rather than across two.
- CacheWeaver — Prefix-cache evidence reordering — squeezing more reuse out of same-model prefix caching.
- Latent Context LMs — Encoder-decoder compression — the other way to rewrite a context into a cheaper representation.