Compress reasoning KV caches 5.8× with beacon queries — Query-cluster KV residency prediction — What does it mean?

The news. On September 4, 2026, a paper titled BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference was posted to arXiv. It reports that across four open-source large reasoning models and a range of reasoning benchmarks, BeaconKV generally outperforms existing compression methods, reaching up to 5.8× memory reduction while nearly preserving full-cache accuracy and improving throughput by over 4.3×. Read the paper →

Picture the reasoning trace as a coastline. The model sails left to right, one token at a time, and every stretch it passes is a piece of KV cache it might need to look back at. Keeping the whole coast lit is the ideal and it is exactly what you cannot afford: the cache grows linearly with sequence length, and a long chain of thought keeps extending it. So you buy a few lamps and choose where to point them.

The obvious place to point them is behind you. Every widely used compression policy is some version of that rule — a sliding window over the last N tokens, or an importance score accumulated from the queries you just issued. Both are betting the same bet, and the paper names it: existing methods "rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns." The paper's contribution starts by demonstrating that this assumption fails in long-horizon reasoning.

The failure has a name. Certain decoding steps produce what the authors call Thought Revisiting Tokens — tokens whose queries "re-attend to distant previous context," such as the task-solving plan the model wrote at the very start. That is the ship turning around and sailing back toward the early harbor. A recency lamp is pointed the wrong way when it happens, and the early plan was evicted long before. Nothing errors when this happens: the evicted plan is gone from the cache, and short of recomputing it there is nothing left for attention to reach.

Try the eviction strategies above and the shape of the problem shows up immediately. "Drop oldest" and "sliding window" throw away the beginning by construction. "Drop least-attended" looks smarter, but its evidence is the attention scores it has already computed — which is to say the past. None of the three can keep a token because a query that has not been issued yet is going to want it.

The obvious fix is to keep the whole query history and match against it, and the obvious reason nobody does that is cost: the history grows just as fast as the cache you are trying to shrink. BeaconKV's insight is that you do not need the whole history, because the revisiting queries are not all different from each other — they cluster into a small number of similarity groups in embedding space — the same coordinate space where similar meanings sit near each other. A plan-checking query early in the trace looks a lot like a plan-checking query much later, because they are asking the same kind of question.

So the method stores one compact representative per cluster — a beacon query — and uses that short list to anticipate which KV pairs will be revisited, "without storing the entire query history". Those are the two lighthouses in the illustration, planted on the routes ships actually come back to rather than on the stretch of water most recently sailed. Residency stops being a question about the past and becomes a prediction about the next query. And because a beacon is just a stored vector, none of this requires fine-tuning: the method is training-free and drops onto an existing checkpoint.

Put numbers on the coastline to see what a multiplier buys. Hold the model shape fixed at 32 layers, 8 KV heads, head size 128, fp16 — a plausible grouped-query reasoning model, and illustrative, since the paper reports ratios rather than per-model byte counts. One token costs 2 (K and V) × 8 heads × 128 × 2 bytes = 4 KiB per layer, and × 32 layers = 128 KiB per token. Now let the model think for 60,000 tokens, a length a long chain of thought can reach: 60,000 × 128 KiB = about 7.3 GiB of KV cache for a single sequence. At the paper's best-case 5.8× that becomes about 1.3 GiB, freeing roughly 6 GiB. The paper reports its memory and throughput gains together — over 4.3× — without spelling out the link between them; the standard reason the two travel together is that decode is memory-bandwidth-bound, so a smaller resident cache leaves room for a bigger batch, and a bigger batch means more tokens per second from the same GPU.

The assumption to keep in view is in the last column of the table. Beacons rest on a bet that the clusters found so far describe the queries still to come — the paper's own framing is representatives for global query clusters — so a workload whose query distribution shifts partway through has no reason to obey it. Whether that actually bites is not something the abstract settles: it reports no failure analysis, no per-model results and no benchmark breakdown, and its headline figures are best cases from its own evaluation rather than a guarantee for arbitrary traffic. What survives the specific numbers is the reframing: the question a cache policy should answer is not "what did this trace look at" but "what is it about to come back to," and those turn out to be different questions. That reframing is why this is worth reading even if BeaconKV itself is superseded next quarter, and it is the question to carry into the eviction machinery a production serving engine already runs.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

• Entropy-aware KV-cache compression (InfoKV) — the token-side counterpart: scores what to keep from the cached token itself rather than from a predicted future query.
• Per-head KV cache budgets (Tangram) — decides how much each head may keep, the question beacons leave open.
• Hadamard-rotated 2-bit KV cache (KVarN) — the other way to shrink a cache: keep every token, store it in fewer bits.