Why does training against the full-cache model help?

The language-model loss mixes the error caused by eviction with the frozen model's own mistakes, and the gate can only fix the first. Matching the full-cache model's output distribution with a KL loss removes the second term, so the gradient vanishes exactly when eviction stops changing the output.

How much does IndexMem++ compress the KV cache?

The paper reports **75% KV cache compression** on Qwen3-8B with 90.14 on RULER at 8K and 46.03 on LongBench, which it puts 3.72 and 1.36 points below full-cache inference.

How does this relate to training-free eviction like SnapKV?

SnapKV ranks keys by the attention they received from recently seen queries, with no training. IndexMem++ learns a query-independent score per key, and the paper reports that it holds up better on retrieval tasks with many look-alike distractors, where similarity-based rules degrade.

LLM· 2026-11-09

The Gating Mechanism of KV Cache Eviction — The constant-gate no-op — What does it mean?

Q: What is the constant-gate no-op in KV cache eviction?

A learned eviction gate multiplies each cached key's attention weight before softmax re-normalizes. If the gate has the same value for every key, softmax cancels it and the model behaves exactly like the full-cache model. The paper finds that ordinary language-model loss drives the gate toward near-constant scores, which rank entries poorly.

TL;DR

What is it: The Gating Mechanism of KV Cache Eviction (arXiv 2611.07605, the paper behind IndexMem++) explains why a learned KV eviction gate placed inside softmax tends to learn nothing, and how to train one that ranks cache entries usefully.
Why it’s needed: A long-context model's KV cache grows with every token and becomes the main memory cost at decode time, so a gate that reliably picks which cached keys to throw away lets a server fit far longer contexts on the same GPU.
vs previous: The direct recipe trains the gate on the frozen model's ordinary language-model loss, but softmax cancels any gate that is the same for every key, so that loss is happiest with a constant gate that changes nothing; the paper pins some entries, fixes the shared budget, and trains against the dense model's own predictions instead.

Jargon

KV cache: The stored key and value vectors of every earlier token, kept so each new token does not recompute them. It grows linearly with context length. See the memory cost.
KV eviction: Deleting some cached entries to stay under a memory budget. Later tokens can no longer attend to what was evicted.
Softmax normalizer: The sum in the denominator of softmax. Dividing by it makes every query's attention weights add up to exactly 1.
Logit: A raw score before softmax. In attention it is the scaled dot product of a query and a key.
Eviction gate: A learned number per cached key, between 0 and 1, that multiplies that key's attention weight during training. A gate near 0 means evict this key.
Attention sink: The first few tokens of a sequence, which many heads attend to heavily regardless of content. Eviction policies almost always keep them.
Distillation (KL): Training a model to match another model's full output distribution, measured with KL divergence, instead of matching only the single correct next token.
RULER: A long-context benchmark of synthetic retrieval and tracing tasks, such as finding a needle fact hidden in thousands of filler tokens.

The news. On November 5, 2026, a team from HKUST, NYU, Tencent HunYuan and others posted The Gating Mechanism of KV Cache Eviction on arXiv. It reports that the most direct learned-eviction design collapses to near-constant scores, diagnoses why, and builds IndexMem++: a gate with pinned entries and a fixed budget, trained by distillation from the full-cache model, plus centroid slots that recover part of what eviction discards. At 75% KV cache compression on Qwen3-8B it scores 90.14 on RULER at 8K and 46.03 on LongBench, which the authors put 3.72 and 1.36 points below full-cache inference. Read the paper →

Picture an election night graphic: a pie chart where each slice is one candidate's share of the vote. Now double every ballot. Every count changes, but when the chart is re-drawn as percentages, every slice is exactly the same size. Attention works the same way. Each cached key gets a weight from softmax, and softmax divides by the sum of all the exponentiated scores, so the weights always add up to 100%.

A learned eviction gate multiplies one key's weight by a number g before that re-draw. Because exp(score) × g = exp(score + log g), a gate inside softmax is just an extra bias added to that key's logit, and setting g → 0 is exactly hard eviction. The placement is deliberate: when one key is suppressed, the re-draw hands its share to the keys that remain, which is what real eviction does.

The trap is the same multiplier on every ballot: if the gate gives every key the same value, softmax cancels it and the model behaves exactly like the original. The backbone is frozen and was trained for full attention, so on the ordinary language-model loss the best gate is a tiny nudge away from that constant. The paper puts it sharply: the no-op is what the objective asks for. A near-constant gate is useless for eviction, because the cut between kept and evicted keys falls among near-ties and the retained set is effectively arbitrary.

Two fixes: take away the free pie, then change the teacher

The first fix removes the escape route. Every eviction policy already keeps the attention sinks and a local window of recent tokens, so the paper pins their gates at exactly 1. Those slices become a fixed reference, so the level of the other gates is no longer free. The remaining evictable keys must then share a fixed total gate budget β that is far smaller than their count, so the gate cannot reproduce full attention and has to rank. Raising one evictable key's slice now shrinks another's, the same way a fixed share of the pie must be split among the remaining candidates.

The second fix changes what the gate is graded against. Under the plain language-model loss, the gradient on the output mixes two errors: the gap between the gated model and the full-cache model, and the gap between the full-cache model and the correct token. The gate can only shrink the first one; the second is the frozen model's own mistake, and it can dominate on hard tokens. Training the gate to match the full-cache model's output distribution removes the frozen model's own error from the signal, so the gradient goes to zero exactly when eviction stops changing the output. This also gives a full distribution per position instead of one target token. See how logits become probabilities for the distribution being matched.

Where the score actually comes from

Hold three cached keys fixed with attention logits 2, 1 and 0. Softmax gives e², e¹, e⁰ = 7.39, 2.72, 1.00, a sum of 11.11, so the weights are 66.5%, 24.5% and 9.0%. Apply a gate of 0.5 to all three: the numerators become 3.69, 1.36 and 0.50, the sum becomes 5.56, and the weights are again 66.5%, 24.5% and 9.0% (illustrative). Nothing moved. Now gate only the third key at 0.1: the numerators are 7.39, 2.72 and 0.10, the sum is 10.21, and the weights become 72.4%, 26.6% and 1.0%. Only a gate that differs between keys does any work.

The paper's ablation on Qwen3-0.6B (RULER at 8K, 2,048 of 8,192 entries kept, which is 75% compression) shows each fix composing. The direct design scores 4.33 against the full-cache model's 70.01; pinning sinks and window with a fixed budget lifts it to 44.86, distillation to 58.86, and a per-head budget to 67.60. Most of the gap closes from changing what the gate is allowed to do and what it is graded against, not from a bigger scorer. The authors report that a larger scorer, explicit access to past tokens, and a trainable backbone did not help once these two conditions held.

Configuration (Qwen3-0.6B, RULER 8K, 75% compression)	RULER avg.	Change	Source
Direct gate, LM loss, uniform per-head budget	4.33	—	Table 1
Add pinned sinks and window, fixed budget β	44.86	+40.53	Table 1
Add distillation from the full-cache model (KL)	58.86	+14.00	Table 1
Add per-head budget from retained attention mass	67.60	+8.74	Table 1
Full KV cache, no eviction	70.01	reference	Table 1

Recovering what eviction throws away

Even a good ranking loses information: the paper measures roughly 20% of the attention mass landing on evicted entries. A side memory read after softmax struggles to return that content, because it does not know what share of the pie the evicted keys held. IndexMem++ clusters the evicted keys and values into a few averaged centroid entries and reads them inside the same softmax, so the re-draw itself decides how much of the pie they get back. With 64 centroid slots counted inside the same 512-entry budget, Qwen3-0.6B on the Math500 math benchmark rises from 48.00 to 52.00. For why the budget matters at all, see what the KV cache costs and how a serving engine's memory manager hands out that space.

Goes deeper in: LLM Internals → Self-Attention → Computing Attention Scores

Related explainers

DropKV: residual-output KV eviction — a training-free eviction rule that scores keys by their effect on the output instead of by attention weight
Who should score the cache: mid-depth KV eviction — which layer is in a position to judge what a later query will need
Softmax collapse and the precision cliff — another case where softmax's normalizer cancels a signal you expected to matter

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based