Who Should Score the Cache paper — Mid-depth layer scoring — What does it mean?

The news. On November 1, 2026, a Colby College preprint evaluated 21 models and found that KV-cache eviction accuracy depends on the depth of the layer that does the scoring. On the 18 models with per-layer caches (0.5B to 109B parameters), accuracy collapses at shallow layers, peaks at 0.4 to 0.75 of model depth, and often dips again at the last layer. Read the paper →

Picture an assembly line of desks that all read the same 32,000-page case file, and a binder with room for only a few hundred pages. Each desk first reads a sticky note with the question, then puts tabs on the pages it thinks matter. When the binder is full, eviction keeps the top-k tabbed pages and throws away the rest. In a real model the desks are layers, the tabs are attention scores, and the binder is the KV cache you can afford to keep in GPU memory.

The usual rule hands each desk its own binder: layer 7 judges what layer 7 keeps. The paper points out that no eviction method it knows of asks the obvious follow-up, which is whether layer 7 is a good judge. The score for a token is the mean softmax attention that the question's tokens place on it, and the first 4 tokens and last 32 tokens are always kept.

The answer is that depth decides. A layer judges tokens well only once it has used the question to find the answer, and that takes several layers. In the picture, desks 1 and 2 have read the sticky note but have not yet matched it to the right pages, so their tabs land almost anywhere. Desks 3 and 4 have found the evidence and tab it heavily. The last desk has already moved on to writing the reply, so its tabs drift again.

The paper tests this directly. When it scores with the context's own final tokens instead of the question, the attention on the answer flattens across depth. With the question, attention on the answer's tokens peaks at mid-depth, within one grid step of the accuracy peak, on every model tested. The cut is also all-or-nothing: an answer survives only if every one of its tokens is kept, so moderate ranking errors add up to a lost answer.

To choose the judge desk without trying every layer on real traffic, the paper computes answer survival probability for each layer. For each calibration prompt it rebuilds the top-k set that layer would keep, checks whether the gold answer lies entirely inside, and averages. It uses 25 prompts per task from rows the evaluation never touches, and it picks the layer with the highest value (the shallower one when the top two are within one standard error). That pick is the best measured layer on 14 of 18 per-layer models and never significantly below it on the other four.

One layer then scores every cache and one shared set of tokens is kept for all heads. Together these two changes beat per-head SnapKV by 18 to 43 points on five models up to 14B at 5 to 15% budgets, and scoring reads one layer's attention at prefill, adding under 1% to its time. Averaging rank over all layers is worse: it loses 21 to 37 points to the selected layer on two models.

Here is what the cut means in numbers. Take Llama-3.1-8B at 32K context, where the paper reports a full decode-resident cache of about 4.0 GB (rounded from 3.96 GB). A 5% budget keeps 5% of 32,768 tokens, which is 1,638 tokens. Of those, 36 are fixed (4 sink plus 32 recent), leaving 1,602 slots for scored tokens, and the cache shrinks to roughly 0.2 GB (illustrative: 4.0 GB times 5%; the paper measures 45 MB at a 1% budget). If the answer is a 6-token span (illustrative), all six tokens must land inside those 1,602 slots out of about 32,000. The judge desk decides whether they do: on Qwen2.5-14B at a 5% budget, the mid-depth layer L30 gained 17.7 points over scoring from each layer's own cache, and the worst layer lost 50.7 points.

The same curve explains a failure in models that share a cache across layers. There, one cache serves several reader layers, and by default the layer that wrote it scores for everyone. When that writer is shallow, the scores come from the blind part of the line. On Gemma-4-E2B the shallow writer loses 28 points to a mid-depth reader at a 10% budget and needs roughly twice the budget to match. This rests on one sibling pair of models, so the writer's depth is the likely cause, not a proven one.

The gain also has limits. From 32B up the paper detects no gain over the default, which is already close to the best mid-depth layer, though the worst layer still loses 25 to 56 points at every scale. In open-book question answering (LongBench) no mid-depth layer beats the default. The pick needs calibration prompts with known answers, every scorer here sees the question, and the 70B and 72B runs used 4-bit checkpoints. The architectural specifics of deployed engines are not covered.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

• MoEZip routing-aware KV eviction — asks a different eviction question: which tokens an MoE router depends on.
• Hy4 IndexCache cross-layer index reuse — another case of one layer's token selection being reused by other layers.