What is KV policy co-adaptation?

It is finetuning a language model while the sparse KV selection policy it will be served under is already applied, so the weights update under the same information they will have at inference. In the usual setup the policy is a deployment-time decision: the model trains with the full KV cache available and only meets eviction afterwards. Co-adaptation moves that decision earlier, making the dropped keys part of the training distribution rather than a change the model encounters for the first time in production.

Why does it matter that the model trains with the full cache?

Because a common way to make long-context serving affordable is to throw most of the KV cache away, and the model doing the work was trained with none of it thrown away. Attention learned during dense training routes each query through particular past keys; if the eviction policy removes those keys at serving time, the model is running on habits built for information it no longer has. Training under the policy is a way of making the weights lean on keys that will still be there.

How does this relate to KV cache compression like quantization or GQA?

Those methods reduce how many bytes each cache entry costs — fewer bits per number, or fewer key-value heads — and some of them, such as quantization-aware training or a grouped-query architecture, already reach into training. What none of them changes is what the weights learn about the selection policy they will be served under, and that is the gap this paper closes. The two lines of work are described as orthogonal and combinable rather than competing: you can quantize a cache and still have the train-serve mismatch this paper is aimed at.

KeysAndValues finetunes models under the KV policy they will serve with — KV policy co-adaptation

TL;DR

What is it: The news anchor is an arXiv paper releasing KeysAndValues, a library for long-context finetuning; the concept it makes concrete is KV policy co-adaptation — finetuning a model while the sparse KV selection policy it will be served under is already switched on, so the weights adapt to the keys that will actually survive.
Why it’s needed: One common way to make long-context serving affordable is to throw away most of the KV cache, but the model that does the throwing was trained with all of it present — so the policy is applied to weights that were never told it was coming.
vs previous: The established approach treats sparsity as a serving-only compression problem — train densely, then drop old or low-attention keys at inference; this paper reframes it as a joint model-and-cache-policy problem and trains under the policy, so the missing keys are part of the training distribution rather than a surprise at deployment.

Think of it as

studying with the whole shelf open, then sitting the exam with one index card

the open shelf during study = the full KV cache during training, where every past token is still there to attend to
the single index card in the exam = the sparse KV budget at serving time, where most of those tokens have been dropped
the rule for what earns a place on the card = the KV selection policy, such as keeping the tokens the model attended to most
studying from the shelf and then losing it = training densely and then bolting on eviction afterwards, so the weights lean on keys that will not be there
practising from the card from day one = finetuning with the eviction policy already applied, so the weights adapt to what survives
the student learning which facts to lean on = the model learning to route its attention through the keys the policy keeps

Jargon

KV cache: The keys and values every past token produced, stored so the model does not recompute them on each new token. It is what makes generation fast and what makes long context expensive. Walk the KV cache →
KV selection policy: The rule that decides which entries stay in the cache and which get dropped once the budget is full — oldest-first, a sliding window, or lowest-attention. Also called an eviction policy, and normally applied only at serving time.
Sparse attention: Attention where each query is scored against a subset of the past keys instead of all of them. Dropping keys from the cache is one way to get it: a key that is gone cannot be attended to.
H2O (Heavy-Hitter Oracle): A published KV policy that keeps the “heavy hitter” tokens — the ones accumulating the most attention so far — and evicts the rest. The paper reports it as the strongest policy in its experiments.
Train/serve mismatch: Any gap between the conditions a model was trained under and the conditions it runs under. Here the gap is information: the weights learned with every past key available and are then run with most of them missing.
SDPA kernel: Scaled dot product attention — the fused kernel that computes attention without materializing the full score matrix. A policy like H2O ranks tokens by how much attention they have accumulated, so it needs a kernel that hands those summed attention weights back — which a fused kernel normally discards. See why attention gets fused →
Long-context finetuning: Continuing to train an already-pretrained model on long sequences, so it handles them well at inference — the starting checkpoint may already be long-context. It is the stage this paper modifies.

The news. On August 20, 2026, an arXiv paper argued that long-context inference should be treated as a joint model-and-cache-policy problem rather than a serving-only compression problem. Its method finetunes a transformer while applying the target sparse-attention KV selection policy, so the weights adapt to the information that will remain available at inference. The authors report the method runs on a single NVIDIA A100 with 40 GB, name H2O as the leading policy in their experiments, and release the KeysAndValues library for long-context inference and finetuning. Read the paper →

Picture a student who spends every night of term at a desk with the whole shelf open — eight textbooks, every page reachable, nothing they might need more than an arm's length away. They get very good at working that way. Then they walk into the exam and the shelf is gone; all they are allowed is one index card. The problem is not that the card is small. It is that every study habit they built assumed the shelf, so they never learned which facts had to be on the card in the first place.

That student is a long-context language model, and the shelf is its KV cache. During training, every past token's keys and values are there to be attended to. During serving they are not, because keeping them all is what makes long context expensive — the cache grows linearly with the sequence, and past some length it is the thing that decides how many requests fit on a GPU. So production systems evict: drop the oldest entries, keep a sliding window, or keep whichever tokens have drawn the most attention so far and discard the rest.

Remove earliest tokens first

The mismatch is the whole point: the policy that decides what to forget is applied to weights that were trained with nothing forgotten. Every one of those eviction rules is a decision made after training, on a model that has no idea it is coming. The student walks in expecting the shelf. Whatever the eviction rule gets wrong, the model has no learned habit to fall back on, because the habit it does have — route this query's attention through those particular keys — assumes keys that are no longer in the cache.

The paper's move is to hand the student the index card on the first night of term. It finetunes the model with the target KV selection policy already switched on, so the weights are updated under exactly the conditions they will be served under. When the policy drops a key during training, the gradient sees a model that has to get the answer right without it, and the weights adjust to lean on the keys the policy keeps. The card and the studying are chosen together, which is where the name co-adaptation comes from — the same reason the practice matters more than the card's size.

Hold the budget fixed and it is easier to see what is being repaired. Suppose you can afford a cache of 4,000 tokens while the context runs to 32,000 — a policy that keeps one key in eight. Train densely, and the weights spent every step of finetuning attending across all 32,000; at serving time 28,000 of them — 7 of every 8 past keys — are simply gone from the cache it attends over. A policy like H2O does not drop them at random; it keeps the tokens that have drawn the most attention so far. But nothing during that dense finetuning told the weights a selection was coming, or which 4,000 it would leave. Now finetune under the same policy: the model is trained against 4,000 and served against 4,000. The budget did not change and the cache did not get bigger — the training distribution just stopped disagreeing with the serving one (illustrative: the paper does not publish this breakdown, and real budgets vary by model and workload).

	Serving-only compression	Policy-aware finetuning
When the KV policy is chosen	after training, at deployment	before finetuning, as part of it
What the weights know about it	nothing	they were updated with it applied
Which keys are available while the weights update	every past key	only the ones the policy keeps
Reported hardware to train this way	~a single 40 GB NVIDIA A100 (setup-dependent, as reported by the authors)

What makes this an attention result rather than a serving one is where it moves the decision. Compression work asks how few bytes can carry this cache — quantize the entries, share them across heads, score each one and evict the losers. Some of those are architectural or training-aware in their own right, but none of them changes what the weights learn about the selection they will be served under. This paper puts that selection into the finetuning instead, changing the lesson: from "drop old KV tokens after training" to "train the model under the same forgetting policy it will serve with." The paper is 39 pages with no figures and reports H2O as its strongest policy; the released library is the part most people will touch first.

Goes deeper in: LLM Internals → KV Cache → KV Cache in Production

Related explainers

ConSA — Controllable attention sparsity — the closest neighbour, and a useful contrast: ConSA learns which pattern each head should use, while this paper keeps the policy fixed and adapts the weights to it
SP-KV — Utility predictor for the KV cache — a serving-side answer to the same budget question: predict which entries are worth keeping
TokenStack — Processing-in-memory attention — a third answer that changes neither the weights nor the policy, only where the attention math runs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based