Let language models declare which KV-cache regions to attend — Model-declared attention scope vs external sparsity predictors — What does it mean?
The news. On September 2, 2026, a paper titled "Language Models Can Control Their Own Attention" introduced Declarative Attention (DA): a protocol that elicits the model to announce, inside its chain-of-thought, whether the next stretch of generation needs the full context, one specific region, or only its own recent output. The inference engine parses those announcements the way it parses a tool call, then skips most of the KV-cache read. Evaluated zero-shot across 15 long-context tasks on off-the-shelf Gemma-4-31B and Qwen-3.6-27B, DA cut total attended tokens during decoding by 52.0% and 31.1%, at accuracy costs of 1.27 and 2.75 percentage points. Read the paper →
Picture the reading room. You are writing an essay at a desk, and every single word you put down sends a clerk to walk the entire closed stacks — every aisle, every shelf — in case one page back there matters. That is what decoding does today: the model reads the whole KV cache to produce each new token, then uses a tiny fraction of what it read. The obvious fix is to make the clerk smarter, and that is what a prominent line of sparse-attention work does — teach the clerk to guess which aisles are worth walking. Declarative Attention does something else: it lets the person at the desk write the aisle number down. Concretely, DA is a prompting protocol: the model emits one of three declarations — <global>, <focus> or <local> — as plain text in its own reasoning, and the serving engine treats that declaration as an instruction about which part of the cache to read.
Nothing about the model changes: the results are zero-shot on off-the-shelf Gemma-4-31B and Qwen-3.6-27B, so the declaration is elicited from weights that were never trained for it. It goes inside the reasoning trace, and the serving engine intercepts it the way it intercepts a tool call: parse, act, continue. The useful way to picture what the engine does with it is masking, not scoring. A decoder already applies a causal mask so a token cannot see the future; a declaration acts like a second restriction on top of it, chosen by the model, that also hides most of the past. The paper describes the effect at that level — attention is limited to the full context, a named region, or recent output — rather than specifying how a server should lay the restriction over its cache blocks.
| Declaration | What the model is saying | What the engine reads |
|---|---|---|
<global> | I need the whole conversation | the full KV cache |
<focus> | I need one specific region of it | only that region |
<local> | I only need what I just wrote | recent output only |
The contrast with the prominent existing approach is the whole point of the paper. Proxy-score methods still have to look at every cached token in order to decide which ones to ignore — they rank first and prune second, so the per-step cost stays O(N) however aggressive the pruning is. DA never ranks. Its motivating question, in the authors' words, is whether the model would not already know which parts of the context are relevant — and the model is the one component that has been over the whole context, during prefill. The paper's wager is that asking it is therefore cheaper than measuring, and it reports that the engine then skips most of the KV-cache read.
It helps to put numbers on what that saves (illustrative). Say the context holds 200,000 tokens and the reply runs 400 tokens. Under ordinary decoding every one of those 400 tokens reads all 200,000 cached entries: 400 × 200,000 = 80 million attended tokens for a single reply. Now suppose the model declares <local> for the 300 tokens of connective prose, reading only its own last 1,000 tokens, and <focus> on a 20,000-token region for the 100 tokens that actually cite the source. The bill becomes 300 × 1,000 + 100 × 20,000 = 300,000 + 2,000,000 = 2.3 million attended tokens, under 3% of the original. Treat that as an upper bound on a favourable split, not a forecast: across 15 real long-context tasks the measured reductions were the more sober 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B.
None of this is free, and it is worth being precise about what does and does not shrink. Accuracy fell by 1.27 percentage points on Gemma-4-31B and 2.75 on Qwen-3.6-27B; the authors report that the gap narrows as models get larger, which is a claim about a trend rather than a guarantee at any given size. More importantly, what the paper measures is attended tokens — how much of the cache is read — not how much memory the cache occupies. The whole context still has to be kept, because a later <global> may need it, so the memory cost of a long conversation does not change. Whether fewer attended tokens turn into proportionally faster or cheaper serving depends on the engine and the hardware, and the paper does not report wall-clock or bandwidth numbers. The results are also zero-shot on unmodified weights: the authors present training-based versions as future work, not as something measured here.
Goes deeper in: LLM Internals → KV Cache → Prefill vs Decode
Related explainers
- ConSA — Controllable attention sparsity — the same goal, decided for the model instead of by it.
- FlashMemory — Lookahead Sparse Attention — a trained indexer that predicts which cache chunks a token will use.
- Compute Where It Counts — per-token compute controller — an external policy network choosing the per-token effort.