What is On-Demand Attention?

A local-first decoding method. A small trained **recall head** reads the model's current decoding state and predicts whether reading the full KV cache will help the next token; the step runs against a local window unless the head says otherwise. Only the head is trained — the pretrained weights and the entire KV cache are left untouched.

Why does deciding when to recall matter more than deciding how often?

Because the same call budget spent at the wrong moments is worth almost nothing. On five RULER16K tasks ODA scored **88.96 at a 41.33% call rate**, while a random baseline calling at **40.99%** scored **32.79**. The frequency is nearly identical; the placement is what recovers full-attention quality.

How much faster is it in practice?

At 128K input, 1K output and a 12.5% global-call rate on one A100, the authors measure **24.26% of full attention's decoding FLOPs** and median single-request throughput rising from **75.54 to 149.52 tokens/s (1.98×)** in vLLM. At a 4K context it is **15.8% slower** than full attention — the overhead only pays off once the global reads are long.

How is this different from sparse attention like MSA or DSA?

Sparse attention decides **which** parts of the cached past to read on a step that is happening anyway. On-Demand Attention decides **whether** that global read happens at all on this step. The two are on different axes and are not mutually exclusive.

LLM· 2026-09-19

Let models trigger full attention only when recall helps — On-Demand Attention recall-head gating — What does it mean?

TL;DR

What is it: The On-Demand Attention (ODA) paper (arXiv 2609.20734) introduces recall-head gating — a small trained head that decides, at each decode step and before the read happens, whether reading the full KV cache will actually help the next token.
Why it’s needed: Full-attention decoding reads the entire history at every single step regardless of whether it helps, which is what makes long-context decode bandwidth-bound rather than compute-bound; at a 128K context and a 12.5% global-call rate the authors measure 24.26% of full attention's decoding FLOPs and 1.98× the single-request throughput in vLLM.
vs previous: Where sparse-attention schemes such as MSA or DSA answer which blocks of the cache to read, and ConSA learns a static full-versus-window allocation once at training time, ODA answers whether to read the global cache on this step at all — decided at runtime, with the pretrained weights frozen and the complete KV cache still retained.

Jargon

KV cache: The stored Keys and Values for every token already processed, so the model never recomputes them. It grows with every token and, at long context, dominates decode-time memory traffic — full explainer.
Decode step: One pass through the model to produce one output token. Distinct from prefill, which processes the whole prompt in a single batched pass.
Local attention: Attention restricted to a fixed recent window rather than the whole history, called the Local baseline in the paper. Cheap and constant-cost per step, but it cannot retrieve anything older than the window.
Global attention: Ordinary full attention over the entire history, which is what every decode step normally does. The paper calls this Full.
Recall head: The small trained module ODA adds, which reads the model's current decoding state and predicts the benefit of doing the global read on this step. On Qwen3-1.7B it is 28,325,889 parameters, roughly 1.7% of the backbone.
RULER16K: A long-context benchmark suite run at a 16,000-token context, built from synthetic retrieval and tracking tasks. Used here because the gap between local-only and full attention is wide and easy to measure.
FLOPs: Floating-point operations — a count of arithmetic work. A useful proxy for cost, but not the same thing as elapsed time, because memory traffic and kernel-launch overhead are not arithmetic.
CUDA Graphs: A CUDA feature that captures a fixed sequence of kernel launches once and replays it, removing per-launch CPU overhead. It assumes that sequence does not change between steps — full explainer.

The news. On September 17, 2026, Haibo Feng, Ruiqi Liang, Hanyang Peng and Shiqi Yu published On-Demand Attention: Language Models Know When to Recall. The finding underneath it is that a pretrained model's own decoding state already carries information about whether reading the full history will help the next token — before that read happens. They train a small recall head on that signal, leave the backbone frozen, keep the entire KV cache available, and implement GPU-side conditional execution in vLLM. Experiments cover Qwen and Gemma models, including hybrid-attention backbones. Read the paper →

Picture the writer at the desk. The recent tokens are the stack at their elbow — reachable without standing up. Everything older is in the archive room down the hall, filed and intact. A model doing ordinary full-attention decoding takes that walk for every single word it writes: it reads the whole KV cache at every step, whether the next word needs a name from page one or is just the word the.

That is what the paper's title means by knowing when to recall. Many decode steps plainly do not need the archive — the word after the is usually settled by the sentence you are in, not by something 90,000 tokens back. But standard attention has no way to say so in advance. Computing attention scores compares the current query against every prior key by construction, so the cost of looking back is paid before anyone finds out whether looking back mattered.

On-Demand Attention puts a gauge by the door. A lightweight recall head reads the model's current decoding state and predicts the benefit of the global read before that read occurs. When the predicted benefit is low the step runs local-first, against the window at the desk; when it rises, the step invokes global attention and the whole archive is still there. Nothing is compressed, evicted or summarised — ODA trains only the head, and the pretrained weights and the complete KV cache are both left alone. On Qwen3-1.7B the head is 28,325,889 parameters, about 1.7% of the backbone.

The result that carries the paper is not the savings — it is the ablation that shows the timing is doing the work. On five RULER16K tasks the authors ran a Random baseline that calls global attention at the same rate ODA chose, just at randomly chosen moments. ODA scored 88.96 at a 41.33% global-call rate; Random, at a near-identical 40.99%, scored 32.79 — against a freshly measured full-attention baseline of 88.63. Same budget, scattered timing, and long-context quality collapses. The gap holds across all eight call budgets they tested. Choosing when to look back is the mechanism; choosing how often is not.

Decode mode	RULER16K score	Global-attention calls	Source
Local only	19.23	0%	arXiv 2609.20734
On-Demand Attention	81.17	41.6%	arXiv 2609.20734
Full attention	81.94	100%	arXiv 2609.20734

All three rows are Qwen3-1.7B on the same benchmark. The Full-to-Local gap is 62.71 points, and the head recovers 98.8% of it while skipping the archive on nearly six steps in ten. On LongBench v1 the same head reaches 36.82 against Full's 37.94 — 92.2% of that gap — but chooses a much higher 70.6% call rate, which is the point: the call rate is not a knob someone set, it is what the head decided each task needed. Recovery is not uniform across tasks; English passage retrieval still trails full attention by 10.33 points.

Why does skipping the read buy wall-clock time at all? Because at long context and small batch a decode step is not waiting on math — it is waiting on memory. Every step streams the KV cache out of HBM, and the arithmetic performed per byte fetched is tiny. Batching hides this for the model weights, whose single read is shared across every request in the batch — but each request carries its own KV cache, so that traffic scales with the batch rather than amortising away. A single long-context request gets no help at all. ODA's throughput measurement is taken at batch size one — the leftmost regime in this picture.

Now hold the operating point fixed: 128K input tokens, 1K output tokens, one A100-SXM4-80GB, batch size one, CUDA Graphs enabled on both sides, and a 12.5% global-call rate. Naively, calling global attention on one step in eight should make attention eight times cheaper. It does not, and the paper is careful about why. Every recalled step still pays for the local attempt that preceded it, plus the recall head, plus the projection from the final hidden state onto the vocabulary — so the measured total is 24.26% of full attention's decoding FLOPs, a 4.12× reduction rather than 8×. Then FLOPs are not seconds: the feed-forward blocks, the projections and the kernel-launch overhead do not shrink at all, so measured in vLLM the median single-request throughput rises from 75.54 to 149.52 tokens/s — 1.98×. 8× on paper, 4.12× in FLOPs, 1.98× on the clock. The authors make the same point analytically: a fixed positive recall rate still leaves O(n) expected attention cost per step, so the speedup approaches 1/ρ — the inverse of the recall rate ρ, so 8× at this operating point — only when global attention dominates every other term in the step.

That clause has a sharp edge, and the paper reports it rather than burying it. At a 4K context the same machinery runs 15.8% slower than plain full attention — the local attempt and the conditional branch cost more than the short global reads they avoid. Conditional execution also works against the thing that makes decode fast in the first place: CUDA Graphs replay a fixed sequence of kernel launches, and a data-dependent branch is precisely what a captured graph is not built for. ODA's win is a long-context win with a crossover point, and that crossover is a number you would have to measure on your own stack rather than inherit from the paper. The authors are also explicit about what they have not measured: joint task quality and speed for the same learned policy, prefill-inclusive latency, peak memory, and multi-request batch scaling are all still open.

Goes deeper in: LLM Internals → Attention → Computing Attention Scores

Related explainers

All three of these answer which cached information to read; ODA answers whether to read it at all on this step.

ConSA: controllable attention sparsity — the static counterpart, which learns which units get full attention once at training time instead of deciding per step.
FlashMemory's Lookahead Sparse Attention — answers which chunks of the cache to load, rather than whether to load any of it.
MiniMax M3's block-sparse attention — the same which-blocks question at block granularity, shipped inside an open-weight 1M-context model.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based