What is SAGA attention?

SAGA (Sparse Asymmetric Group-Query Attention) is an attention layout with **separate head counts for keys and values**. It keeps few key heads, so the query-key scoring step reads less memory, and many value heads, so the model keeps its capacity.

Why does sparse attention make keys the bottleneck?

Top-N sparse attention reads only N value vectors per query head, so the value side becomes cheap. The query-key step still has to score **every cached token** to find those N, so it reads all the keys and dominates the decode step.

How is SAGA different from grouped-query attention (GQA)?

GQA shares key and value heads between groups of query heads but always uses the **same number of key heads and value heads**. SAGA lets the two numbers differ, for example 1 key head and 8 value heads, which GQA cannot express.

How much faster is SAGA in the paper?

Combined with the paper's approximate top-N attention, it reports up to **2.2× on Qwen2.5-1.5B and 2.1× on Llama 3.2-1B** end-to-end at 128K tokens and batch size 1, against a full-attention GQA baseline. At short contexts the sparse step adds overhead instead.

LLM· 2026-10-07

Decouple key and value heads for 2× sparse-attention decoding — Asymmetric key and value heads — What does it mean?

TL;DR

What is it: SAGA (Sparse Asymmetric Group-Query Attention), from an October 2026 NeurIPS paper, lets a transformer keep fewer key heads than value heads instead of tying the two counts together the way grouped-query attention does.
Why it’s needed: During long-context decoding with dense attention the model reads every cached key and value for each new token, so cutting the bytes read per token is what makes generation faster, and the paper reports more than 2× end-to-end decoding speedup at 128K tokens when SAGA is paired with sparse attention.
vs previous: Standard GQA uses one count for both key and value heads, which made sense when the two sides cost the same; once top-N sparse attention shrinks the value side, the keys become the bottleneck, and GQA has no way to cut keys without also cutting value capacity.

Jargon

Decoding: The phase where the model produces one new token at a time. Each new token must look back over every earlier token, so long contexts make each step slower.
KV cache: The stored keys and values of every earlier token, kept in GPU memory so they are not recomputed. Reading it is the main cost of a long-context decode step. See the KV cache module.
Key head / value head: Attention splits keys and values into several parallel heads. Each key head is compared against the query; each value head holds the content that gets mixed into the output.
GQA: Grouped-query attention: several query heads share one key head and one value head, which shrinks the KV cache. The number of key heads and value heads is always the same.
Query-key step: Comparing the new token's query against every cached key to get a score per token. It must read all cached keys.
Probability-value step: Turning the scores into weights and adding up the value vectors with those weights. This is the step that produces the attention output.
Top-N attention (Atop-N): A sparse attention rule that keeps only the N highest-scoring tokens per query head and ignores the rest. Atop-N is the paper's approximate, fast version of it.
Memory bandwidth: How many bytes per second the GPU can move from its main memory. A decode step is usually limited by this, not by arithmetic.

The news. On October 3, 2026, Noam Elata, Itay Lamprecht, Mikey Shechter, Daniel Ohayon, Itay Hubara and Daniel Soudry posted More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding, accepted to NeurIPS 2026. The paper observes that sparse attention moves the decode bottleneck from the probability-value step to the query-key step, proposes SAGA to cut key heads independently of value heads, and tests it on models up to 1.5 billion parameters, plus a fine-tuning recipe that converts existing GQA models. Read the paper →

Picture the library at closing time. A reader asks a question, and the librarian has to flip through every card in the catalog to rank which books are relevant. Then they walk to the shelves and pull the matching books. In a normal library both trips are long: the catalog is huge, and so is the pile of books to carry back.

Attention in a decoding model works the same way. The new token's query is compared against every cached key to compute a score per earlier token, and then the value vectors are mixed together using those scores. The query-key step reads all cached keys and the probability-value step reads all cached values, so in dense attention the keys and the values cost about the same. That is why GQA uses one dial for both: fewer shared heads means fewer keys and fewer values, and the two savings are equal.

Now change the rule at the shelves: the librarian still ranks every card, but carries back only the top few books. That is top-N sparse attention. The walk to the shelves becomes short no matter how big the library is. The catalog flip did not get shorter, so it is now almost the whole trip. The paper's point is exactly this: under top-N attention, the value side reads only N vectors per query head, while the key side still reads every cached key.

If the drawers are the slow part, remove drawers, not shelves. SAGA splits GQA's single head count into two: a small number of key heads and a larger number of value heads. Fewer key heads means fewer keys to read in the query-key step, which is the step that now limits speed. Keeping the value heads preserves the model's capacity, and it costs little, because only the top-N values are ever read. Fewer key heads also shrink the KV cache, because keys are half of a symmetric GQA cache.

The paper tests the trade in a small training run. With 16 query heads on a SmolLM2-360M model, the configuration with 1 key head and 8 value heads closely tracked the 8-key, 8-value baseline while using a little over half its KV cache, and it beat both a 1-and-1 model with the same key budget and a 4-and-4 model with a similar parameter count. The win came from spending capacity unevenly, not from adding parameters.

Setup (illustrative, one layer, one decode step)	Key vectors read	Value vectors read	Total
Dense GQA, 8 key + 8 value heads	~1,048,576	~1,048,576	~2,097,152
GQA + top-N (N = 1,024)	~1,048,576	~32,768	~1,081,344
SAGA + top-N, 4 key + 8 value heads	~524,288	~32,768	~557,056

Here is the arithmetic behind that table (illustrative). It counts vector reads, assumes keys and values are the same size, and leaves out the cost of choosing the top-N tokens. Take a Llama 3.2-1B-shaped layer, which the paper uses: 32 query heads and 8 key-value heads, with a context of 131,072 tokens, batch size 1. Dense attention reads 8 × 131,072 = 1,048,576 key vectors and the same number of value vectors. Turn on top-N with N = 1,024, and the value side drops to 32 query heads × 1,024 = 32,768 vectors, while the key side stays at 1,048,576, so keys are now about 97% of the attention reads. Halve the key heads from 8 to 4, as the paper does for this model, and the key side falls to 524,288: total attention reads drop from ~1.08 million to ~0.56 million vectors, about 1.9× less than GQA with top-N, and about 3.8× less than dense GQA. The measured end-to-end gain is smaller than that attention-only ratio, because a decode step also reads the model weights and the sparse selection adds its own overhead: the paper reports up to 2.1× on Llama 3.2-1B at 128K tokens and batch size 1 for SAGA with Atop-N against full-attention GQA, and up to 1.47× from merging key heads alone at batch size 16.

The quality bill is real but modest at this scale. A Qwen2.5-1.5B model converted to SAGA by short distillation (trained for a short time to copy the original model's outputs) kept 96.3% of its teacher's aggregate RULER score across 8K to 32K contexts, where RULER is a benchmark of finding and using facts in long inputs. On HellaSwag, a common-sense benchmark, its score fell from 0.678 to 0.644. The losses from Atop-N concentrate in multi-key retrieval, where the model needs many scattered tokens at once and a top-N cut drops some of them. The paper evaluates SAGA quality only on models up to 1.5 billion parameters; it also measures decoding speed on an untrained 32B architecture, but whether the same head split preserves quality at frontier scale is not yet shown.

Goes deeper in: LLM Internals → KV Cache → GQA: Shrinking the Cache

Why the decode step is limited by bytes read rather than arithmetic is the subject of GPU & CUDA → Roofline Model → Prefill vs Decode.

Related explainers

MiniMax Sparse Attention (MSA) — another way to choose which cached tokens the value step reads, by blocks instead of single tokens
IndexCache cross-layer sparse-index reuse — a different attack on the same scoring cost: compute the token selection once and reuse it across layers

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based