LatentIndex is a method for **DeepSeek Sparse Attention** that lets a group of four layers share **one compact indexer cache** instead of four separate ones. Each layer reads the shared cache through its own decoder and still selects **its own 2,048 tokens**.

Why does the sparse-attention indexer need its own optimization?

Sparse attention makes the main attention cheap, but the **indexer** still stores a key per token in every layer and scans the full history on every decode step. On DeepSeek-V3.2 that side cache is **8,052 bytes per token** across 61 layers, about 1.06 GB at 128K tokens.

How is LatentIndex different from IndexCache?

IndexCache shares the **selected token list** across layers, so later layers must use tokens chosen by an earlier one. LatentIndex shares the **stored representation** and lets every layer score and choose for itself, which cut average follower recall loss at 128K on DeepSeek-V3.2 from **6.69 to 1.81 points** in the paper's tests.

Does LatentIndex shrink the main KV cache?

No. It only replaces the **indexer's** key caches with a shared latent. The main-attention KV cache is unchanged, so the 61.1% saving applies to the indexer side cache alone.

LLM· 2026-10-07

LatentIndex — Shared indexer cache, per-layer token choice — What does it mean?

TL;DR

What is it: LatentIndex (an October 2026 arXiv paper) is a way to run DeepSeek Sparse Attention's per-layer indexers off one shared latent cache per group of layers, while each layer still picks its own tokens.
Why it’s needed: The indexer that decides which 2,048 tokens each layer attends to keeps its own key cache in every layer and rescans the whole history, so at long context it becomes a memory and decode-latency cost of its own, on top of the main KV cache.
vs previous: IndexCache cuts that cost by reusing one layer's token choices in the layers after it, which forces those layers to read tokens chosen for a different query; LatentIndex shares the stored representation, not the decision, and in the full-scan version on DeepSeek-V3.2 cuts average follower recall loss at 128K from 6.69 to 1.81 points.

Jargon

Sparse attention: Attention where each new token reads only a selected subset of earlier tokens instead of all of them. It saves compute, but something has to do the selecting.
DSA lightning indexer: The small, cheap scorer in DeepSeek Sparse Attention that, in every layer, scores all earlier tokens and hands the top 2,048 to the main attention. See the DSA explainer.
Indexer key cache: The per-token keys the indexer stores so it can score the history on each decode step. It is separate from the main KV cache: on DeepSeek-V3.2 it is 132 bytes per token per layer.
IndexCache: An earlier method that keeps the indexer in a few layers and lets the next layers reuse the same selected token list. See the IndexCache explainer.
MLA: Multi-head Latent Attention, DeepSeek's design that stores one compact latent per token and lets every attention head read it through its own projection. LatentIndex applies the same idea across layers.
Anchor and followers: In a LatentIndex group of four layers, the anchor is the first layer, whose hidden state builds the shared latent; the other three are followers.
Decoder absorption: Folding a layer's decoder matrix into its query instead of into every cached key, so the shared cache can be scored without rebuilding per-layer keys.
Attention-mass recall: The attention weight the indexer's chosen tokens capture, divided by the weight that each head's best possible 2,048-token pick would capture, averaged across heads. Higher means closer to that same-budget ideal.

The news. On October 3, 2026, researchers from Peking University and Xiaohongshu posted LatentIndex (arXiv 2610.04635). It targets the indexer of DeepSeek Sparse Attention (DSA): the small scorer that, in every layer, scans the whole history to choose the 2,048 tokens the main attention will read. Training-free tests on DeepSeek-V3.2 and GLM-5 keep downstream scores close to native DSA; on DeepSeek-V3.2, four-layer sharing cuts logical indexer-cache storage by 61.1%, and a variant called hierarchical selection runs the decode indexers 2.30–2.72× faster than native DSA across 8K–128K contexts. Read the paper →

Picture a library floor with four researchers, each answering a slightly different question about the same shelf of 128,000 books. In native DSA, every researcher keeps a private card catalog of the whole shelf and reads all of it before pulling their 2,048 books. In model terms, each layer stores its own indexer key for every past token. That is the memory bill a KV cache already teaches, paid a second time by the indexer: DeepSeek-V3.2 has 61 indexer layers, each storing 132 bytes per token.

The obvious shortcut is IndexCache's: let the first researcher choose, and hand the same pile to the other three. Neighbouring layers do choose similar tokens, so this mostly works, but the followers lose the right to disagree, and the cost grows with context. At 128K tokens, the paper measures IndexCache followers losing 5.97–7.38 percentage points of attention-mass recall against what their own native indexer would have picked.

LatentIndex shares the catalog instead of the choices. The anchor layer of each four-layer group writes one compact shorthand card per token (a 192-value latent), and every layer in the group reads that same card through its own small decoder, then scores and picks its own 2,048 tokens. This is the trick MLA uses across attention heads, moved across layers. It is also a cousin of GQA, which lets several query heads share one key. Each researcher still answers their own question; they just stop copying out a private catalog first.

The catch is the cost of each lookup. Turning the shorthand back into a full key for every past token, in every layer, on every decode step would bring back the work the method was meant to remove. Instead, each layer folds its decoder into its query once per new token, so it scores the shared cards directly. This is plain linear algebra on the query-key dot product: q · (D c) equals (Dᵀ q) · c, so the decoder D (plus a small fixed bias term) is applied once to the one new query instead of to 128,000 cached latents c. In the library, the researcher translates their question into shorthand, rather than translating the whole catalog into their own words.

Sharing the cards saves memory, but each follower still reads the whole catalog. Hierarchical selection (HS) removes that scan: the anchor fills a cart with its 8,192 best candidates, and each follower picks its own 2,048 from the cart only. Followers keep their own judgment inside a smaller pool, which is why HS lands between native DSA and IndexCache on both speed and recall in the table below.

Method (DeepSeek-V3.2)	What layers share	Recall (%), avg. of six lengths	Indexer time per decode token, 128K	Source
Native DSA	Nothing: own cache, own scan, own pick	94.40	49.60 ms	Table 4
IndexCache	The token choice itself	91.54	13.40 ms	Table 4
LatentIndex (full scan)	The latent cache; each layer picks	93.48	not reported	Sec. 4.5
LatentIndex + HS	Latent cache and an 8,192-token candidate cart	93.19	20.48 ms (2.42× vs DSA)	Table 4

Four-layer sharing cuts the indexer's side cache by 61.1%, and HS cuts each follower's scoring work. Hold the model at DeepSeek-V3.2's 61 indexer layers and the context at 128K tokens (131,072). Native DSA stores 61 × 132 B = 8,052 bytes of indexer keys per token, about 1.06 GB for one sequence. LatentIndex keeps layer 0 native and turns the other 60 into 15 groups of four, each storing one 200-byte latent: 132 + 15 × 200 = 3,132 bytes per token, about 0.41 GB, which is 61.1% less. The main-attention KV cache is not touched; this is the indexer's side cache only, counted as logical bytes. On the compute side, each HS follower at 128K scores 8,192 candidates instead of 131,072 positions, 16× fewer, and the measured time for all 61 indexers per decode token (batch size one, eight GPUs) falls from 49.60 ms to 20.48 ms. IndexCache is faster still at 13.40 ms, because its followers score nothing; that 7 ms gap is the price of letting followers choose.

The trade has a second cost, measured on the full-scan version without HS. Because the anchor now reads shorthand too, even it loses a little: 1.32 points of recall at 128K, where IndexCache's anchor loses none. In exchange, average follower loss at 128K falls from 6.69 to 1.81 points, a 73% cut, and the paper finds IndexCache's loss growing with context length, which is where sparse attention matters most. The training-free version also needs a one-time calibration per model: the shared projections and decoders are fitted offline by regression and then frozen. On RULER and LongBench, two long-context benchmarks, scores stay close to native DSA's on both models tested.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

Hy4 Preview — IndexCache cross-layer index reuse — the method LatentIndex is measured against: share the choice, not the cache
Keye-VL-2.0 — DeepSeek Sparse Attention for video — how the DSA indexer picks tokens in the first place

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based