SinkState is a serving system for hybrid LLMs that stores each linear-attention recurrent state as low-rank SVD factors instead of a dense matrix. It buffers 8 tokens of updates, re-factors only the top 8 components when the buffer fills, and merges the remaining tail on the CPU in the background.

Why are recurrent states low-rank in the first place?

The paper finds that queries, keys and values in linear-attention layers concentrate in a few stable directions, a pattern it calls the **linear attention sink**. Because each token adds a key-times-value outer product along those same directions, the state piles up in a few components: the top 2–15 hold 99% of its energy.

How is this different from KV-cache compression?

A KV cache grows by one row per token and old rows are only read, so it can be compressed once. A recurrent state has a fixed size but every value changes each step, so its compression must be maintained online; SinkState does that with local SVD updates instead of a full SVD per step.

Does it hurt accuracy?

In the paper's tests, accuracy moved between −1.6 and +2.6 points across 24 model-benchmark pairs with no consistent direction, matched the baseline within 0.1% on RULER needle-in-a-haystack tests at 16K context, and lost at most 0.5% on 32K-token reasoning runs. These numbers come from four models on one GPU type.

LLM· 2026-11-12

SinkState — On-the-fly low-rank recurrent state — What does it mean?

TL;DR

What is it: SinkState (arXiv 2611.14246, Rutgers, November 10, 2026) is a serving system that keeps every recurrent state in a hybrid model's linear-attention layers as low-rank factors (a few column patterns, row patterns and their strengths, found by an SVD) updated on the fly, instead of as a dense matrix.
Why it’s needed: In hybrid models such as Nemotron 3 and Qwen3.5, every decode step reads and rewrites each dense state in full, so state memory traffic grows with batch size; storing only the few directions that matter cut peak state memory by up to 2.6× and sped decoding up to 1.58× with accuracy held.
vs previous: Earlier fixes either keep the state dense (state reuse, deferred writes) or compress it with a static, offline-chosen basis (state pruning, calibrated projections) that may not fit unseen inputs; SinkState tracks each state's own rank online with a cheap local SVD over its top 8 directions.

Jargon

Hybrid model: A language model that mixes a few standard attention layers, which compare each new token with every past token, with many linear-attention (recurrent) layers, such as Mamba-2 or GDN, which summarize past tokens in a fixed-size state to cut memory and compute on long sequences.
Recurrent state: The fixed-size matrix a linear-attention layer keeps per head and per request. Each token decays it and adds its key times its value; the query then reads the output from it. It plays the role the KV cache plays for softmax attention.
Mamba-2, GDN, KDA: Three linear-attention layer designs. They differ mainly in how the state forgets old content: Mamba-2 uses a scalar decay, GDN a delta rule that first erases what the state holds along the new key, and KDA a per-column decay. All of them still write one key-times-value outer product per token.
SVD: Singular value decomposition: it rewrites a matrix as a sum of rank-1 pieces, each a column pattern times a row pattern times a strength (the singular value). Dropping the weakest pieces gives the best smaller approximation.
Rank: How many independent rank-1 pieces a matrix really needs. An 80 × 128 state can have rank up to 80, but SinkState's profiling finds that a handful of pieces carry almost all of it.
Linear attention sink: The paper's name for its finding that queries, keys and values in linear-attention layers concentrate in a few stable directions, even on random tokens, which is why each state is effectively low-rank.
Memory-bandwidth-bound: A step whose time is set by how many bytes move between GPU memory and the compute units, not by arithmetic. Decode steps usually are, so moving fewer bytes makes them faster.

The news. On November 10, 2026, Haolin Jiang, Weihang Li and co-authors (Rutgers, University of Sydney, University of South Florida) posted SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving. They tested four hybrid models, Nemotron-3-Nano-4B and Nemotron-3-Super-120B (Mamba-2 layers) and Qwen3.5-9B and Qwen3-Next-80B (GDN layers), on one RTX PRO 6000 Blackwell GPU against the dense Mamba-SSM and FLA kernels that vLLM and SGLang use. Peak state memory fell 1.6–2.6×, decoding sped up 1.25× on average and up to 1.58×, and accuracy moved between −1.6 and +2.6 points with no consistent direction. Read the paper →

A linear-attention layer squeezes the whole history into one fixed-size matrix, and SinkState's finding is that this matrix is mostly made of a few repeated patterns. Picture a tartan. You can store it as a photo, writing down the color of every cell, or as its thread recipe: a few stripe patterns running down, a few running across, and how bold each one is. The recipe is far smaller and redraws the same cloth. A recurrent state is built the same way. Each token adds one outer product, its key times its value, which is one new stripe in the cloth.

The authors profiled Mamba-2, GDN and KDA layers and found that the queries, keys and values keep landing in the same few directions, so the mill keeps reusing the same few stripes. On PG-19 (a corpus of books) the top 8 directions hold 73–88% of the energy of the stacked vectors (their total squared size) in most cases, the pattern holds at 64K tokens of context, and it is even stronger on random tokens. They call this the linear attention sink. The consequence: across the profiled states, which have up to 128 components, the top 2–15 components hold 99% of a state's energy, and keeping only the top 2–16 recovers 99% of its output.

The waste is not only storage: a dense state is read and rewritten in full on every decoding step. A KV cache grows by one row per token and old rows are only read; a recurrent state stays the same size, but every cell changes each step. Decode does little arithmetic per byte it moves, so it is memory-bandwidth-bound, and the paper measures dense state-kernel time growing linearly with batch size. Storing the photo when a recipe would do means every token pays to move the whole photo twice.

The obvious fix, recomputing an exact SVD after every update, is far too slow. At batch size 128, Nemotron-3-Nano-4B holds about 258,000 states, and an exact SVD of all of them takes about 4,000 ms, against a normal decode step of 20–30 ms. SinkState instead keeps the recipe up to date with three cheap moves.

Prefill: randomized SVDfactor each state block by block; stop when a singular value drops below ε × the largest

Decode: bufferdecay scales only the strengths; each token's key and value join an 8-slot pad; outputs read from factors plus pad

Pad full: local SVDre-factor only the top 8 components plus the pad; the weaker tail stays frozen

Truncatemerge with the frozen tail and drop components below ε × the largest

Background: CPU consolidationexact SVD of each frozen tail on idle CPU cores, layers in round-robin order

The trick that makes on-the-fly SVD affordable is touching only the boldest stripes. New keys and values mostly line up with a state's top components, and the paper measures the last components absorbing as little as 0.03–0.1% of each update. So when the 8-token pad fills, SinkState runs a small SVD on just the top 8 components plus the pad (using Brand's incremental SVD, which decomposes a small core matrix rather than the full factors), then drops any component weaker than ε times the strongest (ε = 3 × 10^-3 for the Nemotron models, 10^-2 for the Qwen models). Between compactions, the decay only rescales the strengths, and the query reads its output straight from the factors and the pad without rebuilding the dense matrix. By staggering compactions so only a few layers run them on any one step, SinkState cut the slowest 1% of decode steps (P99) by 15.9–40.9 ms versus compacting every layer together.

Back at the mill, leaving the faint stripes frozen has a cost: later updates re-add similar directions as separate stripes, and the stored rank creeps above what an exact SVD would keep. That is the night-shift clerk's job. The host CPUs sit at about 2% utilization during decode, so SinkState copies each frozen tail to the CPU, runs an exact SVD there to merge duplicates, and copies the result back before the next compaction, cycling through layers in round-robin order because the CPU cannot keep up with every layer at once.

Here is the memory arithmetic for one Nemotron-3-Nano-4B state (rank 8 is illustrative; the shape, FP32 format, 8-token pad and state count come from the paper). The dense state is 80 × 128 = 10,240 FP32 values, 40,960 bytes, and at batch 128 the model holds 21 layers × 96 heads × 128 requests = 258,048 of them, about 10.6 GB read and rewritten every step. A rank-8 recipe stores 8 × (80 + 128 + 1) = 1,672 values, and a full pad adds another 8 × (80 + 128 + 1) = 1,672 (each buffered token keeps its key, its value and a strength weight), for 3,344 values or 13,376 bytes. That is about 3× smaller than the dense state. The paper's measured peak state memory on this model is 0.38–0.43 of the dense baseline (about 2.3–2.6× smaller), in the same range, because real ranks vary per state and the frozen tail carries extra components between consolidations.

Approach	How the state is stored	Who picks what to keep
Dense kernels (Mamba-SSM, FLA)	Full matrix, read and rewritten each step	Nothing is dropped
State reuse, deferred state writes	Full matrix, fewer reads or writes	Nothing is dropped
State pruning, calibrated projections	Smaller, on a fixed basis	A static choice made offline; no guarantee on unseen inputs
SinkState	Low-rank SVD factors plus an 8-token pad	Each state's own singular values, re-checked online at every compaction

The payoff scales with how much of the decode step the states own. Speedup was largest on Nemotron-3-Nano-4B (1.30–1.58×), whose recurrent states dominate step time, and smallest on Nemotron-3-Super-120B (1.04–1.16×), where mixture-of-experts (MoE) layers take a larger share and big weights leave less memory for large batches. At small batches the state kernels do not saturate memory bandwidth, so shrinking their traffic helps less. The memory side matters for the memory manager that decides how many requests fit: the dense baselines ran out of memory beyond batch 640 on Nemotron-3-Nano-4B and 768 on Qwen3.5-9B, while SinkState still ran at 1,280 on both. On long-context retrieval (RULER needle-in-a-haystack tests at 16K tokens) it matched the baseline within 0.1%, and on 32K-token reasoning runs it lost at most 0.5%. All results are from one GPU model and four models, so treat the exact ratios as setup-dependent.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

Decay-aware bit allocation for low-bit recurrent states: shrinks the same dense state by cutting bits per value, not rank
Stochastic rounding for BF16 recurrent state: why the state's number format changes what the model says
Basis-synchronized online low-rank KV compression: the softmax-attention cousin, online low-rank factors for a growing KV cache

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based