MERLIN — Second-moment routing for multi-state linear attention — What does it mean?
The news. On October 27, 2026, a Princeton and Google DeepMind team posted MERLIN, a routing method for multi-state linear attention. On multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M, the authors report gains of +8.3% and +7.0% on RULER retrieval and long-context QA, and +13.5% and +12.0% on LongBench, over the strongest baseline. Read the paper →
Picture a library with one summary card per shelf instead of a catalogue entry per book. Full softmax attention keeps a KV cache that grows with every token, the equivalent of reading every book. A single-state linear-attention model has one card for the entire library, so it forgets detail from far back. Multi-state linear attention keeps a card per shelf: each segment of the context gets its own fixed-size state, and a new token reads all visible states and adds their readouts together as a weighted sum. The open question is the weights: how much should the librarian trust each shelf's card for this particular question? That step is called aggregation, and it is the part MERLIN changes.
Two routers existed before. A position router computes the weights from the current token alone, with no look at what a shelf contains, so it can only learn habits such as “recent shelves matter more”. A mean router compares the query with the average key of each segment. That is content-aware, but the average hides the spread: a shelf where seven books each match a little and a shelf where six books are irrelevant but one matches exactly can have the same average.
MERLIN stores two things per segment beside its memory state: the first moment μ, the mean-pooled key, and the second moment Σ, the average of each key's outer product with itself. Both are built from keys projected into a smaller dimension d_r, so their size does not grow with segment length. At query time the router computes two scalars per segment with plain matrix-vector products: the mean score q·μ, and the variance score qᵀΣq − (q·μ)², which measures how much the per-token matches inside the segment differ. Each score goes through its own softmax across segments, and a learned gate computed from the token's hidden state mixes the two softmaxes into the final weights. Two learnable temperatures set how sharp each softmax is.
The variance is not an arbitrary extra feature. Suppose full softmax attention ran over every key and you then summed the attention mass that lands on each segment. The logarithm of that mass has an expansion around zero temperature whose first terms are the mean match, then half the variance of the match, then higher-order terms. The first two moments therefore carry the same two statistics that lead softmax attention's own segment mass, and MERLIN reads them from the stored moments without keeping any keys. It does not use the expansion's fixed coefficients: it feeds the two statistics into a learned gated mixture of separately normalized softmaxes, which the authors describe as flexible to implement because each term is scaled on its own.
| Router | Weight comes from | Sees the spread inside a segment? | Source |
|---|---|---|---|
| Position | The current token's hidden state or query only | No | MERLIN §2 |
| Mean-pool | Query against the segment's average key | No | MERLIN §2 |
| MERLIN | Gated mix of a mean softmax and a variance softmax | Yes, from the second moment | MERLIN §3 |
Take one query and two shelves of four books each (illustrative numbers). Shelf A matches the query at [0.5, 0.5, 0.5, 0.5] and shelf B at [2.0, 0, 0, 0]. Both have a mean of 0.5, so the mean router scores them the same. The variance score is the mean of the squares minus the square of the mean: shelf A gives 0.25 − 0.25 = 0, and shelf B gives 4 ÷ 4 − 0.25 = 0.75. The second softmax now favours shelf B, and the gate decides how much of the final weight comes from that second view.
On the real benchmark, the paper's Table 2 reports a multi-state Gated DeltaNet-1.3B, mid-trained and tested on RULER retrieval tasks at 16K context. Its average is 57.9 with the mean router and 62.7 with MERLIN: 4.8 points, or 8.3% relative, consistent with the +8.3% in the abstract. The position router scored 54.6 and the vanilla aggregation 49.6, against 13.0 for the base model before mid-training.
The price is a second state per segment. Multi-state linear attention sits between single-state linear attention (constant memory) and full attention (memory that grows with length), because the number of states grows with the number of segments L. The paper's Table 1 lists MERLIN's memory as L(d² + d_r²), against L·d² for the mean-only router. The authors report that the extra memory from the second-moment features is at most 12.6% for Gated DeltaNet and 4.2% for Mamba-2 at d_r = 128, falling quadratically as d_r shrinks, and that parameter overhead is 4.6% and 2.2%. Their ablation shows the LongBench average still rising from d_r = 16 to 128 but saturating near 32 for Gated DeltaNet and 64 for Mamba-2. They also say MERLIN can beat baselines that use four times more memory. The evidence covers two small backbones; whether the gain holds at larger scale or other segment lengths is not documented in the paper.
Goes deeper in: LLM Internals → Attention → Computing Attention Scores
Related explainers
- Training-free linear attention — converts a trained softmax model to linear attention and recovers long-context recall with cache routing, a different way of deciding what to keep.