What is training-free attention linearization?

It is converting a model that was trained with softmax attention into one that uses linear attention — a fixed-size, constant-memory form of attention — without retraining the base model. The backbone weights are left frozen, and only lightweight structure (sink tokens, a short convolution, and fixed-budget cache routing) is added on top. The study analyzes this on LLaMA and Qwen models up to 32B parameters and reports long-context behavior matching far more complex adaptive-caching frameworks.

Why does linear attention lose long-context recall?

Softmax attention makes a sharp, key-dependent selection over the whole history — the paper frames it as a rank-1 orthogonal projection — and a plain linear map cannot reproduce that sharpness, so a naive swap onto a trained model forgets what happened far back. Linear attention also collapses the entire history into one fixed-size state, so without help the earliest context washes out and short-range detail blurs. The fixes add those pieces back: sink tokens keep a few anchors, a short convolution restores local precision, and fixed-budget cache routing preserves far-back tokens within a bounded cache.

How does fixed-budget cache routing help?

Instead of a KV cache that grows with every token, fixed-budget cache routing keeps a bounded number of slots and uses a rule to decide which tokens occupy them — the same kind of choice a serving engine makes when it evicts from a bounded cache. The paper isolates this as the intervention that most recovers long-context retrieval: by spending the limited slots on the tokens it keeps — a few sinks, a recent window, and the rest of the fixed budget — the model keeps a constant-size cache yet still answers long-context questions, all without retraining the base model.

Linearization study isolates cache routing for long-context attention — Training-free linear attention

TL;DR

What is it: A frozen-backbone study (arXiv:2607.07706) converts a trained softmax attention model to linear attention — a fixed-size, constant-memory form of attention — without retraining the base model, and isolates fixed-budget cache routing as the piece that recovers long-context retrieval.
Why it’s needed: Softmax attention keeps a KV cache that grows with every token, so long context gets expensive; linear attention keeps a fixed-size state instead — but a plain swap onto a trained model forgets what happened far back, and this work recovers that recall while the base model stays frozen.
vs previous: Earlier routes to linear attention retrain or distill the model into the new form (for example Taylor-Calibrate's gate initialization); here the backbone stays frozen and only cheap add-ons — sink tokens, a short convolution, and fixed-budget cache routing — are bolted on top.

Jargon

Softmax attention: The standard attention every token uses to look back over all prior tokens, weighting them with a softmax. It is exact but its KV cache grows linearly with context length. Full primer →
Linear attention: An attention variant that replaces the softmax with a kernel feature map, so the whole history collapses into a single fixed-size recurrent state instead of a growing cache — cheap and constant-memory, but weaker at sharp, far-back lookups.
Linearization (training-free): Converting a trained softmax model into a linear-attention model without retraining the base model — the base weights ("the backbone"), the expensive part, are left frozen; only lightweight structure is added on top, which may need minor calibration rather than a full training run.
Rank-1 key-dependent projection: The paper's framing of what softmax attention actually does: each step is a sharp, key-dependent selection that behaves like a rank-1 projection. A plain linear map cannot reproduce it, which is why a naive swap loses long-range recall.
Attention sink: A few tokens (often the first ones) that the model always keeps and attends to. Bolting these into the linear state stabilizes it and stops the far-back context from washing out.
Short convolution: A small local operation over the last few tokens, added alongside linear attention to restore the short-range precision the fixed state blurs.
Fixed-budget cache routing: Instead of a cache that grows with context, keep a bounded number of slots and use a rule to decide which tokens occupy them — the piece this study isolates as the one that recovers long-context retrieval. Related to cache eviction.
Delta rule vs gated accumulation: Two ways to update the linear state as tokens arrive: a delta rule writes and erases entries explicitly, while gated accumulation decays the old state and adds the new. The paper compares them as the state-update backbone.

The news. On July 8, 2026, a study (arXiv:2607.07706) analyzed post-hoc transformer linearization in a frozen-backbone setting across LLaMA and Qwen models up to 32B parameters. It argues softmax attention relies on key-dependent, rank-1 orthogonal projections, then adds structural interventions — sink tokens, short convolutions, and fixed-budget cache routing — that improve long-context behavior without retraining the base model, reportedly matching the long-context retrieval of far more complex adaptive-caching frameworks. The point is not a new model but a recipe: the same frozen weights keep their training and are handed cheap structural add-ons instead of being retrained. Read the paper →

Training-free linear attention takes a model that was trained with ordinary softmax attention and converts it to a cheaper, fixed-memory form, with the original weights left frozen. Picture a reference librarian who has always answered questions by walking into an endless archive and pulling any book she needs — for every question, from anywhere in the stacks. That is softmax attention: exact, but the archive she has to search keeps growing with every book added. Now the budget is cut and she is handed one small cart with a fixed number of shelves. She is the same librarian with the same training — nobody re-taught her — she simply has to work from the cart. On her own, she would forget most of the archive. Training-free linear attention is the trick of keeping the same librarian and the same training, but handing her one fixed-size cart plus a few tricks, so she rarely misses the book she actually needs.

Underneath the metaphor, the growing archive is the KV cache. Ordinary softmax attention stores a Key and a Value for every token, so the memory it must scan grows linearly with context length — the deeper into a long document you go, the more it costs. Linear attention swaps that growing archive for a single fixed-size state — constant memory no matter how long the context — but a plain swap on an already-trained model loses the sharp, far-back lookups softmax was doing. The paper frames softmax attention as leaning on key-dependent, rank-1 selections that a plain linear map struggles to reproduce, which is why a naive conversion tends to forget. How you update that fixed state matters too — the study weighs a delta rule (write and erase entries explicitly) against gated accumulation (decay the old, add the new) as the state-update backbone.

So instead of retraining, the study bolts three cheap fixes onto the frozen model, and each maps to something the cart was missing. Sink tokens, a short convolution, and fixed-budget cache routing add back exactly the pieces the linear state drops — a few always-kept anchors, the local detail, and a rule for which tokens earn a slot in the bounded cache. Sink tokens are the reference books bolted to the cart so the earliest context never washes out; the short convolution is the sticky note of the last page or two, restoring short-range precision; and fixed-budget cache routing is the rule that decides which of the many past tokens get to occupy the cart's limited shelves — the piece the paper isolates as the one that actually recovers long-context retrieval. That routing decision is the same family of choice a serving engine makes when it evicts from a bounded cache:

Remove earliest tokens first

Where it earns its keep

Take a 32B model at a 32,768-token context (illustrative numbers — the paper reports models up to 32B and long-context retrieval matching adaptive-caching frameworks, not these exact counts). With full softmax attention, the model keeps a Key and Value for every one of the 32,768 tokens, and that cache grows without bound as the conversation gets longer. Now fix the cache budget at 2,048 slots and hand the model the three add-ons. The question is no longer "store everything" but "which 2,048 of the 32,768 tokens earn a slot?" — sink tokens claim the first few, the short convolution covers the last few, and fixed-budget cache routing spends the rest of the budget on the tokens it keeps. 32,768 tokens, but still only 2,048 cache slots — a ~16× smaller cache, held constant as the context grows instead of growing with it. That is the kind of trade the frozen-backbone recipe is built to make — the base model itself is never retrained.

Route to linear attention	Retrains the model?	Long-context memory	Long-context recall
Full softmax attention (baseline)	no — it is the original	grows with context (KV cache)	strong, but increasingly expensive
Distill / retrain into a linear or hybrid form (e.g. Taylor-Calibrate)	yes — training or distillation tokens	fixed-size state	recoverable, but needs a training budget
Training-free linearization + fixed-budget cache routing	no — the backbone stays frozen	fixed-budget cache (~bounded slots)	matches adaptive-caching frameworks, per the paper (setup-dependent, illustrative)

Because the change lives in how attention reads its history rather than in the model's weights, it should in principle stack with the rest of the long-context toolkit — a smarter eviction rule, a different state update, or grouped-query sharing of the cache are all orthogonal to it — and the real-world gain will depend on the model family and how much long-range recall the task demands. The headline is not a new architecture; it is a recipe for making an existing trained model attend linearly at long context without retraining the base model — with fixed-budget cache routing doing the heavy lifting.

Goes deeper in: LLM Internals → Attention → Computing Attention Scores

Continue in trackKV Cache — why softmax attention's memory grows, and what a fixed budget buys

Related explainers

Taylor-Calibrate — Taylor-guided gate initialization — the distillation route to linear attention; this study instead keeps the backbone frozen and adds structure on top
Gated DeltaNet 2 — decoupled erase/write gates — the delta-rule vs gated state update this paper weighs, built into an architecture from the start
WorldKV — evict then reinsert the KV cache — another take on the fixed-budget question: which past tokens deserve a cache slot
Baidu Unlimited OCR — constant KV via reference sliding window — holding the cache constant over long inputs by a different structural trick

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based