What is the Kronecker Delta Rule?

It is a write rule for the memory matrix of a linear-attention model. It reads one scalar, the memory's current score for a key–value pair, and corrects only that scalar along the pair's outer product. Other stored pairs shift only if they overlap in both key and value.

How is it different from the delta rule in DeltaNet?

The delta rule corrects along a residual that depends on the memory's current contents, so values under overlapping keys can be disturbed. The Kronecker rule fixes the direction from the key and value alone, so the disturbance to another pair is the product of key overlap and value overlap, which the paper identifies as less invasive.

Does KronLSTM replace softmax attention?

Not in the paper's claim. It is a linear-attention alternative with a fixed-size memory, evaluated both as a pure model and in a hybrid with sliding-window attention layers. The results are for 1.5B-parameter models trained on 100B tokens, so larger scales are untested there.

Why does bounded memory matter?

An unbounded recurrent memory can make readouts and gradients arbitrarily large, which destabilises training. The authors prove the expected memory norm stays bounded under their data model, so no extra normaliser states are needed.

LLM· 2026-10-21

KronLSTM — Kronecker Delta Rule memory updates — What does it mean?

TL;DR

What is it: KronLSTM, an arXiv paper posted on October 19, 2026, introduces the Kronecker Delta Rule: a way to correct one stored association in the fixed-size memory matrix of a linear-attention model without rewriting its neighbours.
Why it’s needed: Linear attention swaps a growing KV cache for a constant-size memory matrix, so how that matrix is edited decides whether a long context stays readable or decays into interference and unstable training.
vs previous: The delta rule used by DeltaNet and Gated DeltaNet corrects along a residual that depends on the memory's current content, so values stored under keys that overlap the written key get disturbed; the Kronecker rule fixes its direction from the key–value pair alone, so only entries that overlap in both key and value are touched.

Jargon

Linear attention: An attention variant that folds every past token into one fixed-size matrix, updated once per token. Cost per token stays constant, where a KV cache grows with every token.
Associative memory: A matrix S that maps keys to values: reading with a key k returns S k, which approximates the value that was filed under k.
Hebbian update: The plain way to write into that matrix: add the outer product of the value and the key. It only adds, so an earlier value can be buried by later writes but never corrected.
Outer product: A column vector times a row vector gives a matrix. Adding one to the memory changes it along a single direction, so which two vectors you pick decides which stored facts move.
Delta rule: An error-correcting write: read what the memory currently returns for key k, then write the difference from the wanted value. It is one gradient-descent step on the squared error, and it is the update inside DeltaNet and Gated DeltaNet.
Interference: The side effect of a write: it changes what the memory returns for other keys. The less a write interferes, the longer old facts stay readable.
Forget gate: A learned number between 0 and 1 that shrinks the memory before each write. It caps growth, but a gate close to 0 erases the start of a long context.
Steady state: The long-run behaviour of a repeated random update: the memory keeps changing but its size stays within a fixed, predictable range instead of blowing up.

The news. On October 19, 2026, a team including Sepp Hochreiter (an author of the original LSTM) posted Kronecker Delta Rule for Long Context Linear Attention on arXiv. It proposes KronLSTM, a linear-attention architecture built on a new write rule. At 1.5B parameters trained on 100B tokens of FineWeb-Edu, the authors report the best six-task average and the lowest LAMBADA perplexity among the linear-attention models they compared. Read the paper →

Picture the model's memory as a spreadsheet that never grows. Each fact it has read is a number written where a key column crosses a value row, and reading the sheet costs the same on token 100 as on token 100,000. That is the promise of linear attention: instead of keeping every key and value around, as a KV cache does, it keeps one fixed matrix. The open question is how to edit that sheet.

The oldest rule, the Hebbian write, never edits at all. It adds the new fact on top, so a wrong or outdated number is buried but never fixed, and the sheet's values keep growing unless a forget gate or a normaliser shrinks them. The delta rule fixes that. Before writing, it reads what the sheet currently says for this key and writes only the difference. But the correction is applied along the residual, a vector that depends on what the memory currently holds, so facts filed under a similar key get edited too. In spreadsheet terms, one wrong number makes the rule touch the whole column.

The Kronecker Delta Rule corrects the single cell where the column and the row cross. It reads one scalar, the memory's current score for this exact key–value pair, and moves only that scalar towards a target: a target of 1 stores the association at unit strength and a target of 0 erases it. The direction of the change comes from the input pair alone, never from the memory's contents. The paper shows the disturbance to any other stored pair is the product of value overlap and key overlap. Making either factor small protects that pair, whereas the paper identifies key overlap alone as the controlling factor for the Hebbian and delta rules. The spreadsheet is a schematic: real writes are dense outer products, and a cell stands for an association whose stored value moves. The Q, K, V roles are the same ones you already know; only the write changes.

Write rule	What one write does	What else it disturbs	How memory size behaves
Hebbian (linear attention, mLSTM)	Adds the value–key outer product on top	Nothing is revised; old values are buried	Grows with the sequence unless a forget gate and normaliser states hold it
Delta rule (DeltaNet, Gated DeltaNet, KDA)	Removes the value currently stored at the key, then writes the new one	Every value stored under an overlapping key	Bounded, converges to a steady state
Kronecker Delta Rule (KronLSTM)	Corrects one scalar association along the pair's outer product	Only pairs overlapping in both key and value	Bounded; the paper proves a bound on the expected memory norm

KronLSTM wraps the rule in three separate gates, all computed from the input: a forget gate per key dimension that decays old memory, an erase gate and a write gate, the last two bounded by a sigmoid. Under a simple model of the data (independent random key–value pairs), the authors prove the memory is a Markov chain that converges geometrically to a steady state, and that its expected norm stays bounded from the first step on. That bound is why the architecture needs none of the extra normaliser and stabiliser states that xLSTM's mLSTM uses; in the authors' high-learning-rate stress test, mLSTM shows gradient-norm spikes while KronLSTM trains steadily. The proof covers the expected norm under that data model, not a guarantee for every run.

Worked example (illustrative). Take one attention head with key and value width 128. The KV cache stores 2 × 128 = 256 numbers per token, so at an 8,192-token context it holds 8,192 × 256 = 2,097,152 numbers. The linear-attention memory is a 128 × 128 matrix, a fixed 16,384 numbers, so the matrix is 128× smaller at 8k tokens and stays that size at 80k. The price is that 16,384 numbers must hold every fact, so write interference matters. Now suppose a new write has key overlap 0.8 and value overlap 0.1 with a stored pair, with step size and error both 1. Under the Kronecker rule that pair's association shifts by 0.8 × 0.1 = 0.08. A rule whose interference is set by key overlap alone passes the full 0.8 key-overlap factor through (the actual shift for the delta rule also depends on the residual, so 0.8 is a ceiling on the factor, not a measured value). The widths and overlaps are made up for illustration; the paper's own head sizes may differ, and the KV-cache count ignores grouped-query attention and other cache-shrinking tricks.

On the paper's own benchmarks, the less invasive write helps most where interference piles up. As a pure linear model at 1.5B parameters and 100B training tokens, KronLSTM reports a six-task average of 59.72 against 58.12 for the strongest competitor, KDA (+1.6 points), and a LAMBADA perplexity of 11.72 against 12.58 for Gated DeltaNet (Table 1). It also leads most needle-in-a-haystack retrieval tasks at longer contexts, which fits the interference argument. Read it with care: this is one paper, one model size, and the authors' own evaluation. The exception is the most repetitive needle task (S1), where KronLSTM underperforms, which the authors attribute to residual accumulation for recurring keys. The paper's appendix also notes that DeltaNet has the smaller norm bound, so the Kronecker rule trades a looser bound for gentler writes.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

Complex KDA extends Kimi Delta Attention — the delta-rule family this paper departs from, seen through its gates
HydraHead mixes full and linear attention per head — what a hybrid of the two attention types looks like

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based