Complex KDA extends Kimi Delta Attention — Signed gates for state tracking — What does it mean?

The news. On September 21, 2026, researchers from the University of Freiburg, EPFL, Tübingen, Microsoft Research and others posted Complex KDA (arXiv 2609.24797). It keeps Kimi Delta Attention's update shape — a diagonal gate times one identity-minus-rank-one matrix — and changes only two ranges: gate entries may now be negative, down to -1, and the delta-rule strength beta may reach 2. The paper proves this lets a single update realize any 2D rotation, and that — under its exact-arithmetic assumptions — one CKDA layer can track every finite group of 3D rotations, matching DeltaProduct₂ with one update per token instead of two. Read the paper →

Picture the layer's memory as a clock hand it carries from token to token. Each new token is allowed to do one thing to the hand before passing it on. In standard KDA that thing is a bank of dimmers: every gate entry sits between 0 and 1, so each channel can keep its part of the hand or shorten it, and the delta rule can wipe one direction and write a new value there. Those moves can nudge the hand's direction, but under the standard ranges they cannot carry it around the dial and back without loss. A memory limited to fading and overwriting has no way to go around a cycle and come back intact.

That matters because a large family of tasks is going around a cycle. Parity flips back and forth. A count modulo 4 visits four positions and returns to the first. Following which card is where after a series of swaps is composing permutations. The paper frames the ladder plainly: a sign flip is enough for parity, a 2D rotation for modular counting, and higher-dimensional rotations for permutations. None of these is exotic; they are what "keep track of where things are" means once the input gets long.

The fix is an old fact about mirrors: two reflections, one after another, make a rotation — by twice the angle between the mirrors. The delta rule already contains one mirror. When beta reaches 2, I - 2 k k^T is a Householder reflection whose angle is set by the token's key — the same key that decides where attention writes and reads. CKDA supplies the second mirror from the gate: a gate entry of -1 flips one coordinate, which is a reflection along a fixed axis. Put the two together in one update and the hand turns.

Why does the per-channel gate matter, and not just the sign? Gated DeltaNet's gate is one scalar for the whole state, and a scalar commutes with everything: across many steps all the scalar gates factor out into one overall scale, leaving only the chain of delta-rule steps. Even a signed scalar just adds a global plus or minus — it can never act as a second, differently angled mirror. KDA's per-channel gate can, because different channels can get different signs. The authors show this is the missing ingredient: a signed per-channel gate plus a full-strength delta rule is enough to reach every length-preserving (orthogonal) transition of this cheap diagonal-plus-rank-one shape — the layer keeps its structure and gains rotations.

The two range extensions have to arrive together. With both ranges in the paper's tests, one-layer models extrapolated well on S₃ and S₄ — the shuffles of 3 and 4 items — past the training length of 32, though S₄ accuracy still declined at the longest lengths. Extending only the gate or only beta left long-length S₃ accuracy near 0.2 on the paper's chance-to-perfect scale — about what you get from tracking parity and nothing more, which is exactly what a single mirror buys. When a trained CKDA head did succeed, it had learned the predicted recipe on its own: beta close to 2, gates close to plus or minus 1, and complex eigenvalues near the unit circle.

Put numbers on the clock (illustrative task, real geometry). The job: read a stream of "+1" tokens and report the count modulo 4. Set the key so the two mirrors sit 45° apart; every token then turns the hand by 2 × 45° = 90°. After 4 tokens the hand is back where it started. After 1,003 tokens — 250 full laps plus 3 steps — it points at 270°, and the answer can be read straight off the dial: 3. The hand is also still full length, because a rotation's eigenvalues sit on the unit circle and nothing is shrunk.

Now try the same with a decay-only gate. Suppose the layer stores the count in a direction that decays at a gate of 0.9 per token (illustrative): after 1,003 tokens that direction is scaled by 0.9¹⁰⁰³, roughly 10^-46. Holding the gate at 1 keeps the hand but never moves it. No transition whose eigenvalues are all real and between 0 and 1 — which is every setting of standard KDA's gates and beta — returns the hand to its start after exactly four steps without leaving it still on every step — returning after four steps is the one thing a mod-4 counter has to do, and it takes an eigenvalue that is a genuine quarter-turn.

What does the extra power cost? Less than you might expect, and the paper is careful about what it claims. The signed gate needs sign handling on top of KDA's log-space gate magnitudes, and the authors' kernels keep about 96–97% of KDA's throughput on an H100. DeltaProduct₂, the earlier route to rotations, runs a second delta-rule update per token; the authors say they do not claim an efficiency win over it, only an alternative way to reach similar expressivity. In 1.3B-parameter language models trained on 100B tokens, CKDA scored on par with KDA (average zero-shot accuracy 54.06 against 54.09 for the authors' KDA baseline). The gain shows up where a cycle matters, not on everyday text: on a periodic-audio continuation task, CKDA held 38.1 dB of signal-to-noise at length 264, beyond its training maximum of 136, where a causal Transformer fell to 2.8 dB — though a nonlinear GRU was more accurate still.

There is a ceiling, too. The authors prove one CKDA layer cannot track S₅, the permutations of five items, under their assumptions that each update never stretches the state and the state takes finitely many values; that needs more layers, and three CKDA layers solve every finite-group word problem under the same exact-arithmetic convention. The durable lesson is the geometry, not the benchmark: a fixed-size memory can only do what its transition's eigenvalues allow — for updates like these that never stretch the state, real eigenvalues between 0 and 1 can only hold or fade, while complex ones on the unit circle turn without loss — and that is worth checking in any recurrent layer that claims to replace a KV cache.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

• Gated DeltaNet-2 — Decoupled erase and write gates — the other recent change to the delta-rule gate, aimed at retrieval rather than state tracking.
• Gated DeltaNet — A matmul-only inverse — how these delta-rule layers are actually computed in chunks on a GPU.
• ARMT — Recurrent associative memory — another fixed-size memory that replaces a growing KV cache.