LLM·

Latent Lattice Language Models — Per-layer horizontal latents — What does it mean?

The news. On November 16, 2026, Shivam Duggal and colleagues at MIT CSAIL and Adobe posted Latent Lattice Language Models (arXiv 2611.21608). Instead of giving a model more layers, a wider residual stream, or latent reasoning steps, the paper adds one recurrent state per middle layer that persists across tokens and conditions that layer's attention, KV-cache writes and MLP. At about 406M parameters and 20B FineWeb-Edu training tokens, it lowers validation perplexity from 15.41 to 14.40 against a parameter-matched Transformer, and it also improves Gated DeltaNet, Mamba-2 and hybrid backbones. Read the paper →

Picture the tower in a standard Transformer. Every envelope climbs the stairs alone, and a floor remembers earlier envelopes only through the pages in its filing cabinet. On each floor the clerk reads the cabinet (attention), adds the floor's work to the envelope (the residual connection), files a new page of keys and values, and passes the envelope up. A filed page never changes afterwards. The paper's starting observation is that one latent per token does all of the active work: it reads memory, writes memory, runs the feed-forward network, and at the top becomes the next-token prediction. The diagram below shows that single path through one layer.

Inputembeddings
Thecatsatonthemat
Attentioncontext mixing
+ Residualadd & norm
Feed-Forwardtransform
Outputto next layer

The lattice gives each middle floor a clerk with one running note. The clerk updates the note every time an envelope passes, and uses it to decide how the floor handles the next envelope. In model terms, the note uses FiLM to scale and shift the floor's query, key and value vectors right after the Q, K, V projections compute them, with a gain bounded between 0 and 2, and adds a learned shift to the inputs inside the floor's feed-forward network (MLP). Because the keys and values the floor files have already passed through that modulation, the note changes both what the floor reads from its cabinet and what it writes into it. All of these modulation weights start at zero, so a new lattice model begins as a plain Transformer and learns how much to rely on the notes. Only a middle block of layers carries notes; the first and last few layers run unchanged.

A clerk who must read the previous note before writing the next one sounds strictly sequential, which would make training slow. The paper avoids that with an affine, Mamba-style update: the new note keeps a gated fraction of the old note and adds a new contribution, and both the gate and the contribution are computed from the arriving envelope only, never from the old note. That restriction is what lets an associative scan compute every position of a training sequence in parallel. Two additions enrich the note while keeping the scan: input injection hands every clerk the raw token embedding, and a depth-wise readout stream lets each floor's reading of its note depend on the floors below. An optional third addition, which feeds the top floor's note into the bottom floor at the next token, breaks the scan; the paper approximates it with three Jacobi passes (repeating the parallel computation so each pass feeds the next a better estimate), at 1.85 instead of 0.72 GFLOP per token in training.

Hyper-connections hang more envelopes on the same staircase; the lattice adds a second direction of travel. Four parallel residual streams, as in Hy4's identity Hyper-Connections, still move only upward through one token's layers and are discarded when that token is finished. A clerk's note moves sideways, from one token to the next on the same floor. The note is also not the cabinet. In hybrid Transformer–SSM models (an SSM, or state-space model, is a recurrent layer such as Mamba) the recurrent state is sequence memory; in the lattice, attention stays the memory and the note only changes how each floor uses it. The two upgrades stack: with hyper-connections and input residuals already in place, adding the lattice moved perplexity from 14.22 to 13.61.

What the note buys at matched parameters

Hold three things fixed: 20B FineWeb-Edu training tokens, a 2,048-token context, and a target perplexity of 14.40. The baseline 24-layer Transformer with a 1,024-wide residual stream has 406.7M parameters, costs 0.71 GFLOP per token, and reaches 15.41. To get to 14.40 by adding floors, the paper needs 40 layers: 609.1M parameters and 1.11 GFLOP per token, and every token now writes keys and values on 40 floors instead of 24, about 1.7× the KV-cache entries per token. Widening does worse: a 24-layer model at width 1,620 spends 603.3M parameters and reaches only 14.69. The lattice narrows the residual stream to 908 and gives each lattice floor a 256-wide note: 405.6M parameters, 0.72 GFLOP per token, perplexity 14.40. That is the 40-layer model's score with about 33% fewer parameters, about 35% less compute per token, and no added KV-cache entries.

Model (20B tokens, 2,048 context)ParametersGFLOP per tokenFineWeb-Edu perplexitySource
Transformer, 24 layers, width 1,024406.7M0.7115.41Table 1
Lattice, width 908 + 256-wide notes405.6M0.7214.40Table 1
Transformer, 24 layers, width 1,620603.3M1.0414.69Table 1
Transformer, 40 layers, width 1,024609.1M1.1114.40Table 1

What the paper does not show

The recurrent-backbone results point the same way: on Gated DeltaNet, Mamba-2, Triadic GDN and GDN–Mamba-2 hybrids trained on 10B tokens, adding the lattice improved perplexity at every native state size tested, and at the tested state budgets, Lattice GDN beat plain baselines that held 4× as many native-state entries (the backbone's own recurrent memory). The evidence stops at models of a few hundred million parameters and 10B–30B training tokens; whether the gain holds at frontier scale is not tested. One practical consequence follows from the design rather than from the paper: like an SSM state, each request's notes are extra per-sequence state that a server has to keep alongside its KV cache, and a cached prompt prefix would need a saved copy of the notes at that point, not only its KV blocks.

Goes deeper in: LLM Internals → Transformer Block → Residual Connections

Related explainers

Continue in trackLLM Internals — Transformer Block & Residual Connections

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based