Library of Layers — Dynamic layer ordering from a shared block library — What does it mean?

The news. On November 2, 2026, researchers from EPFL, New York University, Columbia University and the University of Maryland posted Library of Layers. They report competitive pretraining results against dense, looped and MoE baselines, plus routing paths that differ by language and by puzzle-solving strategy. Read the paper →

Picture a workshop. A normal Transformer is an assembly line: every job passes the same row of stations in the same order, even if the job only needed a few of them. A looped model is a line that sends every job around the same few stations again and again. A mixture-of-experts model keeps the row of stations fixed, but at each station a job uses only a few of several interchangeable tools. In all three, the sequence of stations is decided when the model is built, not when the work arrives.

LoL works like a workshop with one shared shelf and a foreman. The shelf holds the library of blocks, and at every step of the middle of the model the router picks one block to apply next. Each block is a normal Transformer block: attention, a feed-forward network and a residual connection. Nothing about the blocks changes. What changes is who decides the order: the same B2 can appear twice in a row for one input and never for another.

Letting every token pick its own block would break attention, because a query made by one block would be compared with keys made by a different block. The paper reports that this mismatch makes training unstable. Picking one path for the whole sequence from the first token alone would be too early, since later text changes what the model needs. LoL's answer is to route per window of tokens, using only earlier windows, and to shift the window edges at every depth. All tokens in a window share one block and use causal attention inside that window, so the query-key mismatch never happens, and the shifted edges let information cross from one window to the next. The router itself is a linear layer that reads the average hidden state of earlier windows. The first window has no history, so it uses its first token plus a learned vector for that depth. It picks one block (top-1) and the block's output is scaled by the router probability so gradients reach the router.

The key design choice is the size of the library. One hyperparameter, the library size, moves the model between a looped model and an MoE model. The core depth is how many blocks run; the library size is how many distinct blocks are stored. With a library of 1 block, every depth reuses it. With a library as large as the depth, each block can run about once, like a dense stack. With a library larger than the depth, some blocks are never used for a given input, which is sparse computation.

Here is the arithmetic with the core depth held at 16 block applications, the depth of the paper's MoE comparison. Parameters in the core grow with the library size, and compute grows with the depth. A library of 1 block gives a compute-to-parameter ratio of 16 to 1 (illustrative); a library of 16 gives 1 to 1 (illustrative); the paper's library of 24 gives 16 to 24, about 0.67, with at least 8 of the 24 blocks idle on every path. So the same 16-step computation can look like a heavily reused looped model at one end and a sparse MoE-like model at the other, and only the number of stored blocks changed. The paper trains both regimes on web text and reports LoL competitive with matched looped and MoE baselines. In the MoE comparison (three experts per layer with two active, against 24 experts over 16 routed depths) the MoE baseline reached a slightly lower training loss.

The second result is about what the path means. The authors trained small models on sudoku and mazes where each puzzle is solved with one of five strategies. They extracted each strategy's typical routing path, then at test time removed the strategy hint and forced that path. Forcing the path made the model follow the requested strategy with close to 100% accuracy, even in mazes where all five strategies write their answer in the same format and differ only in the order cells are visited. In multilingual text, windows in different languages took different paths, and languages sharing a script took similar ones. A linear probe on router outputs separated safe from unsafe text with a high AUROC. The paper does not claim the paths are a full explanation of the model, only that they carry readable, steerable signal.

There are limits worth knowing. Results at 1 billion parameters on math data are competitive with dense baselines but not better, and the authors note higher variance across seeds because the router can explore different paths from different initializations. Routing is per window, which gives a coarser training signal than per-token MoE routing. Training also starts with a shallow routed core and grows the depth, so training compute depends on that schedule. What a different path per window does to serving, such as batching and kernel reuse, is a separate question that the experiments summarized here do not address.

Goes deeper in: LLM Internals → Transformer Block → Modern Variants & Scale