What is looped depth reuse?

It is getting extra depth out of a transformer by running the same layers more than once, instead of stacking new layers with new weights. SMELT's version runs the middle half of the stack twice and leaves the layers on either side single-pass.

Why does SMELT match three budgets instead of one?

Because each unmatched budget is a cheaper explanation for the same result. Matching only model size lets the looped model spend extra arithmetic, so its win could just be extra compute. Matching per-token FLOPs, non-embedding parameters and KV cache together removes all three explanations, and the remaining difference is architectural.

How does looping relate to Mixture of Experts?

MoE separates a layer's parameter count from its per-token compute, because only a few experts run per token. That separation is what makes it possible to build fewer distinct layers and still carry the same total parameters as a deeper baseline, which is exactly the trade a looped stack needs.

Does looping change how attention behaves?

The paper reports that it does: the second visit reduces the attention sink and redirects attention mass toward content-relevant tokens. The authors describe this as an inductive bias that may underlie the measured gains, not as a proven cause.

LLM· 2026-09-05

SMELT loops the middle half of an MoE transformer twice — Looped depth reuse — What does it mean?

TL;DR

What is it: SMELT (Sparse MoE Transformer, middle layers Loop Twice) is a September 2026 scaling-law recipe that gets extra depth out of a Mixture-of-Experts transformer by sending each token's hidden state through the middle half of the layers a second time, instead of stacking new layers with new weights.
Why it’s needed: Depth is one of the few reliable ways to make a transformer better, and it normally costs new parameters and a longer stack to build and serve — so a way to buy depth with weights the model already has changes what a fixed training budget can produce.
vs previous: Earlier looped-transformer results compared models at fixed model size, which quietly handed the looped model extra arithmetic; SMELT instead matches per-token FLOPs, non-embedding parameters and KV cache all at once, so the gain cannot be explained away as simply spending more compute.

Jargon

Looped transformer: A transformer that gets more effective depth by running the same block of layers more than once, rather than stacking new layers with new weights. The idea predates SMELT; what is new is testing it under matched budgets.
MoE: Mixture of Experts. A layer whose feed-forward part is split into many separate expert networks, only a few of which run for any given token — so the layer can hold a lot of parameters while spending little compute per token. Dense vs MoE, explained
Non-embedding parameters: Every weight in the model except the token-embedding and output tables. It is the standard basis for comparing architectures, because embedding size tracks vocabulary rather than the network itself.
FLOPs: Floating-point operations, the raw arithmetic a pass costs. Per-token FLOPs is what one token costs to push through the network; training FLOPs is the total spent over a whole training run.
KV cache: The keys and values every attention layer has already computed for the tokens so far, kept so they are not recomputed on each new token. It is a memory cost that grows with layers and sequence length. How the KV cache works
Attention sink: The habit softmax attention has of dumping a large share of its weight on one uninformative position — usually the very first token — when a head has nothing better to attend to.
Compute-optimal frontier: The lowest loss reachable for a given training-compute budget, once model size and training tokens are traded off against each other. Chinchilla-style scaling laws are fits to this curve.

The news. On September 1, 2026, researchers posted "SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers" to arXiv. The paper asks whether a looped transformer is genuinely a better architecture, or whether it only looks better because looping quietly spends more arithmetic. That question only has an answer if the budgets are matched, so the authors closely match three of them at once: per-token FLOPs, total non-embedding parameters, and KV cache. Under that constraint they fit a separate Chinchilla-style scaling law per architecture across four model sizes, up to 54 billion non-embedding parameters, and report that the looped recipe reaches the same loss for 6.8-18.0% fewer training FLOPs on the compute-optimal frontier. Read the paper →

Picture the car wash. A car rolls in, passes the pre-soak bay, then the long scrub bay in the middle, then the dryer, and comes out the far end. If cars are coming out not-quite-clean, the obvious fix is to build another bay: more equipment, more floor space, more money. SMELT's fix is to send the car back through the middle bay a second time. The equipment is the same equipment; the car simply goes over it twice.

In the model, the car is one token's hidden state and each bay is a transformer layer — an attention part and a feed-forward part, both writing their result back into the residual stream that carries the token through the network. SMELT splits the stack in three. The layers before the middle run once, the middle half runs twice with the same weights, and the layers after it run once. A stack laid out that way puts every token through half again as many layer-visits as it has distinct layers. Nothing new was built; the middle bay just ran again.

The Mixture-of-Experts part is what makes the budget matching possible. In an MoE layer most of the weights sit in expert feed-forward networks and only a few experts run per token, so a layer's parameter count and its per-token compute are two separate dials rather than one. The paper does not publish the per-configuration layout, but that separation is what lets an architecture with fewer distinct layers still carry the same total non-embedding parameters as a deeper baseline.

Early layersvisited once

Middle halfvisit 1

Middle halfvisit 2, same weights

Late layersvisited once

The budget matching is the paper's real contribution, not the loop. Looped transformers are an old idea and they have looked good before — but usually in comparisons held at fixed model size, where running a block twice simply spends twice the arithmetic on it. A model handed more FLOPs beating one handed fewer tells you very little. Pinning per-token FLOPs, non-embedding parameters and KV cache together removes the three cheapest explanations at once, and what is left is much closer to the architecture itself. Across four sizes up to 54 billion non-embedding parameters, each fitted with its own scaling law, the looped curve falls faster: 6.8-18.0% fewer training FLOPs to reach the same loss on the compute-optimal frontier, with the largest reported gains on code, and an advantage that grows with sample length and with the number of in-context examples.

The mechanistic finding is the part most likely to outlive the recipe. Softmax has to put its weight somewhere, so when an attention head has nothing informative to look at it dumps that weight on one uninformative position — usually the first token. That is the attention sink, and it is wasted capacity. The authors report that the second visit reduces the attention sink and redirects mass toward content-relevant tokens. One reading, which the paper does not establish, is that the first pass leaves behind a summary of the sequence that the layer did not have the first time round, so on the second visit there is finally something worth attending to. The authors themselves call it an inductive bias — a built-in leaning that shapes what the network learns — that may underlie the gains, and they present it as a hypothesis, not a demonstrated cause.

Budget	Unlooped baseline	SMELT (middle layers loop twice)
Distinct layers built	~48 (illustrative)	~32 (illustrative)
Layer-visits per token	~48 (illustrative)	~48 = 16 outer + 16 middle x 2 (illustrative)
Per-token FLOPs	baseline	closely matched (SMELT)
Non-embedding parameters	baseline	closely matched (SMELT)
KV cache	baseline	closely matched (SMELT)
Training FLOPs to reach the same loss	baseline	6.8-18.0% fewer, compute-optimal frontier (SMELT)

Hold the three budgets fixed and count layer-visits. Take an unlooped baseline with 48 layers — illustrative, since the paper does not publish per-configuration layer counts. Every token passes through all 48, so the per-token arithmetic is 48 layers' worth. Now build the SMELT version with 32 distinct layers: the middle half, 16 of them, runs twice, and the 16 outer layers run once. Layer-visits per token = 16 + (16 x 2) = 48 — identical arithmetic, so the FLOP budget matches. But the weights that have to exist are 32 layers' worth, not 48, and the paper holds the KV cache matched too, so the loop is not buying its depth with extra cache memory either. Something has to absorb the parameters those 16 missing layers would have held, and in an MoE stack the experts are the obvious place; the paper reports only that the totals match. The trade is 16 blocks you never build, in exchange for walking the same 16 blocks twice — and on the measured frontier that is worth 6.8-18.0% of the training FLOPs needed to reach a given loss.

Goes deeper in: LLM Internals → The Transformer Block → Modern Variants & Scale

Related explainers

Variable-width (hourglass) transformers — the other axis: vary the width per layer instead of reusing the layers
Active vs total parameters — the sparse-activation split that lets an MoE stack hold parameters fixed while its layer count moves
Dense vs MoE — start here if expert routing was new above

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based