UpLoop is a pretraining schedule that starts with a shallow Transformer, grows its depth by copying learned layers, and in the last stage runs a shared middle block more than once. It keeps the dense baseline's final parameter count, training tokens and training FLOPs.

Why is UpLoop cheaper than training a looped model normally?

A normal looped model pays the full cost of every extra pass through the shared block. UpLoop spends its early tokens on shallow models that cost fewer FLOPs per token, and that saved compute funds the repeated passes in the final stage.

How does UpLoop relate to ordinary depth growth?

Depth growth alone, as in the LIDAS baseline, copies layers to train cheaply and spends the savings on more tokens. UpLoop spends the savings on looping instead, and at 370M it reached NLL 2.74 against 2.75 for LIDAS at matched FLOPs.

Does a UpLoop model cost less to run?

No. It stores the same number of parameters as the dense model but executes more layers per token, so inference takes more compute. The equal-cost claim applies to pretraining.

LLM· 2026-11-03

UpLoop trains looped Transformers at no extra pretraining compute — Upcycling a shallow Transformer into a looped model — What does it mean?

Jargon

Looped Transformer: A Transformer that gets more execution depth by running the same block of layers more than once with shared weights, instead of stacking new layers with new weights. See SMELT for the matched-budget version of the idea.
Execution depth: How many layer-passes one token goes through. A looped model can have 16 distinct layers and an execution depth of 20, because four of them run twice.
Upcycling (depth growth): Starting from a smaller trained model and building a bigger one out of it by copying layers, so the copied layers begin with what the smaller model already learned instead of random weights.
FLOPs: Floating-point operations, a count of arithmetic. Training FLOPs per token grow roughly in step with how many layers a token passes through, so a shallow model is cheap to train per token.
Adam state: The two running averages (momentum and variance) the Adam optimizer keeps for every weight. UpLoop copies these along with the weights when it copies a layer, so the copy does not restart its optimizer history.
Dense baseline: The ordinary Transformer UpLoop is compared against: every layer has its own weights and every token passes through each layer exactly once.
NLL (negative log-likelihood): The language-modeling loss measured on held-out text: how surprised the model is by the real next tokens. Lower is better.

The news. On October 30, 2026, researchers posted "UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute" to arXiv. The paper asks whether a looped language model can beat a standard dense model that has the same final parameter count, the same training tokens and the same training FLOPs. At 1.3B parameters, its average score over 22 downstream tasks rose from 22.99 to 27.52. Read the paper →

Picture a runner who has to log exactly the same total mileage as a rival, but the coach wants the last workouts to be on a longer course. The rival runs the same 16-stretch track every day; that is the dense model, where every token passes through every layer once. UpLoop's coach starts the runner on a short track, lengthens it by copying stretches, and only in the final weeks sends the runner around the middle of the course twice. The early short laps are cheap, and the miles they save are spent later on the repeat laps.

The paper makes this a four-stage schedule. Stage one trains a shallow Transformer. Stage two stacks its layers twice, and each copy inherits both its weights and its Adam state, then all layers are trained independently. Stage three grows the model to full depth by keeping the first and last layers and copying the middle block. In stage four the model picks a run of consecutive middle layers as a shared block and executes it more than once, while the head and tail each run once. That last switch keeps every parameter and optimizer state and only raises execution depth.

Stage 1: shallowfew layers, cheap tokens

Stage 2: stack twicecopy weights + Adam state

Stage 3: grow the middlehead and tail kept

Stage 4: loop the middleshared weights, run again

Why can the same block be run twice at all? Each layer writes its result back onto the residual stream, so a second pass over the same layers receives a state the first pass already changed. Looping reuses weights; it does not reuse the arithmetic. Every extra pass still costs a full forward and backward computation, which is exactly why a looped model is normally more expensive to train than its dense twin.

The paper's accounting makes the trade explicit. At a fixed width and sequence length, training FLOPs per token are affine in execution depth: a cost per layer times the depth, plus a small depth-independent part. Choosing the number of tokens per stage so that the shallow-stage savings exactly offset the looping overhead leaves total tokens and total FLOPs identical to the dense baseline. The learning-rate schedule runs on cumulative tokens and never restarts at a stage boundary, so a stage switch changes the architecture and not the learning rate. The peak learning rates were also left untuned for UpLoop, so the baseline keeps the recipe it was tuned for.

Method (370M scale)	Parameters	Budget matched to UpLoop	Held-out NLL
Vanilla dense	371M	Tokens and FLOPs	2.78
Loop from scratch	371M	Tokens only; the paper reports more FLOPs than UpLoop	2.76
Loop with dense initialization	371M	Tokens only; the paper reports more FLOPs than UpLoop	2.76
Dense-20 (iso-FLOPs)	438M	FLOPs	2.76
Dense-20 (iso-tokens)	438M	Tokens	2.74
LIDAS, growth without looping (iso-FLOPs)	371M	FLOPs	2.75
UpLoop	371M	Tokens and FLOPs	2.74

Read the table by what each row gives up. The dense model with 20 independent layers reaches the same 2.74, but it needs 438M parameters instead of 371M. The growth-only baseline spends the same FLOPs and lands at 2.75. UpLoop is the only row that gets 2.74 with 371M parameters and the dense model's exact token and FLOP budget. Downstream, the average over 22 tasks rose from 18.95 to 20.90 at 370M, from 21.38 to 24.82 at 760M, and from 22.99 to 27.52 at 1.3B. After supervised fine-tuning on GSM8K at 1.3B, exact match went from 8.72 to 12.05 zero-shot and from 5.46 to 8.11 five-shot.

Here is the budget idea with round numbers (illustrative; the paper's own stage lengths are given in its appendix, not here). Take a dense model with 16 layers trained on 100B tokens, which costs 16 × 100B = 1,600 layer-token units. An UpLoop-style schedule trains 10B tokens at 6 layers, 15B at 12, 35B at 16, and 40B at 20 execution layers (16 distinct layers with a 4-layer block run twice). The cost is 60 + 180 + 560 + 800 = 1,600 units, the same. Equal bill, but 40% of the training tokens saw 20 layers of depth instead of 16. The real accounting also carries the small depth-independent term, which the paper's integer schedules satisfy exactly.

Three limits are worth keeping in view. The experiments cover 370M to 1.3B non-embedding parameters on FineWeb-Edu at 100 tokens per parameter, and the appendix describes a single fixed seed per run, so the gaps are modest-size results at small scale. The 1.3B zero-shot gains are uneven: UpLoop beat Vanilla on five of seven eligible commonsense tasks. And the parameter and training-FLOP match does not extend to inference: the model still executes more layers per token than the dense baseline, so it costs more compute to run even though it stores the same weights.

Goes deeper in: LLM Internals → The Transformer Block → The Transformer Block

Related explainers

SMELT loops the middle half of an MoE transformer twice — the same reuse-the-middle idea, judged at matched budgets in a sparse model
LoopCoder-v2 — how many passes through a shared block are worth running
Tuned layer dropout for depth-elastic Transformers — another way to cut training FLOPs by changing how many layers a token sees

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based