ARLoop is a fully open looped Transformer that swaps the usual residual stream for attention over depth. Each sublayer scores earlier residual blocks with a learned query and reads a weighted mix, as described in the ARLoop paper on arXiv (2611.05234).

Why does a looped Transformer need a different residual design?

A looped model reuses the same layers across recurrences, so a plain PreNorm stream gives later passes only limited control over which earlier outputs to reuse. Depth attention adds that choice without adding new layers.

Does sliding-window depth attention work in ordinary Transformers?

The paper reports it does not: with matched effective depth in a non-looped model, the same window degrades performance. It works in looped models because repeated passes of shared layers produce similar, redundant residual blocks.

How does this relate to the residual connection in a standard Transformer block?

A standard block adds every sublayer's output to one stream with weight 1 (see the Residual Connections step in the LLM Internals Transformer Block module). ARLoop keeps that additive form inside a block but replaces the weight-1 sum across blocks with learned attention.

LLM· 2026-11-05

ARLoop — Attention residuals over recurrent depth — What does it mean?

Jargon

Looped Transformer: A Transformer that applies the same stack of layers several times in a row, so it gets more computation (effective depth) without adding new parameters.
Recurrence (R): One full pass through the shared layer stack. The paper tests R from 1 to 5 in controlled runs and R = 3 in its open 0.6B and 1.2B models.
Residual stream: The running hidden state each sublayer adds its output to. See Residual Connections.
PreNorm: The common design that normalizes a sublayer's input and then adds the result back to the stream. Every earlier update reaches later layers with weight 1. See Pre-Norm vs Post-Norm.
Residual block: A small group of consecutive sublayers whose outputs are summed into one saved state. Depth attention picks among these blocks instead of among every single sublayer.
Depth attention: Attention that runs across layers instead of across tokens: a learned query scores earlier residual blocks and mixes them with softmax weights. Compare token attention in Computing Attention Scores.
Sliding-window depth attention: Depth attention restricted to the input embedding plus the W most recent residual blocks, instead of the full history.
Huginn: An earlier fully open looped language model (3.5B parameters) that the paper uses as its main open baseline.

The news. On November 3, 2026, researchers at The Ohio State University and Saarland University released ARLoop, a fully open looped Transformer with code, data processing, checkpoints and 0.6B and 1.2B models. The 1.2B model, pretrained on 0.5T tokens with three recurrences, is reported to outperform the 3.5B-parameter Huginn on every evaluated task. Read the paper →

A looped Transformer is an editor who revises the same manuscript again and again, always using the same skills. The question ARLoop asks is what the editor should keep on the desk between rounds. In a normal Transformer the answer is built in: every sublayer adds its notes to one shared page, and that page is the residual stream. Under PreNorm, each earlier note reaches every later layer with the same weight of 1, so nothing in the stream lets round four weigh round one's scribbles differently from round three's fixes.

ARLoop keeps the notes in separate stacks and lets the editor choose. Each sublayer has one learned query, scores every available stack with a softmax, and reads a weighted mix. This is attention applied along depth instead of along the token axis. The mechanics match attention scores: a query, a score per source, a softmax, a weighted sum. The sources here are residual blocks, not tokens.

Reading everything is wasteful, and the paper finds it is also unnecessary. In a looped model, direct access to the full residual history is not needed. Because the same layers run every round, the notes from different rounds look alike: the paper measures high cosine similarity between residual blocks across recurrences, and the attention weights pile up on the newest blocks. So ARLoop's sliding-window variant keeps the original brief plus only the last W stacks. It matches full-history attention at shallow recurrence and beats it in perplexity at deeper recurrence, and the paper reports the same window hurts a standard non-looped Transformer of matched depth.

Residual design	What a sublayer reads	Source
PreNorm (standard)	Every earlier update, summed with weight 1	ARLoop paper, §3.1
Full-history depth attention	All residual blocks and the embedding, with learned softmax weights	ARLoop paper, §3.2
Sliding-window depth attention (ARLoop)	The embedding plus the W most recent blocks, with learned weights	ARLoop paper, §3.2

Here is the bookkeeping with named numbers (illustrative, not the paper's configuration). Take 4 recurrences of 12 sublayers each, which is 48 sublayers of effective depth, grouped into residual blocks of 6 sublayers. At the last sublayer, full-history depth attention can see the embedding, 7 finished blocks and the partial current block, which is 9 sources. A window of W = 3 keeps the embedding plus 3 recent blocks, which is 4 sources. With a hidden size of 1,024 in bf16, one source costs 2 KB per token, so the saved state falls from 18 KB to 8 KB per token, about 56% less. The paper's own measurement points the same way: in its 140M model with R = 4, the three newest blocks and the embedding receive 96% of the attention weight at sublayer 47.

The payoff is measured on generation tasks, not only perplexity. At matched size, data and recurrence (0.6B parameters, 180B tokens, R = 3), ARLoop-0.6B beats the PreNorm looped baseline on 7 of 8 tasks, including +5.2 points on GSM8K and +3.1 on MBPP. The paper notes that perplexity gains in the controlled runs were modest, so perplexity alone understates the effect.

Goes deeper in: LLM Internals → Transformer Block → Residual Connections

Related explainers

BeNice shares one KV cache across a looped model's recurrences — the same redundancy across recurrences, exploited in the KV cache instead of the residual path.
ResidualQuant stores looped-Transformer KV caches as 2-bit residuals — a second memory-saving use of how similar the loops are.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based