What is rank preservation in a Transformer?

Rank is the number of independent directions a signal still carries. Rank preservation is keeping that number from falling as you stack layers. It matters because a Transformer's residual branch — a matrix multiply followed by a nonlinear activation — is rank-reducing, so depth compounds the loss until very few independent directions are left. The "Transforming Rank" paper argues that skip connections and normalization placement are the architecture's rank-preservation machinery, and that the rank of the input-output Jacobian at initialization predicts which networks train on CIFAR-10.

Why does rank collapse under Post-Norm but plateau under Pre-Norm?

Because the two placements route the signal differently. Post-Norm normalizes after the residual add, so every block renormalizes the sum and no clean identity path carries from one block to the next. Pre-Norm normalizes inside the branch, so the skip passes the identity on untouched and each block only adds a scaled branch output to it. Both still have a skip and a rank-losing branch; what moves is the normalization. The paper's framing is that normalization placement sets the branch-to-skip ratio across depth, and that ratio is what decides whether rank collapses or plateaus.

Aren't skip connections just there to fix vanishing gradients?

That is the magnitude story, and it is not wrong — it is incomplete. This paper recasts skips as routing the gradient around the residual branch, which is exactly where rank is lost, rather than along the long gradient paths that make layers compose. That reframing exposes a tradeoff the magnitude account hides: relying on skips avoids rank collapse but pushes the network toward ensemble-like behavior, an average of shallow paths instead of one deep composed function. The paper recasts architecture design as navigating rank collapse, ensemble-like behavior, and parameter count together.

Transformer rank study reframes Pre-Norm as gradient preservation — Rank preservation vs magnitude control

TL;DR

What is it: The news anchor is “Transforming Rank”, a theory paper by Katie Everett on how each part of the Transformer feed-forward block decides how much rank survives across depth at initialization; the concept it makes concrete is rank preservation — reading skip connections and normalization placement as the machinery that keeps a deep network’s gradient rank from collapsing.
Why it’s needed: Rank is how many independent directions a signal still carries, and the paper reports that the rank of the input–output Jacobian at initialization predicts which networks train on CIFAR-10 — making it a property you can check before spending a single training step.
vs previous: The textbook story is that skips and LayerNorm are there to control magnitude — to stop gradients exploding or vanishing — but this paper recasts them as rank-preservation mechanisms, which is what explains why rank collapses under Post-Norm yet plateaus under Pre-Norm.

Jargon

Rank: The number of independent directions a matrix (or a signal) actually spans. A width-1024 vector has room for 1024 directions, but if every token ends up pointing the same way, the useful rank is 1 — the width is still there, the variety is gone.
Rank collapse: What happens when rank falls as you stack more layers, until very few independent directions are left. The paper’s framing: matrix multiplies and nonlinear activations are rank-reducing, so depth compounds the loss. It analyses this at initialization, and treats gradient rank and the residual representation as separate things the architecture has to protect.
Jacobian: The matrix of partial derivatives that says how the output moves when each input moves — in practice, the thing gradients flow through. Its rank measures how many independent directions of feedback can reach the earlier layers.
Residual branch vs skip: A block computes x + f(x). The branch is f — the attention or feed-forward sublayer where rank is lost. The skip is the bare x term routing straight past it.
Branch-to-skip ratio: The relative scale of the branch output against the skip. A big ratio means the block leans on the rank-losing branch; a small ratio means the clean skip dominates. The paper’s claim is that where you put the normalization is what sets this ratio across depth.
Pre-Norm vs Post-Norm: Two placements for LayerNorm. Pre-Norm normalizes inside the branch (x + f(norm(x))), leaving the skip untouched. Post-Norm normalizes after the add (norm(x + f(x))), so the sum is renormalized every block and no clean identity path carries across depth. Compare them step by step →
Ensemble-like behavior: What a network drifts toward when the skip dominates: the layers stop composing into one deep function and start acting like an average of many shallow paths. The paper frames this as the price of leaning on skips — the other side of avoiding rank collapse.
Marchenko–Pastur law: A result from random matrix theory describing the spectrum of large random matrices. The paper ties the feed-forward width expansion needed to keep the branch Jacobian full rank to this law — it is what sets how wide the expansion has to be.

The news. On July 15, 2026, Katie Everett posted “Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth” — a theory paper asking how each component of the Transformer feed-forward block determines how much rank survives across depth at initialization. Its central move is a reinterpretation: skip connections and normalization, “long understood as controlling magnitude”, are recast as mechanisms for preserving gradient rank. The paper argues this unifies much of the normalization-placement and depth-scaling literature, and reports that the initialization rank of the input–output Jacobian predicts which networks train on CIFAR-10. Read the paper →

Picture a line of buckets and a stirrer. You start with three clean pigments — pink, green, blue — and at each station you tip the paint through the stirrer and pour it into the next bucket. Stirring is the whole point: it is the only way to make a color you did not start with. But every stir also destroys something — once pink and green are blended, you cannot get them back out. Run that line long enough and the far bucket is mud: not a color you chose, just the one place all mixing eventually lands. The trap is that the stirring is simultaneously what makes the paint interesting and what destroys the variety you need to keep making it interesting.

That trap is the paper's subject, and swapping the paint for a tensor changes almost nothing. The rank of a signal is how many independent directions it still carries — how many distinct hues remain in the bucket. A residual branch is the stirrer: a matrix multiply followed by a nonlinear activation. Both operations are rank-reducing, and they are exactly the operations that make the network expressive. Everett's framing is that this is not an incidental defect to be patched but the intrinsic tension of depth: "the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank." Stack enough blocks and there are few independent directions left to work with — the bucket has gone to mud.

The move the paper makes is to notice that the standard fixes were already solving this, and nobody said so. Skip connections are taught as a remedy for vanishing gradients — a magnitude problem. Recast as rank, they do something sharper: the skip routes the gradient around the branch, which is precisely where rank is lost. It is the clean pipe past the stirrer. And that reframing carries a cost the magnitude story hides: lean too hard on the skip and the layers stop composing into one deep function, drifting instead toward ensemble-like behavior — an average of many shallow paths. Stir too much and you get mud; never stir and you have only your original pigments. Skip connections do not remove the tradeoff, they give you a dial on it.

That dial has a name, and it is where the normalization sits. Write the two blocks out and the whole difference is one set of parentheses. Post-Norm computes norm(x + f(x)): the normalization lands after the residual add, so every block renormalizes the sum and no clean identity path survives from one block to the next — the bucket is re-blended at every station, original pigment and all. Pre-Norm computes x + f(norm(x)): the norm sits inside the branch, so the skip carries x onward untouched and each block only tints it with a splash. Both variants still have a skip and both still have a rank-losing branch; what moves is the normalization. The paper's unifying claim is that this placement is exactly what sets the branch-to-skip ratio across depth, and that this ratio is what separates a rank that collapses (Post-Norm) from one that plateaus (Pre-Norm) — a reading it says unifies much of the normalization-placement and depth-scaling literature.

Across depth, at initialization (the paper’s scope)	Post-Norm	Pre-Norm
Where the norm sits	`norm(x + f(x))` — after the add	`x + f(norm(x))` — inside the branch
What the skip carries across depth	the sum, renormalized every block — no clean identity path survives	the identity `x`, passed on untouched
What the placement controls	the branch-to-skip ratio across depth — the paper does not quantify the ratio in either direction
What happens to rank	collapses	plateaus

Work a signal down the line to see why depth is what makes this bite. Take a width-1024 residual stream and suppose each block ends up retaining about 85% of its directions — a plausible stand-in, not a figure from the paper, which quantifies no such rate. When no clean copy is carried forward, the losses compound rather than subtract: 1024 → ~870 after 1 block, ~279 after 8, ~76 after 16, and roughly 6 directions after 32 (illustrative — compounding 0.85 per block). A per-block loss small enough to look harmless is a geometric decay, and 32 blocks of it turns a 1024-dimensional stream into a handful of directions. A surviving identity path changes that arithmetic without touching the branch: the skip hands x on untouched, so each block’s loss falls on what the branch adds rather than on everything accumulated before it — and the curve flattens toward a plateau instead of racing to zero. The branch is equally lossy in both; what differs is whether the loss compounds.

The same lens explains the part of the block that usually gets waved through as "make it wider." The feed-forward network is two matrices that expand the width and then contract it, and the paper reads both jobs as rank jobs: the width expansion keeps the branch Jacobian full rank, because applying a rank-reducing activation in the expanded space still leaves enough directions to span the original — at a width the paper ties to the Marchenko–Pastur law. The second matrix decorrelates a coherent mean spike that would otherwise grow across blocks with a single matrix and an uncentered activation. On this reading the expansion ratio is less a capacity knob than the price of pushing a signal through a nonlinearity without losing the directions.

Goes deeper in: LLM Internals → Transformer Block → Pre-Norm vs Post-Norm

Related explainers

Variable-width transformers — Hourglass layer width — the other side of the FFN width question: what happens to cost when you stop holding the expansion ratio fixed
SigmaScale — Learned SVD scaling — another paper reading a network through its spectrum rather than its magnitudes
Taylor-Calibrate — Taylor-guided gate initialization — a companion case that initialization, not just training, decides what a deep stack can learn

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based