What are identity Hyper-Connections in Hy4 Preview?

They are Hy4's replacement for the transformer's single residual connection. Instead of one running stream that every block reads from and adds back into, the model carries four parallel residual streams between layers. The word identity points at an initialization that starts the four as copies of one, so the model begins by behaving like an ordinary single-stream network — though Tencent's model card states only the count, not the mechanism.

Why would a model want more than one residual stream?

Because the residual stream is the only channel every layer shares. In a 78-layer stack, 78 blocks all write into the same vector, so an early layer's contribution has to survive 77 later additions on top of it. Four streams give a block somewhere to put a signal that other blocks are not obliged to write over.

How does this relate to pre-norm and post-norm residuals?

Pre-norm and post-norm change where normalisation sits relative to the residual add; they do not change how many streams there are. Hyper-Connections change the count instead. They are independent choices, and Hy4's model card does not say which normalisation placement it pairs with its four streams.

LLM· 2026-09-07

Tencent open-sources Hy4 Preview at 770B — Identity Hyper-Connections — What does it mean?

TL;DR

What is it: Hy4 Preview — Tencent's Apache-2.0 open-weight model at 770B total parameters, 49B active per token and 78 layers — replaces the transformer's single residual pathway with identity Hyper-Connections, which carry four parallel residual streams between layers instead of one.
Why it’s needed: The residual stream is the one channel every layer both reads from and writes back into, so in a 78-layer stack it is a shared bus that 78 blocks take turns on — and how wide that bus is decides how much of an early layer's work can still be recovered at the top.
vs previous: Where a standard pre-norm residual connection hands each block a single stream to add into, Hy4 hands it four, so a block can write to one and leave the other three intact — and the word "identity" in the name most plausibly marks an initialization that starts the four as copies of one, though Tencent does not spell that out.

Jargon

Residual connection: The wiring that makes a block compute x + f(x) rather than f(x). The block's output is added to what came in, so nothing has to be recomputed from scratch and gradients have a short path back. Full step
Residual stream: The running vector that residual connections pass from layer to layer. Every block reads it, adds something, and hands it on, which makes it the only shared channel in the whole stack.
Hyper-Connections: A family of designs that replace the single residual stream with several parallel ones, so a block can read from and write to them separately instead of summing everything into one channel.
Identity initialization: Starting a new mechanism in a state where it computes exactly what the old mechanism computed, so training begins from known-good behaviour and any difference has to be learned. The standard way to add capacity without destabilising a deep stack.
Pre-norm: Normalising the input to a block rather than the sum after it. The modern default, because it keeps very deep stacks trainable. Pre-norm vs post-norm
Hidden width: The size of the vector a transformer passes from layer to layer — the width of the residual stream. It sets how much a single token's running state costs to carry.
MoE (Mixture-of-Experts): A feed-forward layer split into many sub-networks with only a few active per token. Hy4 holds 770B parameters this way and runs 49B of them per token.

The news. On August 29, 2026, Tencent published Hy4 Preview on Hugging Face under Apache-2.0: a 770B-parameter Mixture-of-Experts model activating 49B parameters per token across 78 layers, with a 1M-token context and BF16 plus FP8 weights. The first layer uses a dense feed-forward network; the other 77 route each token to 8 of 256 experts plus one shared expert. Alongside that routing, the model card lists one architecture change to the block's wiring: identity Hyper-Connections expand the residual pathway to four streams. Read the model card →

Picture the whiteboard in the hallway. Every team on the floor works from it: you read what is already there, add your contribution, and leave it for the next team. That is exactly what a residual connection does — each block reads the running representation, computes something, and adds its result back instead of replacing it. It is the single most load-bearing wire in a transformer, and it is why a 78-layer stack trains at all.

With one board and 78 teams in a row, two things go wrong. The late teams find a board carrying 77 earlier contributions summed on top of each other, and a team that wants to hand something specific to the team three doors down has no way to keep it separate from everything else being written in between. The board is the only channel there is, so every layer competes for the same surface.

Hy4 Preview hangs four boards. The point of four is that a block can, in principle, write to one stream and leave the other three untouched, so an early layer's signal has somewhere to sit that later layers are not obliged to write over. The name is the interesting part: identity Hyper-Connections. If that word means what it most likely means — the four boards start as copies of the single one — then an untrained four-stream network computes exactly what an ordinary one-stream network computes, and the extra width costs nothing until training decides to use it. Tencent does not confirm that reading.

The identity reading deserves a warning label. Tencent's model card gives this mechanism exactly one clause: identity Hyper-Connections expand the residual pathway to four streams. Everything past "four streams" — the identity initialization, how a block chooses what to read from and write to, whether the mixing is learned at all — is a reading of the name, not documentation. There is no published mixing rule, no initialization detail, and no ablation against a single-stream baseline of the same size.

What can be said without guessing is why the count matters. Widening the residual pathway is not free: whatever passes between two layers now passes four times over, so the state carried across each layer boundary is four times as large. That is a different kind of cost from the ones that usually dominate a serving budget, and it is worth separating from them, because it scales with the model's depth and width rather than with how much context is in flight.

Residual design	What passes between two layers	What it costs
Post-norm residual (original Transformer)	one stream, normalised after the block's output is added	very deep stacks are harder to train without a warmup schedule
Pre-norm residual (modern default)	one stream, normalised before the block reads it	stable to train, but all 78 blocks still share one channel
Hyper-Connections, n streams	n parallel streams a block reads from and writes to	n× the state carried across each layer boundary (definitional, not measured)
Hy4: identity Hyper-Connections, n = 4	four streams, starting as copies of one	Tencent publishes the count, not the mixing rule or an ablation

What four streams cost per token

Hold two things fixed: the 78 layers Tencent documents, and one token moving up the stack. In an ordinary transformer the state handed across a layer boundary is one vector of the model's hidden width. At a hidden width of 8,192 in BF16 that is 16 KB per token per boundary; with four streams it is 64 KB — a 4× increase, and 48 KB more per token per boundary. (Illustrative: Tencent does not publish Hy4's hidden width, so treat 8,192 as a stand-in for the shape of the arithmetic, not as a spec.)

The reason that number is tolerable is what it is not. It is per token in flight, not per token in the cache: a 1M-token context does not multiply it, because the residual state exists only for the tokens currently being computed, while the KV cache is the thing that grows with everything you have already read. At a batch of 64 tokens in flight the four streams cost about 4 MB of live activation — noise beside a KV cache measured in tens of gigabytes. Widening the residual pathway is expensive in a currency the serving budget has plenty of.

Goes deeper in: LLM Internals → Transformer Block → Residual Connections

Related explainers

Tencent open-sources Hy4 Preview at 770B — IndexCache cross-layer index reuse — the other architecture change in the same model: the sparse-attention index built once and reused across layers.
SMELT — looped depth reuse — a different answer to the same question of what a stack of layers should carry forward.

Continue in trackLLM Internals — Transformer Block & Residual Connections

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based