Audit finds a typical DeepSeek-V4-Flash site uses about two of its four residual streams — Near-identity late residual mixing — What does it mean?

The news. On September 4, 2026, researchers posted an audit of DeepSeek-V4-Flash's four-stream mHC residual pathway. Rather than proposing a new architecture, it asks how broadly blocks read and write, how strongly the pathway mixes the streams, and whether the streams stay distinct — then tests each answer by intervention. A typical attention or FFN site effectively uses about two of the four streams, and replacing the mixers in layers 22–42 with the identity raises C4 perplexity by only 1.9% while preserving the six-task average score. Read the paper →

Picture the residual pathway as a highway running the length of the model. In a standard transformer it has one lane: every block merges onto it, reads what is already there, and adds its own contribution back before the next block arrives. That single lane is the running state, and you can watch a block read it and add to it in Residual Connections.

Hyper-Connections widen that highway. DeepSeek-V4-Flash carries four lanes instead of one, so a block can write into one lane and leave the other three exactly as it found them. Between layers there is also an interchange: a small mixer that blends the four lanes into each other, so something written into lane 1 can reach lane 3 further down the road. The extra lanes and the interchanges are capacity the model is free to use, not capacity it has to use.

This paper drives the whole highway and counts what is actually on it.

Three measurements, all taken on the trained model rather than on its architecture diagram. Effective stream count asks how many of the four lanes a site really reads from or writes to, and at a typical attention or feed-forward (FFN) site the answer is about two. Cross-stream residual weights ask how strongly the interchanges blend lanes, and the answer is that mixing is modest overall and concentrated in the early layers. Inter-stream cosine similarity asks whether the four lanes drift into carrying the same thing; they do not, because the representations stay directionally distinct.

Depth changes the picture in a way a single average would hide. The dominant lane at one site is not the dominant lane at the next, so the model does spread work across the four streams as you move down the stack — it simply does not need all four at any one site, much as a multi-head attention layer runs many heads without every head mattering equally. And from layer 22 to layer 42 the pathway mostly carries each lane forward separately, with little blending left to do.

A measured quantity can be small and still be load-bearing, so the paper does not stop at the numbers. The test is substitution: replace a component with a pass-through, then see what breaks. Replacing a mixer with the identity is exactly that — the lanes still run, they simply stop exchanging traffic at that point.

Run the substitution late and almost nothing happens. Run it on the early mixers instead and perplexity rises 41% — those mixers stay load-bearing. That asymmetry, rather than the size of the learned weights, is what establishes which mixers matter.

There is a subtler version of the same test. Instead of deleting an early mixer, freeze it: fix it at its average value over the C4 corpus so it still blends lanes, but blends them the same way for every token. That barely hurts, which separates two things the raw weights cannot — the mixer's site-specific structure matters, while its token-wise variation matters much less on the evaluated metrics.

Hold the model and the kind of intervention fixed, and compare the two depth ranges the paper reports.

The intervention is the same in both cases: take the mixers in one depth range and replace them with the identity, so the four lanes stop exchanging traffic there. Applied to layers 22–42, C4 perplexity rises 1.9%, with the six-task average score preserved. Applied to the early mixers, perplexity rises 41%.

Divide one by the other: 41 ÷ 1.9 ≈ 22. The reported perplexity cost is about 22 times larger for the early mixers than for layers 22–42. Read that as a comparison of two reported percentages, not as a per-mixer price — the abstract does not say the two ranges hold equally many mixers. The routing result points the same way from a different angle: keeping only the three largest routing weights per token at every site, out of four, costs at most 2.7% perplexity and at most 0.4 points of average score. Each intervention was measured on its own, and the paper's own summary of the set is that the studied model realises only part of the flexibility that four-stream mHC offers.

The habit worth taking from this is not that four streams are too many. It is that capacity added is not capacity used, and the cheap way to tell the two apart is to swap something for a pass-through and re-measure. How cleanly that transfers depends on the component — a residual mixer has an obvious pass-through in the identity, while an attention head or an expert route needs a substitute chosen with more care — but the question is the same one, and it is the question a headline benchmark score never answers on its own. It is also the question the modern architectural variants step is for.

Be careful how far the verdict travels. This is one model, one family of measurements, and one corpus behind the perplexity number. The paper reports what DeepSeek-V4-Flash does with its four streams, not what every mHC model must do — a model trained differently, or evaluated on tasks that stress long-range mixing harder than C4 and a six-task average do, could lean on the late interchanges more. What travels is the method, not the result.

Goes deeper in: LLM Internals → Transformer Block → Residual Connections

Related explainers

• Tencent open-sources Hy4 Preview at 770B — Identity Hyper-Connections — what a four-stream residual pathway is, and why an architecture would widen one in the first place.
• Variable-width transformers — the hourglass shape — the other way to question a stack's uniform capacity: vary it by depth instead of by stream.