Multi-Branch Transformer paper — Parallel branches and Joint-KV — What does it mean?
The news. On November 15, 2026, researchers from the University of Illinois Urbana-Champaign, Rice, Michigan and Cornell posted the Multi-Branch Transformer paper (arXiv 2611.19800). Across four mixture-of-experts model scales, named W1 (smallest) to W4 (largest) and spanning 0.8B to 2.1B total parameters, the authors report that standard transformers needed up to 1.26× as much training compute in the transformer body (the blocks, without the output head) to match their final models' validation loss. The Joint-KV versions cut full-model KV-cache storage by 19% to 34% and decoded up to 51.4% faster on one H200 GPU (64K-token inputs, 8 concurrent requests), using an inference engine the authors built on nano-vLLM. Code and checkpoints are public. Read the paper →
Picture a kitchen line. In a standard transformer, every dish (one token's hidden state) passes through a single row of cook stations, and each station must finish before the next one starts. That row is the stack of transformer blocks, each one an attention step plus a feed-forward step joined by residual connections. The Multi-Branch Transformer keeps shared prep stations at the start and shared finishing stations at the end, but splits the middle of the row into several cook lines that work on the same dish at the same time. The paper calls these the prelude, the branches and the coda. Each branch is an ordinary stack of full blocks with its own weights. Where the lines meet, the fusion step normalizes each branch's output, concatenates them, and projects the result back to the model's width with one linear map; that is the pass where the cooks' work becomes one plate.
Splitting the line raises a budget question, because the paper always compares models at matched training compute. Wide–Shallow keeps every station full-size and deals the middle stations out across the branches, so each line is short. Narrow–Deep keeps each line as long as the original middle stack but makes every station smaller. Balanced sits between the two. A 2-branch Wide–Shallow model already matched a standard transformer trained with 1.23× as much compute, and Balanced scored best of the three. The ablations answer the obvious objection that this is only a wider network in disguise: Wide–Shallow MBT beat both a globally widened stack and a single wide core of the same depth at all four scales. Placement mattered too: branching the middle layers gave the lowest loss at the W3 scale (2.445), ahead of early (2.458) and late (2.464) branching. This puts branching next to the moves in modern transformer variants, not inside them.
Now the ticket rail. Every station keeps a rail of tickets for all earlier orders, so its cook can look back at them. That rail is the KV cache, and during decoding it is the part that grows with context and is re-read at every step. Naive branching makes the rail problem worse: with Independent-KV, every block in every branch keeps its own cache, so four branches means four caches at each depth. Joint-KV hangs one rail per row of stations instead. At a given depth it concatenates the inputs of all branches, projects them once into a single set of keys and values, and lets each branch keep its own queries. The joint projection has the same parameter count and matrix-multiply FLOPs as the separate ones, so the saving is storage, not extra compute. It is the same pattern as grouped-query attention, applied across branches instead of across heads. And at an equal cache size, sharing even beat private caches: Joint-KV reached 0.011 to 0.012 lower validation loss than a cache-matched Independent-KV model at all four scales.
The diagram above shows grouped-query attention inside one layer: several query heads read one shared key-value head. Joint-KV moves that sharing one level up, so the readers of each shared key-value bank are the branches. The KV cache module's GQA step walks through why fewer key-value heads mean a smaller cache; the same arithmetic explains Joint-KV's savings.
The speed does not come from the architecture alone; it needs an engine that runs a row of parallel stations as one step. Decoding is memory-bound: each generated token walks every serial block and re-reads the KV cache, so step time follows serial depth and KV bytes. The authors' engine adds two changes in the model executor. Branch grouping stacks the same-depth blocks of all branches into one batched stage, so the serial path counts depth, not total blocks. Head folding folds the branch dimension into the query heads, so one grouped-query attention call reads each shared bank once instead of once per branch. A naive engine that ran the branches one at a time was slower than the standard model in every tested setting; only both changes together made the MBT faster everywhere.
Worked example (W3 scale, from the paper's Tables 11 and 15 and Appendix B). Hold the scale at W3 and compare the standard model with the 4-branch Balanced Joint-KV MBT. The standard model runs 16 serial blocks for each decoded token. The MBT has 24 physical blocks, but branch grouping runs them in 12 serial stages, so the fixed per-token path is 16 ÷ 12 ≈ 1.33× shorter. On storage, the standard model's keys and values take 40 KiB per token, and Joint-KV saves 32.5%, which leaves about 27 KiB per token. At a 64K-token (65,536-token) context, that is 2.5 GiB of KV cache per request for the standard model against about 1.69 GiB for the MBT, so 8 such requests need about 20 GiB versus 13.5 GiB. The MBT does not pay for this in quality at this scale: its validation loss was 2.454 against 2.471 for the standard model.
| Model at W4 scale | Serial stages per token | KV cache per token | KV saved vs standard | Validation loss | Source |
|---|---|---|---|---|---|
| Standard transformer | 18 | 49.5 KiB | — | 2.392 | Tables 3, 15 |
| 2-branch Joint-KV MBT | 15 | 37.75 KiB | 23.74% | 2.361 | Tables 3, 15 |
| 3-branch Joint-KV MBT | 15 | 35.25 KiB | 28.79% | 2.366 | Tables 3, 15 |
| 4-branch Joint-KV MBT | 15 | 33.75 KiB | 31.82% | 2.364 | Tables 3, 15 |
The table shows the pattern at the largest scale: more branches keep shrinking the cache while the loss stays below the standard model's. Where the gain comes from depends on the workload. With short inputs and few requests, the shorter serial path does most of the work; with long inputs and many requests, KV reads dominate each step and the smaller, once-read shared cache does. It is not free everywhere: the MBT adds fixed per-step work (the branch input map, the fusion, building the shared bank, head folding), and at the smallest W1 scale a 2-branch model removed only one serial block, so it was slower than the standard model in six of its 48 benchmark settings. All results are at 0.8B to 2.1B total parameters on one H200 with the authors' own engine; whether the gains hold at frontier scale, or inside production engines such as vLLM, is not yet shown.
Goes deeper in: LLM Internals → Transformer Block → Modern Variants & Scale
Related explainers
- Qwen3.8-Flash-Next — Gated Residual stream — widening the residual stream into parallel lanes, instead of running whole blocks in parallel
- DepthGraft — cross-layer KV reconstruction — sharing KV across layers, where Joint-KV shares it across branches at one depth
- SMELT — looped depth reuse — the opposite trade: reuse blocks in sequence to get depth without new weights