Skip MoE channels for up to 2.69× faster MoE blocks — top-k channel selection — What does it mean?

The news. In late October 2026, a paper argued that MoE channel selection needs no calibration data. Its weight-only reader keeps 98.1% of full-model accuracy while reading 17.8% of routed-expert weights, and its Triton kernels run MoE blocks 1.60–1.90× faster than optimized vLLM at batch one and up to 2.69× at larger batches. Read the paper →

Picture a runner who walks to a far pantry for every single dish, and the walk, not the cooking, sets the pace. That is MoE decoding at small batch sizes: the time goes into reading routed-expert weights out of HBM, not into arithmetic. Most of what the runner could carry is not needed for this dish. Inside each expert's feed-forward network, most channels add little to one token's output, so a selector scores the channels and reads only the promising ones.

The usual instruction to the runner is a threshold: "bring everything above this mark on the shelf." The mark is calibrated once, on offline data, so that the basket averages a target size. The trouble is that the basket size is not under your control. In the paper, a threshold calibrated for an 18% read budget actually read 12.9% on average and as little as 9.9%, and calibrated selectors read less than their budget in all 72 benchmark runs. The count also drifts down as an answer grows: around 6,000 tokens, Qwen3-Thinking kept only 71% of its calibrated count.

The alternative instruction is "bring the k best items, every time." Top-k fixes the read budget per token by construction, so no calibration data is needed to know it. On the same score, top-k matched or beat the threshold version in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks, and it avoided the repetition-loop collapse the thresholds caused on Gemma 4.

Choosing k is half of the design; the other half is scoring channels without reading them. Reading a channel's weights to decide whether to read it would defeat the purpose. The paper's reader instead builds, once at load time, a ternary sketch of the gate and up weights. Each weight row is first rotated with a randomized Hadamard transform, which spreads outliers so that three values (-1, 0, 1) plus one scale per block are enough. A 256-byte BF16 block shrinks to about 30 bytes, adding roughly 7% to resident memory.

Each channel is then scored by the size of what it would add to the layer output: its router weight, times the norm of its down column, times its estimated activation. Because the score is in output units, channels from different experts compete directly, so one expert may get many channels while another gets none. The k winners are read in full and computed exactly. The channels left behind are not simply dropped: a rank-32 map, built once from the down columns, adds back an estimate of their combined contribution for about 0.5% more memory and 0.5–0.8 percentage points more reads. See how the model executor schedules these kernels for where such a kernel would sit.

Here is why 17.8% of the reads did not give a 5.6× speedup. The paper reports that on Qwen3-Thinking the MoE block moves only 18.3% of the full model's memory traffic yet runs 1.90× faster at batch one. Assume (illustrative) that the block's time is a fixed cost f, such as launching kernels, plus a part that shrinks in proportion to the bytes read. Reading 17.8% shrinks that second part to 0.178 of its old size, so the new time is f + (1 − f) × 0.178, and the speedup of 1.90× means this equals 1/1.90 = 0.526. Solving gives f ≈ 0.42. So roughly 42% of the block's time at batch one would be fixed cost that fewer reads cannot shrink (illustrative derivation from the reported numbers). That is consistent with the paper's note that larger batches share fixed costs, which is why the block speedup grows to about 2.4–2.5× at batch four.

The limits are worth stating plainly. The kernels were timed on one RTX PRO 6000 Blackwell GPU, on three post-trained MoE models. Attention is unchanged, so whole-model decoding gains less: 1.15–1.36× at batch one, and up to 1.88× at batch eight or sixteen. The paper also reports that the gain at 8k to 32k tokens of context falls to 1.08–1.54×. The claim to remember is not a speed record but a design rule: a read budget you can state in advance beats a threshold whose budget you can only measure afterwards.

Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network

Related explainers

• Maestro: Markov-chain expert pruning — prunes whole experts, where this article skips channels inside them.
• HCRMAP: hotness-aware expert placement — another way to cut the cost of moving expert weights.