A training method for Mixture-of-Experts models from arXiv 2610.05744. Tokens keep their normal Top-K expert choices, then each expert keeps only the tokens most similar to it, up to a capacity of 8 times the even share, and drops the rest. The paper reports 1.10×–1.94× faster domain fine-tuning on models from 284B to 1.6T parameters.

Why does expert load imbalance slow MoE training?

Under expert parallelism each GPU holds some experts, and the layer finishes only when the busiest device finishes. If a few experts attract most tokens — 10% of experts took over 88% of the routing workload in the paper's DeepSeek-V4-Flash measurement — they set the step time and the memory peak.

How is it different from Expert Choice routing?

Expert Choice routing lets experts pick tokens with no token-side choice at all. CIPHER-MoE keeps the token's own Top-K choice and only filters it at the expert. In its strict-drop form, which the paper recommends, an expert can turn tokens away but never takes a token that did not pick it.

Does a dropped token lose its training signal?

No. Dropping removes one token-to-expert assignment only. Each expert the token picked filters it independently, so the token keeps every assignment that was accepted, still passes through the shared expert, and stays in the training loss.

LLM· 2026-10-08

Filter overloaded MoE experts without changing token-side Top-K — Expert-to-token capacity filtering — What does it mean?

TL;DR

What is it: CIPHER-MoE, from an October 2026 paper, is a training-time fix for Mixture-of-Experts (MoE) models: each token still picks its usual Top-K experts (the K expert networks its router scores highest), and then each expert keeps only the best-matching tokens up to a fixed capacity and drops the rest — expert-to-token capacity filtering.
Why it’s needed: When experts are spread across GPUs (expert parallelism), the step waits for the busiest expert, so when a few experts attract most tokens they set the step time and the memory peak — the paper's 1.6T-parameter DeepSeek-V4-Pro run ran out of memory before its first iteration under plain MoE training.
vs previous: Earlier fixes either nudge the router toward balance on average (auxiliary losses, bias terms) and still allow a spike in any one step, or spend extra resources (replicating or moving hot experts), or give up content-based routing (a fixed router); CIPHER-MoE puts a hard cap on each expert per step while leaving the token's own choice of experts unchanged.

Jargon

Mixture-of-Experts (MoE): A transformer layer that replaces the single feed-forward network with many smaller FFNs called experts, and runs only a few of them per token.
Router and Top-K: The router is a small linear layer that scores every expert for a token; the token is sent to the K highest-scoring experts (Top-K).
Routed vs shared expert: A routed expert runs only for tokens the router sends to it. A shared expert runs for every token, whatever the router decides.
Expert capacity: The maximum number of token assignments one expert processes per micro-batch. The capacity factor states it as a multiple of the even share; CIPHER-MoE uses a factor of 8.
Expert parallelism (EP): Spreading the experts across GPUs, so tokens are sent to whichever device holds their chosen expert, and the layer finishes only when every device has finished.
Cosine similarity (affinity): How closely a token's hidden vector points in the same direction as an expert's router weight vector, from −1 to 1. CIPHER-MoE uses it to rank which tokens an expert keeps.
Token dropping: Removing one token-to-expert assignment so that expert does not process that token. The token itself stays in the training sequence and the loss.
SFT: Supervised fine-tuning: further training a pretrained model on labelled examples. The paper's experiments are domain-specific SFT runs.

The news. On October 5, 2026, a team from Tongji University, Cornell, the Harbin Institute of Technology (Shenzhen) and the Shenzhen Loop Area Institute posted CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training (arXiv 2610.05744). They fine-tune DeepSeek-V3, DeepSeek-V4-Flash, GLM-5 and DeepSeek-V4-Pro (284B to 1.6T parameters) on domain datasets and report 1.10×–1.94× faster training and up to a 64.9 percentage-point cut in the top-1 expert's workload, with domain-task scores close to plain MoE training. The paper says the code will be released soon. Read the paper →

Picture course registration. Every student lists the courses they want, and most of them want the same famous course. If that course has no seat cap, its one lecture hall overflows while the other halls sit half empty — and the term cannot start until the overflowing course has seated everyone. That is an MoE layer under expert parallelism. Each token lists its Top-K experts, a few popular experts collect most of the tokens, and every GPU waits for the device holding the hottest expert.

The skew is not small. On a domain-specific dataset, the paper measures that 10% of DeepSeek-V4-Flash's experts receive over 88% of the routing workload. A hot expert also needs memory for every token it holds, which is how the 1.6T-parameter DeepSeek-V4-Pro ran out of GPU memory before finishing a single step in the authors' setup.

The usual fixes act on the students' lists. An auxiliary load-balancing loss (GShard, Switch Transformer) or DeepSeek-V3's per-expert bias pushes the router toward even use, but it works on averages, so one micro-batch can still overload one expert. A fixed router that assigns experts by token position is perfectly even, but it stops matching tokens to experts by content; in the paper's Table 1 it trains faster yet scores lower on all four operations-research benchmarks than normal routing, which the authors attribute to weaker expert specialization when routing ignores content.

CIPHER-MoE leaves the lists alone and gives each course a seat cap. First the router runs exactly as before, and each token picks its Top-K experts. Then each expert ranks the tokens that picked it by cosine similarity between the token and the expert's router vector, drops any below a threshold, and keeps at most C of the rest — the best-fit students up to the seat cap. The paper sets C at 8 times the even share of tokens per expert. Most experts never reach that cap; the cap only bites on the hot experts, which is exactly where the step time and the memory peak come from.

What happens to a turned-away student matters as much as the cap. Rejection by one expert does not cancel the token's other assignments: each expert it picked filters independently, and the token still passes through the shared expert, the core lecture every student attends. It also stays in the training sequence and the loss. The paper compares two policies: strict drop, which sets that one assignment's routing weight to zero, and reroute, which sends it to another expert with spare seats and enough similarity. The paper recommends strict drop: it is faster in all 9 model-and-dataset pairs of its Table 2 and scores higher in 5 of them. The authors' explanation is that a rerouted token shifts what the receiving expert trains on, and its gradient can pull that expert away from its specialty — like seating a chemistry student in a literature seminar just because a seat is free.

Here is the arithmetic of one hot expert (illustrative numbers, not from the paper). Hold a micro-batch at 4,096 tokens, 256 routed experts, and K = 8. That is 4,096 × 8 = 32,768 assignments, so the even share is 32,768 ÷ 256 = 128 tokens per expert, and a capacity factor of 8 gives a cap of 1,024. Suppose the hottest expert is picked by 2,048 tokens, 16 times its even share, and at least 1,024 of them pass the similarity threshold. CIPHER-MoE keeps its 1,024 best-matching tokens and drops the other 1,024 assignments. That drops about 3% of all assignments (1,024 of 32,768) and halves the work on the busiest expert, from 2,048 token passes to 1,024. Halving that expert's work does not halve the step time, because communication, the other experts and the filtering itself also take time; the paper's measured end-to-end speedups are 1.10× to 1.94×.

Approach	What it changes	Caps the busiest expert in each step?	Cost
Auxiliary balancing loss (GShard, Switch Transformer)	Adds a loss term that rewards even expert use	No — balances on average	Can conflict with the language-modelling loss
Loss-free bias balancing (DeepSeek-V3)	Adjusts a per-expert routing bias from past loads	No — reacts to history	No extra gradient, but a spike can still happen
Expert Choice routing (Zhou et al., 2022)	Experts pick tokens; tokens do not choose	Yes, fixed per-expert capacity	The token-side choice is gone
Fixed router	Assigns experts by token position	Yes, perfectly even	Can weaken expert specialization; NL4Opt 88.93 vs 93.08 for normal routing on DeepSeek-V4-Flash (Table 1)
Expert replication or migration (FasterMoE, FlexMoE)	Copies or moves hot experts to more devices	No — spreads the load across devices instead	Extra replicas, migration and runtime orchestration
CIPHER-MoE, strict drop	Tokens keep Top-K; each expert keeps its top-C tokens by cosine similarity	Yes, C = 8× the even share	Some assignments dropped; 1.13×–1.94× faster training (Table 2)

The result is faster training at roughly matched domain-task scores, but not at zero cost on every benchmark. In its Table 2, strict drop on DeepSeek-V3 trains 1.94× faster on the operations-research data with a weighted average of 61.40 against 60.60 for plain training; on DeepSeek-V4-Flash it is 1.42× faster with 68.59 against 69.09. General-benchmark results are mixed: after operations-research fine-tuning, DeepSeek-V4-Flash's BBH score falls from 84.00 to 71.20 with strict drop, while most other benchmarks move by a few points either way (Table 4). All of the paper's experiments are domain-specific fine-tuning runs, not pretraining from scratch. On DeepSeek-V4-Flash, the filtering itself is reported to cost 1.57% of training time for strict drop and 3.85% for reroute.

Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network

Where MoE fits among the other component swaps since 2017 is covered in LLM Internals → Transformer Block → Modern Variants & Scale.

Related explainers

SGLang LPLB load balancing — the serving-side answer to hot experts: replicate them and rebalance load, instead of capping them
SoftMoE differentiable routing — changes how the router picks experts, where CIPHER-MoE leaves the router's pick alone
Manifold Power Iteration router alignment — improves load balance by reshaping the router's weights during pretraining

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based