MirrorMoE — Shared experts vs fine-grained expert segmentation — What does it mean?

The news. On October 15, 2026, researchers at ETH Zurich, the University of Hong Kong and the Max Planck Institute posted "Rethinking Routing Exploration in Modern Mixture-of-Experts", which introduces MirrorMoE. They pretrained four shared-expert MoEs from 100M to 2.4B total parameters and report lower test perplexity at every size, including 21.17 → 20.50 at 2.4B, plus a 1.64-point gain in average downstream score at that size. Read the paper →

A mixture-of-experts layer replaces the single feed-forward network in each transformer block with many small ones, plus a router that picks a few per token. Think of a kitchen. A shared expert is the head chef who works on every plate, and the routed experts are specialist cooks the expediter calls in for one dish. Two recent trends changed this kitchen. Shared experts took over the common work, so specialists could specialize. Fine-grained segmentation hired many more, smaller specialists and calls in more of them per dish, which multiplies the number of possible crews. Both are now common in DeepSeek-style and Qwen-style MoEs, and they are the layer's feed-forward network split into pieces.

The paper's diagnosis is that the head chef cooks so much of every plate that it hardly matters which specialists are called. The authors measured this three ways on a 498M-parameter model. Zeroing out one shared expert raised perplexity far more than zeroing out one routed expert. Replacing the learned router with uniform random routing hurt shared-expert layers much less than plain MoE layers in the early and middle layers. And when routing was sampled from the router's own probabilities, the output varied about an order of magnitude less with shared experts. Exploring more crews is useless if every crew produces nearly the same plate. Meanwhile fine-grained segmentation makes the number of crews explode, so the model needs more exploration exactly when shared experts take it away.

The cost shows up as wasted capacity. When the number of activated experts is small, shared experts beat the non-shared model. As it grows, that advantage shrinks or vanishes, and at two of the four scales the perplexity gain from activating more experts turned negative with shared experts. Routing also specializes more slowly: the gap between learned and random routing barely widens across the later training checkpoints.

The obvious fix is to push exploration harder, for example with more noise on the router. The authors report this makes training less stable. MirrorMoE goes the other way and shrinks the number of crews to match what the model can actually explore. It does two things. In mirror routing, the router mixes its normal probabilities ξ with a mirrored version ζ computed from the negated logits: p = (1 − ρ)·ξ + ρ·ζ, so some experts with the lowest scores get picked. In mirror aggregation, the chosen experts are weighted by the original ξ only, so those low-score experts contribute almost nothing. They are activated but functionally suppressed, and how many get suppressed varies by token and layer. Training anneals ρ to 0 on a cosine schedule so the full routing space opens up later, once experts have stabilized, and the load-balancing loss is computed on ξ so it counts only the experts that matter.

Here is a toy version (illustrative, not from the paper). Take 6 routed experts with logits 3, 2, 0, −1, −2, −3 and activate 3. Standard routing gives softmax weights of about 0.692, 0.255 and 0.034 to experts 1, 2 and 3, so the layer uses three real experts. With ρ = 0.3, the mixed probabilities are about 0.485 for expert 1, 0.193 for expert 6 and 0.180 for expert 2, so the activated set is experts 1, 6 and 2. But the aggregation weights come from ξ: 0.692, 0.002 and 0.255. Three experts are activated, but expert 6 carries only about 0.2% of a unit weight, so the token is functionally computed by two. A different token with uncertain logits would keep all three, which is the token-by-token adaptivity the paper proves in two propositions.

Read the table with its limits in mind. Every model has at most 2.4B total parameters, all were trained on one dataset (DCLM), and the paper is a preprint. The authors also report that on MoEs without shared experts, MirrorMoE performs about the same as vanilla training, which supports their explanation: the problem comes from shared experts, not from MoE routing in general. A cheaper variant that simply starts with fewer activated experts and adds more later, with the same count for every token and layer, did not match MirrorMoE, so the per-token adaptivity matters. Training cost stayed about the same in their runs (for example 7,871 ms per iteration for vanilla and 7,844 ms for MirrorMoE at 100M), even with an extra softmax computation per layer.

Goes deeper in: LLM Internals → Transformer Block → Modern Variants & Scale

Related explainers

• SoftMoE differentiable routing — another way to make the router train from the task loss instead of a hard top-k pick
• CIPHER-MoE expert-to-token capacity filtering — caps hot experts from the expert side instead of the token side