What is Multi-View MoE?

A mixture-of-experts layer that selects experts with several independent routers and then uses the routers' relative scores to blend learned latent views into each chosen expert's input. Each selected expert still runs once.

Expert selection quality limits an MoE model under a fixed active-expert budget. The paper reports that a single router ranks experts only weakly in line with their measured utility, and that using several routers improves that agreement.

How does it relate to standard top-k routing?

It keeps top-k selection and the same number of active experts, but the top-k runs on summed scores from several sigmoid routers, and the experts receive router-conditioned inputs instead of one shared input.

LLM· 2026-11-02

Multi-View MoE — Multi-router expert selection with latent input views — What does it mean?

Jargon

MoE (mixture of experts): A layer that holds many small feed-forward networks (the experts) and sends each token to only a few of them, so total parameters grow without growing per-token compute. See the feed-forward network.
Router: A small linear layer that scores every expert for the current token. Its scores decide which experts fire and how their outputs are weighted.
Top-k selection: Keep only the k highest-scoring experts and skip the rest. The rule is hard: an expert either runs or gets no direct task-loss gradient from that token.
Sigmoid score: A router score squashed into the range 0 to 1 independently for each expert, instead of competing through a softmax. Scores from different routers can then be added or compared as ratios.
Latent view: A low-dimensional projection of the token's hidden state. Multi-View makes K different projections, so the same token appears to the experts from K different angles.
Expert utility: How much an expert's output matters for a token, which the paper measures with the size (L2 norm) of its output. It is the ground truth the router's ranking is compared against.
Active-expert budget: The fixed number of experts that run per token. It sets the compute and the all-to-all network traffic, so a fair comparison holds it constant.

The news. In late October 2026, Baidu's ERNIE team posted Multi-View MoE, a mixture-of-experts layer that uses several independently initialized routers to choose experts and reuses their scores to build each expert's input. On a 24-layer 10B-A1B model trained on 708B tokens, it lifts the mean of nine downstream tasks by 1.18 percentage points while keeping the active-expert count, expert FLOPs and all-to-all communication unchanged. Read the paper →

Picture a hiring panel with two interviewers. Today's MoE gives the hiring decision to one interviewer, then hands every hired specialist the identical briefing folder. Multi-View changes both halves. Two interviewers score each candidate, and the specialist who wins a seat gets a briefing built from the two interviewers' notes, weighted by how strongly each one backed that candidate. The specialist still does the job exactly once, so the payroll does not grow.

The first half matters because the single interviewer is a weak judge. The authors ran every expert on the same tokens and compared the router's ranking with each expert's measured utility. The two rankings agreed only weakly, especially in middle and deep layers. A badly ranked expert is also stuck: an unselected expert gets no direct task-loss gradient from that token, so a bad pick can reinforce itself.

In a MoE layer the dense feed-forward network is replaced by many expert networks plus a router. Multi-View changes the router in two steps. Step one, Multi-Router: add the sigmoid scores of R independently initialized routers, then take one top-k. The sum comes after the sigmoid on purpose. Adding raw logits would collapse into one linear scorer, because the sum of two linear maps is just another linear map. With the sum after the sigmoid, a strong vote from one router, or moderate support from both, can win a seat.

Step two, Multi-View: for each chosen expert, divide its router scores by their total to get mixing weights, and blend K latent views of the token with those weights before the expert runs. The views are K projections of the same hidden state, one per router. Mixing the inputs, not the outputs, is what keeps the cost flat. The experts are nonlinear, so mixing outputs would need K separate expert passes. Mixing inputs needs one. The projection basis is shared by all experts, and only the weights differ per token and per expert, so the added work is K small projections, not extra expert passes.

A tiny example from the paper shows why the ratio carries information that the sum hides. Take two routers and two experts. Expert i gets scores (0.4, 0.1) and expert j gets (0.2, 0.3). Both sum to 0.5, so selection treats them as equal. Normalizing across routers, expert i receives weights (0.8, 0.2) and expert j receives (0.4, 0.6). Expert i is briefed mostly from router A's view, and expert j mostly from router B's. Same selection score, opposite briefings. Across the real model, nine downstream tasks average 33.07 for the baseline, 33.69 with Multi-Router alone, and 34.25 with full Multi-View, so the views add roughly 0.56 points on top of the routers' 0.62.

Configuration	What changes	Nine-task mean (%)
Baseline	Latent-space MoE, one router, shared expert input	33.07
Multi-Router	Two routers, scores summed before top-k	33.69
Multi-View	Plus per-expert blend of two latent views	34.25

Where it earns its keep

The paper also checks that the model really uses the per-expert briefings. At the final MoE layer, shuffling the mixing weights between experts, while keeping selection and output gates fixed, raises held-out loss by 0.01340 nats per token, so the weights are not decoration. Multi-Router alone raises the rank correlation between router scores and measured expert utility by a relative 8.1% globally, and by roughly 58% in the layer where a single router aligned worst. Treat all of this as one reported setup: one 10B-A1B model, one training budget, compared against a latent-space MoE baseline. The authors do not claim the gain holds at other scales.

Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network

Related explainers

Manifold Power Iteration — router-to-expert alignment — also redesigns the MoE router, but by aiming one router row at each expert's dominant direction
SoftMoE — differentiable top-k routing — keeps one router and makes the top-k choice trainable end to end

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based