The news. On July 9, 2026, researchers posted MAESTRO (arXiv 2607.08601), a method for pruning experts from sparse mixture-of-experts language models. Its argument: a MoE only activates a small fraction of experts per token, but the full expert bank stays resident in memory, so serving it is memory-bound. Prior pruning imported local heuristics from dense transformers that rate each expert on its own. MAESTRO instead reads the model's routing and reports up to 10.61% higher average performance retention than those baselines at a 50% compression setting, across five domains including Safety, Bias, and Ethics. Read the paper →
Picture a city's metro map. Every stop costs money to keep open — staff, power, upkeep — whether or not many riders use it. Over a whole day, riders hop from stop to stop along the lines, and if you watch long enough, a pattern appears: a handful of stops see constant footfall, while others sit nearly empty. If you had to close half the stops to save money, you would close the empty ones — the ones almost no rider ever gets off at.
A mixture-of-experts model faces exactly this bill. Its feed-forward layers are split into a bank of experts, and the router sends each token through only a few — but every expert, busy or idle, still occupies GPU memory. That is the MoE trade: cheap to compute, expensive to store. The pruning question is which experts are the near-empty stops — the ones you can close without riders noticing.
The tempting shortcut is to grade each expert on its own — say by its weight magnitude — the way you might rank metro stops by how big the station building is. But a big station with no riders is still dead weight, and a small one at a busy interchange is essential. Size on the map is not the same as footfall, and an expert's real worth is how often the routing actually sends tokens through it — which depends on what the other layers route, not on the expert alone.
Here is MAESTRO's move, and it is the whole idea. It watches how the routing carries each token from one layer's expert to the next, and treats that path across the layers as a random walk — a Markov chain over the experts. Aggregated over many tokens, it computes the stationary distribution: each expert's long-run share of the traffic, the footfall count you would get by riding forever. Because the walk spans layers, this score captures cross-layer routing dependencies a per-layer heuristic can't see. Then it prunes the experts the walk almost never visits — the empty stops — and keeps the ones the traffic actually flows through.
Walk a bank of experts through it (illustrative numbers). Say one layer holds 128 experts, and generating each token routes through only 8 of them. A local score ranks experts by their own weight and closes the 64 smallest — but some of those were small and heavily used, and some large ones were nearly idle, so quality drops. MAESTRO instead ranks by the stationary distribution and closes the 64 the walk rarely reaches. At the same 50% cut, the paper reports the routing-aware choice keeps up to 10.61% more of the model's performance across five domains — the counts here are illustrative, but the lever is real: at a fixed cut, which experts you keep is what moves the score.
| Pruning signal | How it scores an expert | What it misses | Retention at 50% cut |
|---|---|---|---|
| Local / dense-transformer heuristic | the expert on its own (e.g. weight magnitude) | how often routing actually uses it, across layers | baseline |
| MAESTRO (arXiv 2607.08601) | stationary distribution of a routing Markov chain | — (routing-aware, cross-layer) | ~up to 10.61% higher (50% compression, 5 domains, per paper) |
Because the score is the routing's own long-run behavior rather than a property of each expert in isolation, MAESTRO cuts the experts the routing rarely used — the empty stops — and spares the small-but-central ones a local score can miss. The headline is not a smaller model; it is a better answer to which experts to keep — and the surprise is that a decades-old idea, the stationary distribution of a Markov chain, is what tells the busy stops from the empty ones.
Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network
Related explainers
- SoftMoE — differentiable soft top-k routing — a different lever on the router itself, rather than pruning what it feeds
- Grouped Query Experts — MoE routing inside attention — another place experts get selected and what that costs
- MobileMoE — DRAM-aware MoE scaling — why the resident expert bank, not the active slice, is the memory cost MAESTRO attacks
- ELDR — expert-locality-aware decode routing — routing experts smartly at serving time instead of deleting them
- GLM-5.2 — active vs total parameters — the active-vs-total gap that makes a MoE's expert bank worth pruning