Sparse MoEs overfit repeated data sooner than dense models — Data-repetition tolerance — What does it mean?

The news. On September 10, 2026, Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang and Luke Zettlemoyer posted Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data. Across models from 80M to 1B active parameters and up to 8.5B total, they varied how many times the training corpus was repeated and found that sparse Mixture-of-Experts models degrade faster under repetition than dense Transformers — an effect that grows with sparsity, and that tracks total rather than active parameters. Read the paper →

Picture the training corpus as one big flashcard deck. A dense model is a single generalist student: every card passes in front of them, so a second run through the deck is still a second look at the whole subject range. An MoE is a study group instead. A router reads each card and hands it to one member, and the members specialize — this one takes the Python cards, that one takes the chemistry cards.

Splitting the deck is the whole point of the architecture. An MoE can hold far more total knowledge than a dense model of the same per-token cost, because only a couple of members work on any given card. The feed-forward layer that an MoE replaces is where those parameters live, and modern variants push the expert count higher with each generation. But it also means no single member ever sees the whole deck — each one only ever studies their own pile.

Now run the deck again. The generalist gets a second pass over everything; the specialists get a second pass over a pile that was small to begin with. It is easy to imagine a student re-reading a thin stack until they know the cards rather than the subject — but treat that as intuition, not as the paper's claim, because the paper does not establish slice size as the cause. What it does measure is that the specialists cross the line first: dense 80M models absorb more than 8 passes with minimal degradation, while MoEs begin to suffer at 4.

The Feed-Forward Network is the one slot an MoE replaces, swapping a single network for a router plus a bank of experts. Everything else in the block stays as drawn.

So why do the specialists memorize sooner? The paper analyses two internal patterns that correlate with the damage. First, MoE routing universally stabilizes early in training — the router settles on which expert handles which kind of token and then largely stops changing its mind. Second, expert specialization correlates with overfitting to repeated data: the more sharply each expert has carved out its own slice, the worse the repetition damage. Together these observations suggest a possible mechanism rather than a proven one: a router that stops reshuffling the piles would send each expert a similar slice on every later pass, and sharper specialization is associated with worse repetition damage. The paper reports both as correlates of the overfitting, not as its demonstrated cause.

The severity tracks sparsity, and the paper is specific about which parameter count matters: the effect is dictated by total rather than active parameters. Two models that cost the same per token can behave very differently under repetition if one carries far more total weight — the same count that drives how much memory the model needs before you compress it.

A suggested reading of the paper's evidence, not a demonstrated causal chain: it reports the early routing stabilization and the specialization-overfitting link as correlates.

Hold three numbers fixed to see why one shared repetition count is not shared at all (illustrative — the paper does not publish this configuration). Take a 10B-token corpus, a dense model, and an MoE with 64 experts that routes each token to 2 of them. The dense model's feed-forward weights see all 10B tokens on every pass. Each MoE expert sees roughly 2 of every 64 tokens — about 310M tokens, a slice 32× thinner than the dense model's view. Run four passes over the corpus and both models have consumed 40B tokens, but the dense weights were fit on 10B unique tokens while each expert's weights were fit on 310M unique tokens, seen four times each. The paper does not put a formula on this, and real routing need not be that even. Read the arithmetic as an intuition that is consistent with the measured finding — that the damage is dictated by total rather than active parameters — not as the explanation for it.

The practical read is not that MoEs are worse. It is that the unique-data budget an MoE needs is not the same as the dense budget in the comparisons this paper ran, so the repetition rate you safely used on a dense model is not a number you can carry over. And the fix the paper found is not free either: strong masking keeps the MoE ahead of dense past 64 repetitions, but no method it tested fully matches training on all-unique data.

Goes deeper in: LLM Internals → The Transformer Block → The Feed-Forward Network

Related explainers

• Active vs total parameters — the count this paper says the damage is dictated by.
• 8B dense matches the prior 32B MoE — the same dense-versus-sparse comparison, scored on quality instead of data reuse.
• The verbatim-recall threshold — how much capacity it takes before a model reproduces its training text word for word.