What is cache-aware expert routing?

It is a way of managing which Mixture-of-Experts weights stay in fast GPU memory, by training small routers alongside the model that predict which experts a layer is about to need. The prediction updates the cache before the layer runs, so a needed expert is more often already resident. Crucially, it does not change which experts the model actually selects.

An MoE model only computes with a few experts per token, but the full expert set frequently exceeds GPU memory, so decoding is spent moving weights onto the chip rather than multiplying with them. Cutting that expert-weight traffic attacks the real bottleneck. The paper reports up to a 53.3% reduction on Qwen3 against the strongest prefetching baseline it evaluated.

How is this different from prefetching experts?

A prefetcher sits outside the model and guesses early, so a wrong guess costs bandwidth that bought nothing. This approach trains the predictor jointly with the backbone, and its conservative mode — the Temporal Router — improves hits purely by deciding what to retain, issuing no speculative loads at all. The full mode does refine ahead of access, but under an explicit budget.

Does it change what the model outputs?

The cache never overrides expert selection at inference: the native Top-K rule is preserved, so residency alone cannot redirect a token to a different expert. The backbone is jointly post-trained, however, so the resulting model is not guaranteed to be output-identical to the checkpoint it started from. The abstract reports preserved baseline accuracy specifically for the auxiliary-only ablation.

LLM· 2026-09-09

Cut Qwen3 MoE expert traffic by up to 53.3% with cache-aware routing — Predicting MoE expert demand before the layer runs — What does it mean?

TL;DR

What is it: On September 4, 2026 a paper titled "Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference" (arXiv 2609.04895) proposed predicting which experts a Mixture-of-Experts layer will need before that layer runs, so those experts are already sitting in GPU memory when it asks for them.
Why it’s needed: A Mixture-of-Experts model activates only a handful of experts per token, but the full expert set often does not fit in GPU memory, so decoding spends much of its time moving expert weights onto the chip rather than doing arithmetic with them.
vs previous: Earlier systems leaned on proactive prefetching — guess early, start loading, and eat the bandwidth when the guess is wrong — whereas this work trains lightweight cache routers alongside the model and leaves the native Top-K expert choice untouched, so the cache bends toward the model instead of the model bending toward the cache.

Jargon

MoE (Mixture of Experts): A layer that holds many parallel feed-forward networks — the experts — and sends each token to only a few of them. Total parameters go up while per-token arithmetic stays flat. Where the feed-forward network sits in a block
Top-K routing: The rule that decides which K experts a token goes to: score every expert, keep the K highest. K is small — single digits out of dozens or hundreds.
Expert-weight traffic: The bytes of expert parameters that must be moved onto the chip to run a step. This is the quantity the paper minimises — not FLOPs, not KV cache.
HBM: High Bandwidth Memory, the large fast memory attached to the GPU where weights normally live. When the expert set outgrows it, the overflow has to be streamed in. The memory ladder
Prefetching: Loading data before anything asks for it, on a guess. A right guess hides the transfer behind other work; a wrong one spends bandwidth for nothing.
Temporal Router: The paper's update-only mode. It predicts that an expert used now will be used again by a later token in the same layer, and keeps it, without issuing any extra loads.
Spatio Router: The paper's second router. It reads the causal predecessor's hidden state — the representation produced before the target layer is reached — and uses it to refine the cache ahead of that access.
Load-adjusted hit rate: The paper's cache-efficiency metric for comparing methods while accounting for the proactive loading they did to earn their hits — so a method cannot look good simply by loading everything in sight.

The news. On September 4, 2026, Zhenhe Wu and colleagues posted Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference. The paper reframes expert-cache management as a model-side problem rather than a systems-side one: instead of writing a smarter prefetcher around a fixed model, it continues training (post-trains) the MoE backbone together with small auxiliary cache routers, while keeping the native Top-K expert-selection rule at inference. Two modes are reported — a Temporal Router that only decides what to retain, and a full Spatio-Temporal Router that also refines the cache before the target layer is touched. Both were evaluated on Qwen3 and GPT-OSS across GSM8K, MATH and CommonsenseQA. Read the paper →

Picture the kitchen. The walk-in freezer holds every ingredient tray the restaurant owns; the prep counter next to the stove holds maybe three. Each dish needs a handful of specific trays, and the chef cannot start until they are on the counter. If the tray is already there, the dish starts immediately. If it is not, someone walks to the freezer and the chef stands still. On a Mixture-of-Experts model whose experts do not all fit on the GPU, decode time is dominated by moving expert weights onto the chip, not by the matrix math that uses them — the arithmetic per token is small by design, so the transfers are what you actually wait for. That is the same shape as the decode bottleneck you meet in the memory hierarchy: the work is trivial, the fetching is not.

There is an obvious way to make the counter look brilliant, and it is cheating. Tell the chef to cook whatever the trays on the counter can make, and your hit rate goes to 100%. The cheap fix — steering tokens toward whichever experts happen to be cached — is off the table, because it changes which experts the model uses and therefore changes what the model says. This is the constraint that makes the problem interesting, and it is the line the paper draws explicitly: the native Top-K rule survives untouched at inference. The chef keeps picking the recipe. Everything else is allowed to move.

The paper's answer is a pair of small learned routers, trained alongside the model, whose only job is to decide what the expert cache holds. The first of them, the Temporal Router, is the cook who notices which trays this dish keeps reaching for and simply leaves them out. It predicts same-layer reuse — an expert touched by this token is likely wanted by a later one in the same layer — and retains it. The Temporal Router earns its hits without loading anything extra; it only changes what gets kept. That makes it the conservative option: it adds no proactive traffic of its own, though a poor retention decision can still cost a later miss.

The second mode adds the Spatio Router, which is the prep cook reading the slip at the station before this one. It takes the causal predecessor's hidden state — the representation the model has already computed before the target layer is reached — and uses it to refine the cache ahead of that access. Because refinement means real loads, it comes with a budget, and the paper's own sensitivity analysis frames that budget as the knob: spend more and you cover more accesses before they happen, spend more and you also generate proactive traffic you may not need. Cache capacity is the other dial, and it sets how much transfer demand exists at all.

The third piece is the one that is easy to skim past. The routers are not bolted onto a frozen model — the backbone is post-trained jointly with them. The paper reports an ablation — the same setup with one piece removed — in which only the auxiliary routers are trained: accuracy is preserved, but the cache gains are modest. Larger improvements come from letting the backbone adapt as well, which is why this reads as a training-time result rather than a serving trick you can drop into an existing inference engine's memory manager.

Here is where the hit rate turns into bytes (illustrative numbers, held fixed so the ratio is visible). Take one MoE layer with 128 experts, 8 of them activated per token, and put each expert's weights at 24 MB. Hold all three fixed and vary only the hit rate. At a 40% hit rate, 8 × 0.60 = 4.8 experts miss, and 4.8 × 24 MB = 115 MB of expert weights have to be moved for that single layer, for that single token. Push that ordinary cache hit rate up by 18 points to 58% and the misses fall to 8 × 0.42 = 3.36 experts, or 81 MB. In this illustrative layer that is 115 MB → 81 MB per layer per token, a 30% cut, from nothing but a better guess about which trays to have out. The paper's own measured result is reported on a different metric and is wider at the top end: on Qwen3 the Spatio-Temporal Router improved adjusted hit rate by 1.15–18.03 points and reduced expert-weight traffic by 4.6–53.3% against the strongest prefetching baseline it evaluated (arXiv 2609.04895).

Approach	What it decides	What it pays	Reported effect
Matched LM-only baseline	Nothing — no auxiliary cache-router adaptation	Whatever the underlying cache policy already costs	Reference point (arXiv 2609.04895)
Strongest evaluated prefetcher	Guesses experts early and starts loading them	Bandwidth spent on wrong guesses	The baseline this work is measured against (arXiv 2609.04895)
Temporal Router (update-only)	Which experts to retain for same-layer reuse	Nothing — it issues no proactive load	Consistently improves cache hit rate and cuts expert-weight traffic vs matched LM-only baselines (arXiv 2609.04895)
Spatio-Temporal Router (full)	Retention, plus a pre-access refinement driven by the causal predecessor's hidden state	A refinement budget of proactive traffic	On Qwen3: adjusted hit rate +1.15–18.03 points, traffic −4.6–53.3% vs the strongest evaluated prefetcher (arXiv 2609.04895)

Two things this does not settle. The headline numbers are Qwen3 numbers; on GPT-OSS the paper describes results as competitive but task-dependent, which is the honest way of saying the win does not transfer uniformly across models. And the whole result is a post-training one — it asks you to retrain the backbone, so it is not a flag you flip on a served model. What travels regardless of either caveat is the framing: an expert cache is a prediction problem, and the signal you need is already inside the model — in a hidden state it has already computed before the layer is reached.

Goes deeper in: GPU & CUDA → Memory Hierarchy → HBM: Where Your Model Lives

Related explainers

ELDR: expert-locality-aware decode routing — the same expert-traffic problem attacked at the scheduler instead of inside the model
HCRMap: hotness-aware expert placement — deciding where hot experts live, rather than predicting when they are wanted
dMoE: block-level expert routing — cutting the same memory bill by pooling routing decisions across a block

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based