What is prefetch-steered routing?

Prefetch-steered routing (PSR) is a way to run a Mixture-of-Experts model with only part of its experts on the GPU. The router reads a deterministic logical cache, into which a small fixed number of the next layer's predicted experts are inserted per layer and token, and physical copies are scheduled separately.

Why does a physical cache make outputs depend on timing?

If the router gets a bonus for experts that are already on the GPU, then which experts get the bonus depends on which copies have finished, and that depends on bandwidth and load. The paper shows perplexity changing from 8.755 to 8.627 just by changing the number of copies in flight.

How is PSR different from speculative execution of predicted experts?

Speculative execution runs the predicted experts instead of the router's choices, so it needs about as many copies as exact routing (29.1 against 30.5 per token in the paper) and keeps the link saturated. PSR only biases the router toward predicted experts, which cuts the number of copies and starts the rest early.

No. The paper reports that on Qwen3.6-35B-A3B, where copies are short relative to compute, PSR loses its speed advantage over Cache-Prior, and in greedy generation on the benchmark prompts Cache-Prior was faster on HumanEval at 50% residency. All baselines were reimplemented in one engine.

LLM· 2026-10-28

Prefetch-steered routing — Logical vs physical expert cache — What does it mean?

Jargon

Mixture-of-Experts (MoE): A model whose feed-forward layer is split into many small "experts"; a router picks only a few (here 8) per token, so most weights sit idle on any one token. See the FFN step.
Expert offloading: Keeping the experts in CPU (host) memory and copying the needed ones to the GPU over PCIe on demand, because the whole model does not fit on the GPU.
Residency: The share of a layer's experts that are held on the GPU at once. 50% residency means half of them are there.
Demand copy vs prefetch copy: A demand copy starts when a layer needs an expert that is missing, so the layer waits for all of it. A prefetch copy starts earlier, so the layer waits only for what is left.
Cache-aware routing: Adding a bonus to the router scores of experts already on the GPU, so the router prefers experts that need no copy. The paper cites Cache-Prior (Skliar et al., 2025) as the baseline.
LRU: Least Recently Used: when the cache is full, evict the expert that was used longest ago.
Pre-gate: The next layer's router applied one layer early to the current hidden state, giving a cheap guess of which experts the next layer will want.

The news. On October 26, 2026, Siming Huang (HKUST Guangzhou) posted a paper on prefetch-steered routing for offloaded MoE decoding. On OLMoE-1B-7B with an RTX 4090 and 50% of experts resident, it reports 10.7 ms per token (teacher-forced) against 24.7 ms for exact routing with LRU. Read the paper →

Picture a kitchen where the pantry is too small for every ingredient. A chef (the router) picks dishes for each order, and a runner fetches missing ingredients from a distant warehouse. A model with offloaded experts works the same way: the router picks 8 experts per layer, and every pick that is not on the GPU waits for a PCIe copy. At batch size one, those waits dominate decoding. Because decoding is limited by moving bytes rather than by arithmetic, this is the memory-hierarchy problem in its plainest form: the slow link between host memory and the GPU sets the speed.

The obvious fix is to nudge the chef toward ingredients already on the shelf. That is cache-aware routing, and it works: fewer copies. The trap appears when you add prefetching. If the chef looks at the physical shelf, then whether a tomato is on it at the moment of cooking depends on how fast the runner was, which depends on link bandwidth and load. The same prompt can produce different text on a different machine, and a benchmark harness that does not copy the engine's timing is scoring a different model. The paper measures this: with routing on the physical cache, changing only the number of copies in flight (1, 2 or 4) moves perplexity from 8.755 to 8.627, and repeated runs on the same GPU differ by up to 0.054.

PSR's answer is a menu board. The router reads a logical cache that evolves by a fixed rule from the token history alone: at each layer, up to n (1 or 2) of the next layer's pre-gate predictions are inserted into the logical set, whether or not their copies have arrived yet. The engine then issues the real copies to match the board, and waits only when the chef must cook something that has not landed. Timing changes how long a token takes, never what it computes. In the same experiment the logical cache gives 8.509 perplexity at every setting, with zero run-to-run range. Inserting a pre-gate prediction also helps quality: the bonus lands on experts the router was already likely to choose, so it steers routing onto experts that are already in flight.

Design	What routing reads	Output depends on timing?	Copies on the critical path
Exact routing + LRU	Nothing about the cache	No	Every miss is a demand copy
Cache-aware, physical cache	Bytes that have arrived	Yes	Fewer copies, but each is a demand copy
Cache-aware, logical cache (PSR)	Deterministic bookkeeping set	No	Fewer copies, most started a layer early

Worked example (50% residency, OLMoE-1B-7B; per-token averages from the paper's copy accounting, combined here for illustration). Cache-Prior at lambda 0.2 makes 10.5 copies per token, every one a demand copy, and the paper puts its latency above the all-GPU floor at 0.70 ms per copy: 10.5 × 0.70 ≈ 7.4 ms of stall. PSR makes more copies, 13.8 per token, but only 2.2 are demand copies, and 9.9 of its prefetches were issued one layer earlier and are still in flight, so the layer waits only for the remainder. Averaged over all its copies, PSR's latency above the floor is 0.28 ms per copy: 13.8 × 0.28 ≈ 3.9 ms of stall. PSR makes more copies than Cache-Prior but pays about half the stall, because it hides most of each copy behind the previous layer's compute. The paper's measured token latency is 10.7 ms for PSR (teacher-forced).

The budget n matters. Letting every non-resident pre-gate candidate in (n = 8) gives the closest match to exact routing but commits about 25 copies per token, more than the link can overlap, and decodes at 17.0 ms, slower than Cache-Prior without prefetching. A small fixed budget keeps the extra copies within what the link can carry. The gains are also workload- and hardware-dependent: on Qwen3.6-35B-A3B, where copies are cheap relative to compute, PSR keeps its quality edge on likelihood benchmarks but not its speed, and in greedy generation Cache-Prior is faster on HumanEval at 50% residency. The authors state that the numbers must be re-measured for other models and links, and that the baselines are reimplementations in one engine, not comparisons against tuned systems.

Goes deeper in: GPU & CUDA → Memory Hierarchy → NVLink & PCIe

Related explainers

Stream an 8B MoE from SSD with one-step-ahead expert prerouting — a trained head guesses the next token's experts so reads start before the router runs.
Predicting MoE expert demand before the layer runs — trained cache routers that leave the native Top-K choice untouched.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based