What is one-step-ahead expert prerouting?

It is a trained head that predicts which experts the next token will route to, while the current token is still being computed. The framework uses that prediction to start loading those expert weights from storage early, so the read overlaps the forward pass instead of stalling it. Edge0 reports up to +59% decode throughput from it.

Why does a mixture-of-experts model need this at all?

Because a sparse model only touches a few experts per token — 8 of 128 in Edge0 — the rest of the expert weights can live on SSD, which is what keeps peak active memory near 1.0 GiB for a 7.9B-parameter checkpoint. The catch is that the router does not reveal which experts are needed until the layer is already running, so without prediction every load becomes a wait.

How is this different from fetching experts on demand?

Fetch-after-routing waits for the router's answer, so the storage read is serialized against compute and the processor idles through it. Prerouting removes the ordering dependency by guessing a token early, so the read and the compute run concurrently. Both keep only the active set resident; the card reports no separate peak-memory figure without the prerouter, so the published contrast is about stalling, not about footprint.

What happens when the prediction is wrong?

The model card frames the prerouter as a way to overlap expert loads rather than as a replacement for the router, which suggests a miss costs a wasted read and a late arrival rather than an incorrect token. The release does not publish a prerouter hit rate or a measured miss penalty, so the exact cost of a wrong guess is not verifiable from what has been shared.

LLM· 2026-09-15

Stream an 8B MoE from SSD in 1 GiB active memory — One-step-ahead expert prerouting — What does it mean?

TL;DR

What is it: Edge0 is an open 8B sparse mixture-of-experts checkpoint that stays on SSD in 4-bit and streams in only the experts each token routes to, and the piece that makes it fast enough to use is one-step-ahead expert prerouting — a trained head that guesses the next token's expert choices before the router is asked.
Why it’s needed: Expert weights are the bulk of a sparse model but a single token touches only a sixteenth of them, so moving the rest to storage is what gets an 8B model down to about 1 GiB of active memory — and prerouting is the separate piece that stops the resulting storage reads from making the processor wait on them.
vs previous: The obvious approach, fetch-after-routing, cannot start a read until the layer's router has already chosen, so every expert load is a stall the decode step sits through; prerouting removes that dependency by predicting one token early, which the publisher measures at up to +59% decode throughput.

Jargon

MoE: Mixture of Experts. A layer holding many parallel feed-forward blocks — 128 of them in Edge0 — that sends each token to only a few. Total parameters are large; active parameters per token are small.
Router: The small learned layer that scores a layer's experts for the current token and picks the top K. Its output is what tells the framework which expert weights to load, and it only runs once the token actually reaches that layer.
Routed width (K): How many experts each token is sent to. Edge0 uses K = 8 out of 128, so one token touches one sixteenth of a layer's expert weights.
Prerouter: A separately trained head that predicts the next token's routing one step early. The card says the prediction lets expert loads overlap the forward pass, which reads as a hint used to start a storage read sooner rather than as the final routing decision — though it does not spell the control path out.
Active set: The weights and state actually resident in RAM at a given moment, as opposed to the full checkpoint. Edge0 reports a peak active memory of 1.0 GiB against a checkpoint roughly four times that size.
Decode: The one-token-at-a-time phase of generation that follows the prompt. It moves a lot of memory per unit of arithmetic, which is why one extra wait per MoE layer per token is expensive. How decode differs from prefill
Recover-LoRA: Edge0's quantization-recovery step. The 4-bit model is frozen and a small set of extra trainable weights (a LoRA adapter) is fitted to imitate the original 16-bit model, reported to land within 2.8 points of it on average.

The news. The Edge0 project put Edge0-8b-a1b-preview on Hugging Face on September 8, 2026 and revised its model card through September 14. It is an 8B-class sparse mixture-of-experts checkpoint — about 7.9B total parameters, about 1.2B active per token, 128 experts with 8 routed — quantized to 4 bits and served by a framework that leaves the whole checkpoint on storage and streams in only what each token routes to. On a 24 GB M4 Pro Mac mini the publisher reports 23.9–25.3 tokens/s decode and 1.0 GiB peak active memory. Three mechanisms carry it: SSD expert offload, Recover-LoRA quantization recovery, and the one this article is about, the prerouter. It is an Apache-2.0 preview on an Apple Silicon (MLX) backend. Read the model card →

Stay in the kitchen for a moment. The freezer holds every ingredient the restaurant can cook with and it is enormous, but no single dish needs more than a handful of them. A mixture-of-experts layer works the same way: Edge0 holds 128 experts per layer and routes each token to 8 of them, so any one token touches a sixteenth of that layer's expert weights and ignores the rest. That sparsity is the entire reason keeping the freezer outside the kitchen is thinkable at all. A dense model has no freezer — every ingredient is on the counter for every dish, which is why the size of the checkpoint sets the size of the machine for dense weights in a way it need not for sparse ones.

The counter, in this analogy, is RAM, and the claim Edge0 is really making is that counter space is set by the dish rather than by the freezer: peak resident memory tracks the roughly 1.2B parameters a token actually uses, not the roughly 7.9B the checkpoint contains. Storage stops being the place a model is downloaded from before it runs and becomes a rung on the memory ladder underneath HBM (the high-bandwidth memory attached to the accelerator itself) — far slower than anything above it, but far larger, and now read during the forward pass instead of before it.

Here is where the straightforward version of that idea falls over. The router's choice arrives too late to act on: by the time a layer knows which 8 experts it wants, that layer is already the thing waiting for them. In kitchen terms the runner is not told what to fetch until the chef is standing over an empty pan. It does not matter how fast the runner is, because the walk happens while the cooking is stopped.

Decode is the worst possible place to meet this problem. A decode step does very little arithmetic for each byte it moves, so it already sits on the memory-bound side of the roofline before any disk is involved; bolting a storage round-trip onto each MoE layer stretches a step that had no spare compute to hide it behind. It also puts the problem squarely in serving-architecture territory rather than model territory — it is a question about what the memory manager chooses to keep where, not about what the network computes.

The prerouter is the fix, and the shape of it is simple. It breaks the dependency by predicting the next token's routing during the current token's forward pass, so the read and the compute happen at the same time instead of one after the other. The runner reads the next ticket over the chef's shoulder and leaves for the freezer while the current dish is still on the heat; when the chef turns around, the ingredients are already on the counter.

The model card describes the prerouter as predicting routing one step ahead so that expert loads overlap the forward pass, which reads as a prefetch hint rather than a replacement for the router. If that reading is right, a wrong guess costs a wasted read and a late arrival rather than a wrong token, and that asymmetry is what makes speculating on addresses safe in a way that speculating on outputs is not — the same bargain speculative decoding strikes, moved from tokens to memory addresses. The release reports the throughput gain but publishes no prerouter hit rate and no miss cost, so how often the guess lands is not something the card lets us check.

Getting experts to the processor	Weights resident in RAM	What a decode step does	Reported
Keep every expert resident	the whole 4-bit checkpoint, ~4 GB (7.9B params × 4 bits, arithmetic)	never waits on storage	not what Edge0 ships
Fetch after routing	active set only; no separate peak is reported without the prerouter	puts expert loading on the critical path — the routed set is unknown until the layer runs	the comparison point for the +59%; its configuration is not detailed
Prefetch one step ahead (prerouter)	active set only, ~1.0 GiB peak (measured on one machine)	reads overlap the forward pass	up to +59% decode throughput (model card)

It helps to put the time budget next to the work. Hold three numbers fixed: 24 layers, 8 routed experts per MoE layer per token, and the reported decode rate of 25 tokens/s. Twenty-five tokens per second is a budget of 40 ms per token, and inside that window the framework has to resolve and land up to 24 × 8 = 192 expert selections — one routed set per MoE layer, and 192 only if all 24 layers are MoE layers. Spread evenly, each selection's share of the budget is about 0.21 ms. Treat that as arithmetic rather than as a measured per-read latency: the card does not say how many of the 24 layers are MoE, whether a layer's 8 experts arrive as one read or eight, or how much of the I/O overlaps itself. What the arithmetic does show is how little slack there is to work with. Fetch-after-routing puts every one of those selections on the critical path, because none of them can begin until the layer that needs them is already running — which is also why the publisher notes the gain grows with storage latency, model size and routed width K: each of those makes the loading bill bigger, and so makes hiding it worth more.

The memory side composes the same way. 7.9B parameters at 4 bits is roughly 4 GB of checkpoint, while measured peak active memory is 1.0 GiB — the resident set is about a quarter of the file, and it is pinned by the active parameter count rather than the total one. That ratio, not the throughput number, is the part that generalizes: it is what decides whether a given sparse model fits on a given machine at all.

Two caveats worth carrying out of this. The Edge0 numbers come from the publisher's own benchmark on one machine, a 24 GB M4 Pro Mac mini, and the framework currently targets Apple Silicon through MLX — the technique is general, the measurements are not. And prefetching is only ever as good as the guess. The card publishes neither a hit rate nor a miss penalty, so what the release shows is that this model's routing was predictable enough to be worth 59%, not that MoE routing is predictable one step out in general.

Goes deeper in: LLM Serving → Inference Engine → The Memory Manager

Related explainers

vLLM 0.28 disk-backed tiered KV offload — the same storage tier, used for KV cache instead of expert weights
TF-Engram's SSD-backed phrase memory — the other way to hide SSD latency, by prefetching on a predicted phrase
Qwen3.8-Flash-Next N-gram embedding offload — moving embeddings rather than experts off the accelerator

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based