What is hotness-aware MoE expert replica placement (HCRMap)?

HCRMap (arXiv 2607.11586) is a serving-time policy for multi-chiplet mixture-of-experts inference. Because a few experts receive most routed tokens, HCRMap estimates each expert's hotness, its weight-loading cost, the overhead of migrating it, and how congested each memory tier is, then decides to promote, retain, demote, or evict each expert replica across the chip's fast and slow tiers. It also maps each routed group of tokens to a nearby resident copy, so hot experts load from fast memory and cold ones are cleared out — cutting end-to-end latency by up to 46.7% versus the paper's baselines.

Why does where an expert lives change latency?

A serving step must load an expert's weights before it can run that expert, and loading from a far, slow memory tier is bandwidth-bound dead time. On a multi-chiplet chip, expert weights are scattered across tiers of different speed, so a hot expert stranded in a slow tier stalls every step that needs it. Keeping the hot experts in fast memory shrinks the memory-bound share of each step — for a bandwidth-bound decode step, the loading, not the math, is what you wait on.

How is HCRMap different from routing methods like ELDR?

Routing methods send each request to a worker whose experts are already loaded — they move the work toward the experts. HCRMap instead moves the experts: it re-slots hot experts into fast memory, keeps spare copies of them, and evicts cold ones, then assigns token groups to the nearest resident copy. The two are complementary levers on the same skew; HCRMap's argument is that treating token movement as the only optimization target leaves the placement win on the table.

HCRMap places hot MoE experts across chiplet memory tiers — Hotness-aware MoE expert replica placement

Jargon

MoE (mixture of experts): A model split into many small sub-networks (experts); a router sends each token through only a few of them, so the model holds huge total weights but activates a slice per token. The catch for serving: a worker must have an expert's weights on hand before it can run that expert.
Expert hotness / skew: Routing is uneven — a handful of experts receive most of the tokens while many stay nearly idle. HCRMap estimates this hotness per expert and treats the hot few as the ones worth keeping close.
Memory tier: Hardware memory comes in a speed ladder: small and fast near the compute, large and slow farther away. An expert in a fast tier is cheap to load; one in a slow tier stalls the step.
Expert replica: A copy of one expert's weights kept in more than one place. Keeping several copies of a hot expert means concurrent token groups can each reach a nearby one instead of queuing for a single copy.
Promote / retain / demote / evict: The four moves HCRMap can make for each expert replica: pull it into a faster tier (promote), leave it (retain), push it to a slower tier (demote), or drop it entirely (evict) to free room.
Multi-chiplet (3.5D): A chip built from several smaller compute dies packaged together rather than one big die. It buys more compute, but expert weights end up scattered across dies and memory tiers of different speed — the exact setting HCRMap targets.
Prefill vs decode: Two phases of generation. Prefill reads the whole prompt at once; decode then emits one token at a time. HCRMap reports gains in both.

The news. On July 13, 2026, researchers published HCRMap (arXiv 2607.11586), a system for serving mixture-of-experts LLMs on multi-chiplet chips. Its premise: because a few experts take most of the routed tokens, that hotness skew overloads compute, communication, memory bandwidth, and execution queues all at once. HCRMap manages hot expert replicas across memory tiers and reports lower end-to-end latency than the Hydra, MoEntwine, and PIMoE baselines in both prefill and decode — up to 46.7%. Read the paper →

Picture a fulfillment warehouse. Every order needs a specific product, and how fast you fill it depends on where that product is stored — an item in the front pick face by the packing dock ships in seconds, while one in deep overflow storage at the back means a long walk. A handful of bestsellers account for most orders, so if those bestsellers are stranded in the back, every picker keeps making the long trip. The stock is fine; the slotting is the problem.

That is exactly the bind a mixture-of-experts model hits on a multi-chiplet chip. Each generated token routes to only a few experts, but a serving worker must load that expert's weights before it can compute — and on a chip stitched from several dies, those weights sit in memory tiers of very different speed. When a hot expert lives in a slow, far tier, the step stalls waiting for the load. This is a memory-bandwidth bottleneck, not a compute one: the math is cheap, the fetch is what costs you.

Here is HCRMap's move, and it is the whole idea. It stops treating where the tokens go as the only knob and starts managing where the experts live. For each expert replica it estimates four things — how hot the expert is, what it costs to load its weights, the overhead of migrating it, and how congested each tier already is — and from those it decides to promote, retain, demote, or evict that replica. The hottest experts get pulled into fast memory and kept as several copies; cold experts get demoted or evicted to make room. Then, rather than only shuffling requests around, it maps each routed group of tokens to a suitable resident copy — the nearest picker who already stocks that product. It is, in effect, a memory manager whose unit of caching is a whole expert.

Walk one decode step through it (internal split illustrative; the end-to-end result is measured). Say a step's time is part compute and part weight-loading, and on a multi-chiplet chip the loading is the larger share — call it ~65% loading, ~35% compute. If a batch's hot experts are stranded in a tier that is, say, 3× slower to read, that 65% balloons and dominates the step. Promote those same hot experts into fast memory and the loading share collapses toward the compute floor, because the bytes now come from a near tier. You have not changed the model or the math — only the distance the weights travel — yet the step gets much shorter. On real multi-chiplet serving the measured end-to-end drop is 43.6% versus Hydra and up to 46.7% versus PIMoE.

Compared against	Prefill latency cut	Decode latency cut
Hydra (arXiv 2607.11586)	~43.6% lower	~43.0% lower
MoEntwine (arXiv 2607.11586)	~34.5% lower	~33.1% lower
PIMoE (arXiv 2607.11586)	~46.7% lower	~46.0% lower

Because the hot experts now load from a near tier, the memory-bound share of each step shrinks and the token-to-token latency drops — without touching the model weights or the hardware. The paper does not detail exactly how the Hydra, MoEntwine, and PIMoE baselines differ; its headline is the framing itself — that on a skewed, multi-chiplet MoE, you can beat the bottleneck by re-slotting the experts, not only by re-routing the requests.

Goes deeper in: GPU & CUDA → Memory Hierarchy → The Memory Ladder

Related explainers

ELDR — Expert-locality-aware decode routing — the complementary lever: route requests to workers whose experts are already loaded, instead of moving the experts
MobileMoE — DRAM-aware MoE scaling — the same "loading the right experts is the memory cost" idea, on phones instead of chiplets
MAESTRO — Markov-chain expert pruning — a different response to expert skew: prune the cold experts rather than re-slot the hot ones
SGLang v0.5.14 — LPLB expert-parallel load balancing — balancing compute across experts, the load lever HCRMap says is not enough alone

Continue in trackGPU & CUDA — Memory Hierarchy: the speed ladder that decides load cost

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based