AMD Primus fuses MoE all-to-all into GEMMs — Tile-level communication overlap — What does it mean?

The news. On October 21, 2026, AMD published a ROCm blog post on its open Primus training stack describing MegaMoE, a design that fuses MoE all-to-all communication into the expert grouped GEMMs, together with FP8 and MXFP8 expert GEMMs. The post reports DeepSeek-V3 runs from 256 to 1,024 MI355X GPUs, and a layer-level benchmark on 8 GPUs. Read the AMD post →

Picture two kitchens. In the first, a truck unloads every crate onto the dock, and the cook starts only when the last crate is down. That is how a standard MoE layer runs on a GPU cluster: the all-to-all dispatch is one kernel, the expert matrix multiply (FC1) is the next kernel, and the GPU's matrix cores (the units that do the multiplication) sit idle until the dispatch kernel finishes. AMD's profiling, as the post reports, splits the MoE forward pass roughly evenly between communication and expert compute, so that idle time is large.

A MoE layer has eight stages, and in a conventional stack each one is its own kernel with its output written to GPU memory before the next kernel reads it. The router picks each token's top-k experts, a permutation gathers tokens into expert order, the dispatch all-to-all sends each token to the GPU that owns its expert, FC1 and FC2 with a SwiGLU between them do the expert work, a combine all-to-all returns the results, and a final unpermute-and-scale step merges the top-k contributions per token.

In the second kitchen, each crate carries a ready flag, and the cook starts on a crate the moment its flag goes up. That is MegaMoE. The key move, in the post's words, is not collapsing the whole layer into one kernel but putting each all-to-all inside the grouped GEMM that produces or consumes its data, so the communication has something to hide behind. This is the same lever as operator fusion, but aimed at network traffic instead of memory traffic: the layer becomes three kernels.

• Dispatch + FC1. The GPU's compute units are split into two roles. Some keep pulling tokens in from peer GPUs while the others run the matrix cores on tiles that have already landed. There is no global barrier; a per-tile arrival flag lets the GEMM start on a tile the moment it lands.
• SwiGLU. A small middle kernel that also quantizes its own output, so the MXFP8 cast is not a separate pass over the activations.
• FC2 + combine. The FC2 GEMM's epilogue pushes each finished tile to the GPU that owns it, and the reduction happens there, so results leave as they are produced instead of after the whole GEMM completes.

The backward pass mirrors this: the dispatch of the output gradient fuses with the FC2 data-gradient GEMM, and the FC1 data-gradient fuses with combine and reduction. Because the permute and unpermute stages disappear and the staging buffers are no longer written out between kernels, the layer also avoids the intermediate-tensor traffic that the kernel launch problem describes for small operators.

Worked example. Hold the shapes fixed at the post's setting: DeepSeek-V3 expert dimensions (hidden size 7,168, intermediate size 2,048, 256 routed experts, top-8 routing), 8 GPUs with EP8, and 8,192 tokens per GPU. At BF16 the unfused layer takes 28.62 ms and MegaMoE takes 20.46 ms, so fusion alone is worth 1.40×, with no quantization involved. Now switch to MXFP8. The unfused layer only improves from 28.62 ms to 24.90 ms, just 1.15×, because all-to-all time and the materialized intermediates are still there, and a narrower data type does not remove a kernel boundary. Inside the fused layer the same switch takes 20.46 ms to 13.74 ms, 1.49×. The two compound: 24.90 ms ÷ 13.74 ms is 1.81× for fusion at MXFP8, and 28.62 ms ÷ 13.74 ms is about 2.1× for both changes together.

That is a layer, not a training step. On a 4-layer DeepSeek-V3 proxy (plus its multi-token-prediction module) on one 8-GPU node, swapping only the MoE implementation moves the step from 8,758 ms to 7,700 ms at MXFP8, 1.138×, and from 9,722 ms to 8,991 ms at BF16, 1.081×. Attention, dense layers and the optimizer are untouched, so they dilute the layer-level gain, as the post itself notes.

What this does and does not show. The layer and step numbers above come from one 8-GPU node with a reduced-depth proxy, not a full 61-layer run, and the post reports its separate 256-to-1,024-GPU DeepSeek-V3 results, including a 1,700-step run on 1,024 GPUs, and attributes those gains to pipeline layout and recompute settings, not to MegaMoE. The post also reports that the layer speedup moves by under 1% when expert token counts are deliberately imbalanced, and that the fused path used the same peak memory at BF16 and about 3% less at MXFP8. As for why this works at all: the saving comes from the schedule, not from a faster kernel. Dispatch and GEMM were never individually slow; they were serialized. Moving the overlap boundary from the micro-batch down to the tile is what lets one hide inside the other.

Goes deeper in: GPU & CUDA → Operator Fusion & FlashAttention → Operator Fusion

Related explainers

• Persistent decode megakernel — another case where removing kernel boundaries is the whole win.
• NVGEMM epilogue fusion — the epilogue idea that MegaMoE's FC2 stage uses to ship tiles out.
• ThunderEP on PCIe GPUs — a different take on the same MoE dispatch problem.