NVLinkOneSided speeds up MoE all-to-all — One-sided writes vs count exchange — What does it mean?

The news. On November 9, 2026, a team at NVIDIA posted One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink (arXiv 2611.13188). On eight Blackwell B300 GPUs, its NVLinkOneSided kernels finished dispatch plus combine 1.2–3.0× faster than DeepEP, with the largest gains at small batches, and the whole MoE layer ran up to 2.4× faster than DeepEP with DeepGEMM. The kernels sit behind FlashInfer's MoEEpCommunication interface, so they accept any activation quantization format and any MoE compute backend. Read the paper →

Picture a mailroom in every apartment building, where each building is one GPU and each apartment is one expert. Many designs do one or both of two slow things: the buildings phone each other first — "three parcels coming, for apartments 2 and 3" — or the parcels land in a shared pile that the receiving mailroom then re-sorts into one bin per apartment. NVLinkOneSided removes both the phone call and the re-sort. Each receiving building has a wall of pigeonholes with one column reserved for each sending building, and only that sender ever writes into its column, so two senders never collide and nobody has to agree on space in advance. The sender already knows where every parcel goes, because its own router chose the experts, so it simply walks over and drops the parcel in.

In GPU terms, the wall is a dispatch workspace in symmetric memory: inside one NVLink domain, a GPU can store directly into a peer's memory. The workspace is split into one region per source GPU, which the paper calls the rank-major layout, and a source takes its next slot by incrementing a counter it keeps locally, so no remote atomic and no count exchange is needed. The region size is known ahead of time because engines that run attention data-parallel already share each step's batch size. A token goes to each target GPU once, however many of that GPU's experts it picked, with all its expert ids and router weights in one slot. A small indexing kernel then builds, for each local expert, a list of slot numbers, and the first grouped GEMM gathers its input through that list, like a resident reading a slip of hole numbers instead of waiting for a sorted bin. Combine reuses the same slots in reverse: results are written back at each token's slot, and the source GPU pulls them from the slots it recorded during dispatch. Skipping the per-expert copy also saves memory: for DeepSeek-V4.1-Flash on eight GPUs at up to 4,096 tokens per GPU, the paper's dispatch workspace holds 160 MiB of hidden states, against 1,020 MiB for the per-expert GEMM input that DeepEP V2.5 with DeepGEMM builds, plus its 242 MiB communication buffer.

Skipping the phone call leaves one hard question: how does the receiver know its parcels have all arrived? A remote store does not wait for the peer, and GPU memory is weakly ordered, so stores can land in any order. The paper offers two answers. Fence works like a doorbell: when a GPU's last token is out, it sends its slot counts (after the data, not ahead of it), executes one system-scope release fence, and writes a flag on every peer; each receiver waits until every flag has arrived. Sentinel removes the doorbell. Every landing slot starts out holding a placeholder bit pattern, such as 0xFFFFFFFF (two BF16 NaNs, which the sender is careful never to store), and because NVLink delivers each aligned 32-bit word whole or not at all, a slot with no placeholder word left is complete, with no fence and no flag. To tell an empty slot from one still in flight, every sender also writes expert ids to every receiver, with an invalid id where no token goes. Unlike NCCL's LL protocol, which interleaves flag words with the data, Sentinel adds no flag words at all.

Here is why fixed costs are the whole story at decoding batch sizes. Hold three things fixed: DeepSeek-V4.1-Flash, eight B300 GPUs, and one token per GPU. That token's dispatch payload is 5,328 bytes per target GPU in MXFP8 (an 8-bit number format), and with top-6 routing it reaches at most six GPUs, so at most 31,968 bytes go out; combine brings back at most six BF16 (16-bit) results of 10,240 bytes each, or 61,440 bytes. At the 900 GB/s one-direction NVLink bandwidth of a Blackwell GPU, those ~93 KB need about 0.1 µs (our arithmetic from the paper's figures). The measured round trip is 47.9 µs with DeepEP, 27.6 µs with Fence and 17.4 µs with Sentinel, so in every design more than 99% of the time goes to synchronization, launches and copies, not to moving data — and Sentinel's 30.5 µs saving over DeepEP is collected again in every MoE layer of every decode step.

The two protocols trade places as the batch grows. Sentinel is faster below 256 tokens per GPU and Fence above it, because Sentinel's per-slot work — checking, copying out and resetting every received word — grows with the batch while its saving does not. At large batches a different design wins outright: DeepGEMM's MegaMoE fuses dispatch, both expert GEMMs and combine into one persistent kernel (a single kernel that stays running for the whole layer) and hides the transfers behind Tensor Core work, so it leads from 256 tokens per GPU (by up to 35% at 4,096), while the NVLinkOneSided layer is faster up to 128 tokens per GPU, by up to 30%. That split lines up with how decoding is run: a time-per-output-token target holds a decode batch to a few hundred tokens per GPU, which is exactly where fixed costs dominate. The hard limit is the building's walls. Both protocols depend on direct loads and stores and on whole-word arrival, which RDMA (the network cards that link separate servers) does not offer, so the expert-parallel group must fit inside one NVLink domain — the eight GPUs of an HGX B300 board, or the 72 of a GB200 or GB300 NVL72 rack.

Goes deeper in: GPU & CUDA → Memory Hierarchy → NVLink & PCIe

Related explainers

• ThunderEP: single-step host broadcast — the same dispatch problem on consumer GPUs with no NVLink, solved through shared host memory
• Primus MegaMoE: tile-level all-to-all overlap — the other route: hide the exchange behind expert compute instead of shrinking it
• TPUv7 Ironwood: DP attention with expert parallelism — why serving stacks split attention data-parallel and experts expert-parallel in the first place