What is GPU-initiated SDMA?

It is a way for a running GPU kernel to start a data transfer on a **dedicated DMA copy engine** (SDMA, on AMD GPUs) instead of copying the bytes with its own compute units. The kernel writes a command into the engine's ring buffer and rings a doorbell; the engine moves the data. AMD added it to RCCL and rocSHMEM in ROCm 10.1.

Why does it matter for inference?

Inference kernels increasingly fuse computation with the communication it feeds, because communication sits on the critical path. If the copy runs on compute units, it competes with the math; AMD reports the SDMA path reaches comparable or slightly higher large-message bandwidth with **one compute unit instead of 64**, leaving the rest for computation.

How is it different from host-initiated copies like hipMemcpyAsync?

`hipMemcpyAsync` also uses SDMA engines, but the CPU starts the transfer, outside any running kernel. GPU-initiated SDMA lets the kernel itself decide the moment to send, which is what a fused compute-and-communication kernel needs.

Does SDMA help for small messages?

Not in AMD's configuration. RCCL uses GIN-SDMA only above 32 MiB for AlltoAll and at 512 KiB and larger for AllGather, and rocSHMEM does not use SDMA below 128 KiB. Below those sizes, load/store copies on compute units are kept.

GPU· 2026-11-15

AMD RCCL and rocSHMEM — GPU-initiated SDMA copy engines — What does it mean?

Jargon

Kernel: A function launched on the GPU and run by many threads at once. A fused kernel does several jobs, such as a matmul and the data transfer it feeds, inside one launch.
CU (compute unit): AMD's name for one cluster of GPU cores with its own registers and scheduler. NVIDIA calls the equivalent a streaming multiprocessor (SM).
Wavefront: A group of 64 threads that a CU runs in lockstep on AMD MI300-class GPUs, the AMD counterpart of an NVIDIA warp.
SDMA: System DMA: AMD's dedicated copy engines. They move memory from one place to another without running any instructions on the CUs.
RCCL: ROCm Communication Collectives Library, AMD's counterpart of NCCL. It implements collectives such as AllGather and AlltoAll across GPUs.
rocSHMEM: An AMD library that lets code running on the GPU read and write other GPUs' memory directly (the OpenSHMEM programming model, on the device).
AllGather / AlltoAll: Two collectives. AllGather gives every GPU a copy of every GPU's chunk; AlltoAll sends a different chunk from each GPU to each other GPU, as in mixture-of-experts token dispatch.
Symmetric memory: A buffer registered at the same layout on every GPU, so a GPU can address its peer's copy directly. The RCCL SDMA path requires it.

The news. On November 13, 2026, AMD's collectives team described GPU-initiated SDMA for intra-node data movement in RCCL and rocSHMEM, available from ROCm 10.1. On eight AMD Instinct MI350X GPUs, the SDMA paths matched or slightly exceeded the large-message bandwidth of default RCCL AlltoAll and AllGather while using one compute unit instead of 64. Read the release →

Picture an office that must deliver a stack of boxes to the building next door. The default way is to pull 64 desk workers off their real job and have them carry the boxes across by hand. The new way is for one worker to fill in a delivery slip, drop it in the outbox tray, ring the mailroom bell, and go back to work while the courier van carries the boxes. The other 63 workers never leave their desks.

On a GPU the desk workers are compute units (CUs), the same hardware that runs your thread blocks. A GPU can move data in two ways: CUs execute load and store instructions and copy the bytes themselves, or a dedicated DMA copy engine, called SDMA on AMD hardware, moves them while the CUs do something else. A separate question is who starts the transfer: the CPU, or a kernel that is already running. The CPU could already start SDMA copies (the standard hipMemcpyAsync call uses them); what this release adds is a running kernel ringing the bell itself, for transfers to other GPUs over the in-node links.

Transfer approach	Who starts it	Who moves the bytes	What the CUs do
GPU load/store copy (default RCCL)	GPU kernel	CUs, via load/store instructions	The copying itself
Host-initiated SDMA	CPU	SDMA engines	Nothing for the copy
GPU-initiated SDMA (new)	GPU kernel	SDMA engines	Issue commands, track completion

Why does it matter that the kernel rings the bell? Because AMD's target is the fused compute-and-communication kernel, where a kernel computes part of its output and ships it to other GPUs while it keeps computing. AMD's post argues this fine-grained overlap matters most for inference, where communication sits on the critical path. In such a kernel, a load/store copy makes the same CUs serve both the math and the transfer, so the two compete for the same hardware. A CPU-started copy avoids that contention, but the CPU is outside the kernel and cannot easily react to the moment a chunk of output is ready. GPU-initiated SDMA gives the kernel both: it decides when to send, and a separate engine does the sending. That is a different answer from KOSMOS-style persistent kernels, which keep CUs doing both jobs and let them switch roles as work arrives; here the CUs stop carrying bytes at all.

Under the hood the mailroom is literal. Each SDMA engine reads commands from a ring buffer, and a doorbell write tells it that new entries are waiting. Commands can be plain linear copies, copies with a fused completion signal, or atomic operations, and AMD compares the engine to a network card that writes directly into remote memory, built for traffic inside a node. Device code never assembles those packets by hand: RCCL's GPU-initiated networking (GIN) layer and rocSHMEM build them behind their device-side APIs. The RCCL path also needs symmetric memory registration, and is switched on with NCCL_GIN_TYPE=7.

Kernel writes commandinto the SDMA ring buffer

Doorbell writemarks entries as pending

SDMA engine copiesCUs keep computing

Data and signal landon the peer GPU

The courier is not always the right call, and the libraries know it. The libraries use SDMA only above a message-size threshold and keep smaller messages on the load/store path. RCCL's GIN-SDMA backend takes over above 32 MiB for AlltoAll and at 512 KiB and larger for AllGather; rocSHMEM does not use SDMA below 128 KiB at all. AMD's rocSHMEM measurements show why the switch exists. At 128 KiB, one CU reached similar bandwidth with SDMA on or off, and several CUs copying with load/stores could go higher. From 512 KiB up, SDMA was the fastest configuration tested, and one CU running one wavefront was enough to reach peak bandwidth, while one CU with 16 wavefronts doing its own load/stores could not saturate the links. The post does not explain the crossover, but the shape is familiar: for a small parcel, carrying it yourself is quicker than writing the slip.

Here is what the 64-to-1 figure buys inside a fused kernel. Hold the GPU at 256 CUs and the transfer at a large message (both illustrative). With the default load/store algorithm, 64 CUs copy and 192 are left for the matmul. With GPU-initiated SDMA, 1 CU issues commands and 255 are left, so the math gets 255 / 192 ≈ 1.33×, about 33% more compute units during the overlap, at bandwidth comparable to or slightly higher than the default in AMD's evaluated large-message AlltoAll and AllGather tests. Whether the matmul actually finishes 33% sooner depends on whether it was limited by compute or by memory: a compute-bound matmul gains roughly in proportion, a memory-bound one gains much less. AMD's post reports the communication bandwidth and the CU count, not end-to-end kernel speedups.

Goes deeper in: GPU & CUDA → Memory Hierarchy → NVLink & PCIe

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based