GPU·

KIWI — Programmable reverse offload vs GPU-driven networking — What does it mean?

The news. On November 18, 2026, researchers at AMD posted KIWI (Kernel-Initiated, Workload-Defined Invocations) on arXiv. It argues that reverse offload to CPU proxy threads, long seen as the slower fallback to GDA, can be the faster path when the GPU-to-CPU handoff is cheap. The team built KIWI-P2P, a SHMEM-like point-to-point library in fewer than 1,000 lines of C++, and KIWI-EP, new MoE dispatch and combine kernels. On an eight-GPU-per-node MI300X cluster with 400 Gb/s RoCE, KIWI-EP lowered dispatch-plus-combine latency by up to 2.3× against the best existing AMD backend and vLLM time per output token for GLM-5.3 by up to 29%. The code is open source on GitHub. Read the paper →

Picture the building site. The crews are good at one thing: many people working on many bricks at the same time. Moving a pallet to the other tower needs a different skill, a short chain of levers pulled in order, one after another. Under GDA, a crew member climbs into the crane cab and works the levers, and the rest of the crew stands still while they do it. That is the GPU's situation. Driving a NIC is a sequence of dependent scalar steps: choose the protocol, write a work-request descriptor (a small record of what to send and where), then ring the doorbell (a write to a NIC register that says new work is waiting). On a GPU each step needs branches, global-memory accesses and system-scope memory fences, and a GPU executes one wave of threads in lockstep, so one thread doing serial work leaves its partners idle or divergent.

The cab also has few seats. A GPU has many more threads that want to send than a NIC has queue pairs, so GDA allocates many queue pairs and the threads still have to synchronize to share them. Reverse offload changes who sits in the cab. The crew keeps building and only radios the request; a CPU core, which runs serial code with branches and fences well, works the levers. The number of NIC endpoints now tracks the small number of CPU proxy threads, not the large number of GPU producers. The crane still swings the load straight from tower to tower: in both designs the NIC moves the payload directly between GPU memories, so only the control path moves to the CPU, never the data.

The radio only helps if talking is cheap, and older proxy designs were not. Earlier GPU-to-CPU queues claim each slot with an atomic operation, and traditional proxies put every GPU producer on one shared queue; in the paper's tests their push cost grows 4.6–16× as producers go from 1 to 1,024 waves. KIWI gives each wave its own ring of 8-byte slots in host memory. Every slot pairs 4 bytes of request description with a 4-byte signal written in the same store, so when the CPU sees a valid signal it knows the bytes beside it have arrived; no separate flag write and no fence are needed. The producer keeps its index and a cached copy of the consumer index in registers, and reads the real consumer index across the PCIe link between GPU and CPU only when the ring might be full. On the CPU side, an idle queue is checked with one load that usually hits in the L1 cache, measured at 0.87 ns, so a few proxy threads can watch many queues.

Here is what one push costs on the hardware the paper used: one MI300X wave of 64 lanes, a PCIe Gen5 x16 link, and a single request. Each lane writes 16 bytes, which is two 8-byte slots, and each slot carries 4 bytes of description. So one wave-wide store publishes 64 × 2 × 4 = 512 bytes of request in a single instruction. The measured median push is 0.12 µs, identical to a raw store with no queue logic at all, and it stays at 0.12 µs when 1,024 waves push at the same time, each into its own ring, with no atomic to contend on. In the same harness, UCCL-EP's queue, the fastest existing design at that load, takes 5.56 µs per push at 1,024 waves, with each wave using its own queue, so a KIWI push is about 46× cheaper under contention (5.56 ÷ 0.12). The wave goes back to packing tokens after 0.12 µs; the rest of the measured 1.92 µs round trip is PCIe travel plus the CPU's pop and reply, while the GPU keeps working.

GPU-to-CPU queueRound trip, p50 (µs)Push, 1 wave (µs)Push, 1,024 waves, own queue each (µs)Throughput, 256 waves (Mops/s)Source
KIWI (per-wave SPSC ring)1.920.120.12392.0Table III
MSCCL++2.720.927.32119.4Table III
UCCL-EP2.801.005.56111.3Table III
NVSHMEM (re-implemented on AMD)2.680.9214.9255.7Table III
RCCL GIN3.861.326.0439.1Table III

The second half of the idea is the custom radio code. In a traditional proxy, the GPU can only ask for fixed library operations such as put, get and quiet, so a new pattern has to be composed from many small requests or patched into the vendor library. KIWI exposes one call, invoke, that runs any host function the kernel author registered, so a whole protocol step becomes a single request. KIWI-EP uses three of them for an MoE layer: a multicast put that sends one token to several GPUs, a count all-to-all that exchanges how many tokens each rank will receive, and a barrier that stops the next dispatch from overwriting staging buffers too early. The GPU kernels shrink to packing, posting and unpacking data.

The paper's ablation on eight nodes shows where each part pays. At one token per GPU, switching from GDA to a proxy with plain puts already cuts latency by 27%, because the GPU only writes a description instead of building the RDMA work request and ringing the doorbell itself; per-wave queues cut another 14%. At 64 tokens per GPU the plain proxy versions are 28–29% slower than GDA, because uneven work across proxy threads becomes the bottleneck. Custom callbacks shrink the proxy work by fusing each payload with its signal and coalescing multicasts and collectives (a 21% cut), and a skewed round-robin mapping of endpoints to proxy threads spreads the load (a further 19% at 8 tokens, 9% at 64). End to end, KIWI-EP is 1.77× faster than GDA at one token per GPU, 1.29× at eight and 1.08× at 64.

The gain is largest exactly where agent-style decode lives: few tokens per GPU, where control overhead, not bandwidth, sets the latency. Against ROCm DeepEP, MORI-EP and UCCL-EP with DeepSeek-V3's MoE shape (hidden width 7,168, 256 experts, top-8 routing, BF16), KIWI-EP has the lowest dispatch-plus-combine latency up to 16 tokens per GPU on two nodes, up to 32 on four, and at every measured token count on eight. Past that point RDMA bandwidth dominates, and KIWI-EP sends each token once per destination GPU while MORI sends it once per destination node and forwards it inside the node, which the authors leave as future work. In vLLM serving GLM-5.3 in FP8 with prefill/decode disaggregation, using KIWI-EP for decode lowered time per output token at every scale and concurrency tested, by up to 29% on eight nodes, with GSM8K and needle-in-a-haystack accuracy unchanged. The costs are real but small: a few dedicated CPU cores per GPU, running as ordinary preemptible operating-system threads, and the results so far come from one AMD cluster.

Goes deeper in: GPU & CUDA → Execution Model → Warps: 32 Threads in Lockstep

Related explainers

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based