GPU·

GC-MoE on consumer Blackwell — Green Context SM partitioning — What does it mean?

The news. On November 12, 2026, Alibaba researchers posted GC-MoE on SM120 (arXiv 2611.16219), a rebuild of their GC-MoE operator for consumer and workstation Blackwell GPUs (compute capability 12.0, such as the RTX PRO 5000) linked over PCIe. On eight RTX PRO 5000 GPUs, the paper reports the operator 1.522× faster on average than FlashInfer's CuTe DSL W4A8 MegaMoE and 1.387× faster than Cake MegaMoE across 18 decode points and two model shapes, with all 94 numerical cases passing. Read the paper →

Picture a kitchen where no dish can start until a runner brings ingredients from the kitchens next door. Cooks who are waiting do not leave their station; they stand there and keep checking the counter. If the cooks arrive first and take every station, the runner has nowhere to unpack, so the cooks wait for ingredients that can never arrive. The fix is a fence: six stations belong to the runner, however many cooks show up.

An MoE layer under expert parallelism is that kitchen. Each layer does three dependent jobs: dispatch the tokens to the GPUs that own their experts, run a grouped GEMM over the experts, and combine the results back. At decode time only one to a few hundred tokens per GPU are in flight, so each job alone leaves most of the GPU idle (the decode worst case). GC-MoE removes the barriers between the jobs: it captures the layer as one replayed CUDA Graph of three role kernels, and an expert starts multiplying as soon as its last token row lands, signalled by a readiness word in memory, not when the whole dispatch is done.

That overlap has a catch. The compute kernel's blocks wait by spin-waiting, so they hold their SMs while they wait. If those waiting blocks were allowed to fill the GPU, the dispatch kernel that feeds them could run out of SMs. GC-MoE fences the SMs with CUDA Green Contexts: it queries the device's SMs, splits them by count, and builds two contexts with their own streams, a dispatch context (6 SMs by default on SM120) and a compute context that must keep at least 32 SMs. Both contexts request a balanced work queue, which the paper says guarantees that two co-resident Green Contexts both make forward progress instead of one starving the other. The paper contrasts this with earlier overlap work that relies only on independent hardware queues.

Way to share one GPUDo dispatch and compute overlap?Who decides which SMs each kernel gets?Can a spin-waiting consumer starve its producer?
Separate kernels with a barrier between phasesNo, each phase waits for the lastThe hardware scheduler, one kernel at a timeNo, nothing spins, but the GPU idles between phases
Two CUDA streamsYesThe hardware scheduler, from independent queuesNot ruled out: the paper names forward progress as what streams alone do not guarantee
Green Context partitions (GC-MoE)YesYou, by SM count, before the graph is builtNo: the producer's SMs are reserved, and a balanced work queue keeps both sides moving

Your code creates:

Grid

B0

256 threads

B1

256 threads

B2

256 threads

B3

256 threads

B4

256 threads

B5

256 threads

→GPU
assigns

GPU hardware runs:

SM0
B0B3
SM1
B1B4
SM2
B2B5

You create blocks — the GPU decides which SM runs each one

The fence solves the starvation between the two kernels, but there is a second trap inside one partition. GC-MoE's role kernels synchronize grid-wide through counters in device memory, so every block of a role must be seated on an SM at the same time, like a dish that needs every cook present. If a grid asks for more blocks than the partition can seat, the seated blocks wait forever for blocks that can never be scheduled, which is a deadlock rather than a slowdown. So GC-MoE checks residency instead of assuming it: before it instantiates a graph, it asks the driver how many blocks per SM fit at the role's exact thread count and shared-memory footprint, and rejects any graph that does not fit.

Here is how thin that margin is, as a hypothetical. Hold the compute partition at its minimum of 32 SMs and give a role a grid of 64 blocks (illustrative). If the role's footprint fits four blocks per SM, the partition seats 32 × 4 = 128 blocks, so all 64 are resident and the rendezvous completes. If its shared-memory footprint instead limits it to one block per SM, the partition seats only 32 of the 64; the other 32 cannot be scheduled while the seated blocks spin at the rendezvous, so the job hangs instead of running slowly. The paper notes that a few kilobytes of shared memory can decide between those two occupancy levels, which is why GC-MoE picks its tile geometry per problem shape and rejects a graph that does not fit. Because the dispatch context holds its 6 SMs, the compute grid cannot borrow them either, which is exactly what keeps the runner's stations free.

Goes deeper in: GPU & CUDA → Execution Model → Mapping to Hardware: SMs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based