GC-MoE on consumer Blackwell — Green Context SM partitioning — What does it mean?
The news. On November 12, 2026, Alibaba researchers posted GC-MoE on SM120 (arXiv 2611.16219), a rebuild of their GC-MoE operator for consumer and workstation Blackwell GPUs (compute capability 12.0, such as the RTX PRO 5000) linked over PCIe. On eight RTX PRO 5000 GPUs, the paper reports the operator 1.522× faster on average than FlashInfer's CuTe DSL W4A8 MegaMoE and 1.387× faster than Cake MegaMoE across 18 decode points and two model shapes, with all 94 numerical cases passing. Read the paper →
Picture a kitchen where no dish can start until a runner brings ingredients from the kitchens next door. Cooks who are waiting do not leave their station; they stand there and keep checking the counter. If the cooks arrive first and take every station, the runner has nowhere to unpack, so the cooks wait for ingredients that can never arrive. The fix is a fence: six stations belong to the runner, however many cooks show up.
An MoE layer under expert parallelism is that kitchen. Each layer does three dependent jobs: dispatch the tokens to the GPUs that own their experts, run a grouped GEMM over the experts, and combine the results back. At decode time only one to a few hundred tokens per GPU are in flight, so each job alone leaves most of the GPU idle (the decode worst case). GC-MoE removes the barriers between the jobs: it captures the layer as one replayed CUDA Graph of three role kernels, and an expert starts multiplying as soon as its last token row lands, signalled by a readiness word in memory, not when the whole dispatch is done.
That overlap has a catch. The compute kernel's blocks wait by spin-waiting, so they hold their SMs while they wait. If those waiting blocks were allowed to fill the GPU, the dispatch kernel that feeds them could run out of SMs. GC-MoE fences the SMs with CUDA Green Contexts: it queries the device's SMs, splits them by count, and builds two contexts with their own streams, a dispatch context (6 SMs by default on SM120) and a compute context that must keep at least 32 SMs. Both contexts request a balanced work queue, which the paper says guarantees that two co-resident Green Contexts both make forward progress instead of one starving the other. The paper contrasts this with earlier overlap work that relies only on independent hardware queues.
| Way to share one GPU | Do dispatch and compute overlap? | Who decides which SMs each kernel gets? | Can a spin-waiting consumer starve its producer? |
|---|---|---|---|
| Separate kernels with a barrier between phases | No, each phase waits for the last | The hardware scheduler, one kernel at a time | No, nothing spins, but the GPU idles between phases |
| Two CUDA streams | Yes | The hardware scheduler, from independent queues | Not ruled out: the paper names forward progress as what streams alone do not guarantee |
| Green Context partitions (GC-MoE) | Yes | You, by SM count, before the graph is built | No: the producer's SMs are reserved, and a balanced work queue keeps both sides moving |
Your code creates:
Grid
B0
256 threads
B1
256 threads
B2
256 threads
B3
256 threads
B4
256 threads
B5
256 threads
assigns
GPU hardware runs:
You create blocks — the GPU decides which SM runs each one
The fence solves the starvation between the two kernels, but there is a second trap inside one partition. GC-MoE's role kernels synchronize grid-wide through counters in device memory, so every block of a role must be seated on an SM at the same time, like a dish that needs every cook present. If a grid asks for more blocks than the partition can seat, the seated blocks wait forever for blocks that can never be scheduled, which is a deadlock rather than a slowdown. So GC-MoE checks residency instead of assuming it: before it instantiates a graph, it asks the driver how many blocks per SM fit at the role's exact thread count and shared-memory footprint, and rejects any graph that does not fit.
Here is how thin that margin is, as a hypothetical. Hold the compute partition at its minimum of 32 SMs and give a role a grid of 64 blocks (illustrative). If the role's footprint fits four blocks per SM, the partition seats 32 × 4 = 128 blocks, so all 64 are resident and the rendezvous completes. If its shared-memory footprint instead limits it to one block per SM, the partition seats only 32 of the 64; the other 32 cannot be scheduled while the seated blocks spin at the rendezvous, so the job hangs instead of running slowly. The paper notes that a few kilobytes of shared memory can decide between those two occupancy levels, which is why GC-MoE picks its tile geometry per problem shape and rejects a graph that does not fit. Because the dispatch context holds its 6 SMs, the compute grid cannot borrow them either, which is exactly what keeps the runner's stations free.
Goes deeper in: GPU & CUDA → Execution Model → Mapping to Hardware: SMs