Fuse decode into one megakernel for 1.58× higher H100 throughput — Persistent decode megakernel — What does it mean?

The news. On September 8, 2026, Cohere published what it describes as the first fully fledged serving system built around a decode megakernel: one persistent CUDA kernel that runs an entire decode step for North Mini Code, its 30B-parameter agentic coding model with 3.3B parameters active per token. On a single H100 at batch size 1 it decodes at 292 tok/s against vLLM's 185, a 1.58× gain, while still supporting continuous batching, paged attention and an OpenAI-compatible endpoint. Across five end-to-end serving benchmarks at batch size 8 the average decode gain is 1.25× to 1.41×. Read the post →

Picture that kitchen at the peak of service. In the old arrangement the chef blows a whistle after every course: no cook may start the main until every cook has finished the appetizer, and only then does the chef walk the room handing out the next set of recipe cards. The kitchen therefore moves at the speed of its slowest cook, once per course. That is exactly what a GPU does at a kernel boundary — every SM must finish before any SM starts the next kernel, and the driver has to dispatch the next grid (the kernel launch problem).

A decode step is dozens of those courses per layer: RMSNorm, QKV projection, attention, the router, the MoE experts, the output projection. Each course is small, so the whistle blows almost constantly, and decode is the worst case for exactly this reason.

In the megakernel kitchen nobody leaves their station. Cohere launches exactly one threadblock per SM and keeps it resident for the entire decode step. Each cook then works from a personal ticket rail — a task list sitting in global memory, written by the host before the step starts. A ticket does not say "wait for the kitchen"; it says "wait until this counter reaches 3, then run this tile." The counters are plain integers in global memory: a finishing task bumps one with atomicAdd, and a waiting task spins on one until it is high enough.

The result is a change of unit, and that change is the whole idea. The unit of scheduling shrinks from an entire operation to one tile of one operation, and the unit of synchronization shrinks from the whole GPU to the specific producers a task actually depends on.

You have met a smaller version of this move already. Operator fusion merges two or three neighbouring operations into one kernel so the intermediate result never round-trips through HBM. A megakernel applies the same move at the scale of the whole step — not two operations but every operation, under a single launch — though it buys back something different, and the operations themselves are unchanged. Each one keeps its ordinary tiled implementation and still writes its output tiles to HBM; what disappears is the boundary between them.

So the two are worth keeping apart. Fusion mostly buys back memory traffic; a megakernel mostly buys back *stalls* — the launch gap, the full-grid barrier, and the SMs that a kernel boundary leaves idle.

Cohere names three stalls the megakernel removes, in rough order of impact.

First, wave quantization. Suppose a kernel has 200 tiles of work and the GPU has 132 SMs. The first 132 tiles run together; the remaining 68 run in a second wave while 64 SMs sit idle. The kernel spends two waves of wall-clock doing 1.5 waves of work, and the smaller the kernel, the worse that rounding gets. You cannot tune it away, because GEMM tile shapes are fixed by the matrix dimensions and rarely land on an exact multiple of the SM count. Inside a megakernel there is no boundary to round up to: a tile whose inputs are ready simply starts on whichever SM is free. (The tail-wave arithmetic gets its own explainer.)

Second, false dependencies. A kernel boundary is a full-grid barrier, so the slowest SM sets the pace even for work that was ready long ago. If attention is split across four key/value groups and one group finishes early, that SM idles until the other three catch up — even though the output projection for its own group could already start. Fine-grained counters drop that false dependency: O-proj for a KV group runs as soon as that group's attention output lands.

Third, weight prefetch. Weights are immutable; they do not depend on this step's activations at all. A task can therefore start streaming its weight tiles from HBM into shared memory before its activation dependency is satisfied, which a kernel boundary would forbid. Cohere leans on this hardest for the router and the QKV projection, which pull their weights during the tail of the previous layer's output projection. A task blocked on its inputs is still moving bytes.

Where the bandwidth actually goes. Hold the model and the hardware fixed. North Mini Code in BF16 activates 3.3B parameters per token, which means streaming 6.6 GB of weights on every decode step, plus roughly 0.5 GB of KV cache at 8K context — about 7.1 GB moved per token. An H100 pushes 3.35 TB/s through HBM, so the floor for one step is 7.1 GB divided by 3.35 TB/s, or about 2.1 ms; that is Cohere's ~470 tok/s speed-of-light. vLLM's 185 tok/s works out to 5.4 ms per step and the megakernel's 292 tok/s to 3.4 ms — 3.3 ms and 1.3 ms respectively above that bandwidth-only floor. The megakernel closes about 2.0 ms of vLLM's 3.3 ms gap — roughly 60% of it — and Cohere credits the glue between operations rather than faster arithmetic. It reports that its own GEMM and attention kernels merely match cuBLAS and FlashAttention-3 standalone. That is Cohere's attribution rather than a controlled decomposition, though: the vLLM baseline runs a Triton MoE backend, so some of the margin could still be operation-level. (The per-step milliseconds are just the reciprocals of the published token rates.)

After the kernels are fast, the schedule is the rest of the speedup. The host deals task k to SM k mod 132 and fixes the wave order for the layer — QKV, then the router, then attention, then the MoE projections, then O-proj and RMSNorm. Only the runtime-dependent work is dynamic: attention and MoE tiles go into shared queues that idle SMs steal from, because their counts depend on live sequence lengths and on which experts the router picked.

Reordering those waves changes nothing about the output and a great deal about the throughput. Cohere's ablation, at 8K input with uniform routing, is the clearest evidence that scheduling is a first-class knob rather than a detail:

That list of three is also what marks the limit of the obvious alternative. CUDA graphs record a fixed kernel sequence and replay it as a single submission, which removes the driver's dispatch cost — the launch gap, and only the launch gap. A captured graph still contains every kernel boundary, so the first two stalls — the idle tail waves and the false dependencies — survive capture completely untouched. The difference fits in one line: CUDA graphs make the boundaries cheaper to cross; a megakernel deletes them.

The price is that the schedule is baked in. Cohere's server parks its C++ decode loop whenever the Python control plane admits, evicts or resizes a batch; prefill still runs as ordinary PyTorch kernels and pauses decoding while it does; and the released configuration tops out at batch size 8. Cohere states all three as implementation limits rather than design limits. Every figure above is also Cohere's own measurement of its own engine, against vLLM v0.24 with the FA3 attention backend and the Triton MoE backend, prefill disabled and both engines decoding against a synthetic KV cache — a setup chosen to isolate decode, and one no third party has yet reproduced. The accuracy check is the part that travels best — SciCode scores 38.9% ± 1.6% against vLLM's 38.2%, and LiveCodeBench v6 lands on 70.3% for both.

Goes deeper in: GPU & CUDA → Operator Fusion & FlashAttention → The Kernel Launch Problem

Related explainers

• Wave quantization — the tail-wave arithmetic a megakernel is built to erase, worked out on the same 132 SMs.
• Distributed KV pool rebalancing — what it costs a serving engine to pause a captured CUDA graph, the technique a megakernel replaces.