What is wave quantization on a GPU?

It is the wasted SM-time that appears when a kernel's tile count is not a multiple of the number of SMs. The tiles run in waves, and the final wave is usually partial, so the kernel occupies the whole GPU for a full wave while only some of its SMs have work. On a 132-SM H100, a 200-tile kernel takes two waves to do 1.5 waves of work.

Is wave quantization the same as quantizing a model to 4 bits?

No — the two share only the word. Numeric quantization shrinks the bit width used to store weights or activations. Wave quantization is a scheduling effect: work rounded up to a whole number of waves across the SMs. Nothing about the model's numerics changes.

Why can't you just balance the work better?

Because it is not an imbalance. Every tile is the same size; there are simply not enough of them to fill the last wave. Tile shapes are constrained by the matrix dimensions and by the kernel's design, so the tile count rarely lands on an exact multiple of the SM count, and reshaping the tiling to force a fit means abandoning the tile shape the kernel was tuned around.

How does a megakernel reduce wave quantization?

By removing the boundary that the wave rounds up to. When one persistent kernel runs the whole decode step, an SM that finishes its share of one operation can immediately start a tile of another operation whose inputs are already ready, instead of idling until every SM reaches the next kernel launch.

GPU· 2026-09-09

Fuse decode into one megakernel for 1.58× higher H100 throughput — Wave quantization — What does it mean?

Jargon

SM: Streaming multiprocessor. One of the independent processors a GPU is built from; an H100 has 132. A thread block is assigned to exactly one SM and stays there. How blocks land on SMs
Tile: One chunk of a kernel's output that a single thread block computes — for a matrix multiply, one rectangle of the result matrix. A kernel's tile count is what gets divided by the SM count.
GEMM: General matrix-matrix multiply — the operation almost every layer of a transformer reduces to. Its output is cut into tiles, and the tile shape is pinned by the matrix dimensions and by the kernel's design, which is the root of the rounding problem here.
Wave: One round of thread blocks running concurrently across the SMs. If a kernel has more tiles than the GPU can host at once, it runs in several waves, and the last one is the one that is usually short.
Wave quantization vs numeric quantization: Unrelated ideas that share a word. Numeric quantization shrinks the bit width of weights or activations, as in global quantization granularity. Wave quantization is scheduling: work rounded up to a whole number of waves.
Occupancy: How many warps an SM holds resident at once, set by register and shared-memory use. High occupancy hides memory latency within a wave; it does nothing about a wave that is half empty, which is why the two are separate problems.
Tail effect: The general name for the last, short wave of a kernel. Wave quantization is the tail effect measured as wasted SM-time.
Megakernel: A single persistent kernel that runs a whole forward pass, so there is no kernel boundary for a wave to round up to. Full explainer

The news. On September 8, 2026, Cohere published a serving engine that runs an entire LLM decode step as one persistent CUDA kernel, reaching 292 tok/s against vLLM's 185 on a single H100 at batch size 1. Wave quantization is the first of the three stalls it lists as motivation — ahead of false dependencies and weight prefetch — and the post gives it a plain arithmetic example on the H100's 132 SMs. The concept long predates the release, and the post does not isolate what share of its 1.58× comes from this stall specifically — that figure also carries launch overhead, false dependencies, weight prefetching and scheduling. What the post gives is an unusually clear illustration of the arithmetic. Read the post →

Two hundred cars are queued at the dock and the ferry holds 132. The first crossing sails full. The second sails with 68 cars and 64 empty slots, and it takes exactly as long as the first one did — a ferry does not cross faster because it is half empty. You have paid for two crossings and moved one and a half crossings' worth of cars.

A GPU kernel does the same thing. The work is cut into tiles, each tile handed to one thread block, and blocks are assigned to SMs — an H100 has 132 of them. If a kernel has 200 tiles, the first 132 run together and the remaining 68 run in a second wave while 64 SMs sit doing nothing at all. The kernel's wall-clock is two waves; its useful work is 1.5 waves.

The obvious reaction is to spread the work more evenly, and the obvious reaction is wrong. This is not load imbalance, where one worker got a bigger share. Every tile here is the same size. The tile count is what it is because GEMM tile shapes are constrained by the matrix dimensions and by the kernel's own design, so the total rarely lands on an exact multiple of the SM count — and reshaping the tiling to force a fit means giving up the tile shape the kernel was tuned around.

Your code creates:

Grid

256 threads

→GPU
assigns

GPU hardware runs:

SM0

B0B3

SM1

B1B4

SM2

B2B5

You create blocks — the GPU decides which SM runs each one

The waste is a fixed cost of at most one wave, so it hurts in inverse proportion to the size of the kernel. Hold the GPU fixed at 132 SMs and walk three kernels through it.

A kernel with 200 tiles needs 2 waves for 200 ÷ 132 = 1.52 waves of work: 76% of the SM-time is used, and a quarter of it is idle. Now shrink the kernel by a lot. A kernel with 133 tiles — one single tile past a perfect fit — still costs two full waves to do 1.008 waves of work: 50.4% of the SM-time used, with 131 of the 132 SMs idle through the entire second wave. Grow it instead and the waste dilutes on its own: 1,000 tiles needs 8 waves for 7.58 waves of work, 94.7% used.

Kernel size (132 SMs)	Waves needed	Work done	SM-time used
200 tiles — Cohere's example	2	1.52 waves	76%
133 tiles — illustrative arithmetic	2	1.01 waves	~50%, illustrative
1,000 tiles — illustrative arithmetic	8	7.58 waves	~95%, illustrative

That table is why wave quantization is a decode problem specifically. Prefill runs large compute-bound matrix multiplies with far more tiles than there are SMs, so it lives in the bottom row of that table and the tail wave shrinks to a small fraction of the runtime. Decode at small batch is dozens of small kernels per layer — QKV, attention, the router, each expert projection — and each one rounds up to its own whole number of waves, which is one of several reasons decode is the pathological case for a serving engine.

It is also why the fix is structural rather than a tuning knob. Nothing inside a single kernel can spend the idle SMs of its own last wave; there is no other work in scope. A megakernel changes the scope: with the boundaries gone, a tile whose inputs are ready simply starts on whichever SM is free, so the gaps in one operation's tail get filled with tiles from another operation. Cohere's model helps here in a way not every architecture would — North Mini Code uses parallel transformer layers, where attention and the mixture-of-experts (MoE) feed-forward both read the same normalized input and are rejoined only at the end of the layer, so neither branch waits on the other and there is usually another dependency-ready tile available to backfill with. Cohere places those backfill tasks where they are likely to be ready, not where they are guaranteed to be.

Goes deeper in: GPU & CUDA → Execution Model → Mapping to Hardware: SMs

Related explainers

Persistent decode megakernel — the structural fix: one resident kernel per SM, task lists in global memory, and counters instead of kernel boundaries.
Global quantization granularity — the other quantization, the numeric one, for readers who arrived here by the word.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based