NVIDIA sizes speculative-decoding drafts to the GPU's attention tile — Tile-aligned draft length — What does it mean?
The news. On September 2, 2026, NVIDIA published a co-design guide for speculative decoding that separates two things the field usually reports together: how good a draft is, and how much faster the server actually runs. It compares external draft models, EAGLE-3, MTP, DFlash, and suffix or n-gram methods, and it ties the choice of draft length to batch size and GEMM tile geometry rather than to acceptance alone. Read the guide →
The shuttle leaves either way
Picture the 128-seat shuttle. Eight requests are waiting, and each one brings the token it is decoding plus however many tokens the drafter guessed ahead. If the drafter guessed three, each request takes four seats, the eight of them fill 32 seats, and the shuttle still departs with 96 empty. The trip costs the same.
That shuttle is one attention tile. Speculative decoding has a small drafter propose D tokens and the large target model verify all D + 1 positions in one parallel pass, preserving the target model's own output distribution. The verification pass is therefore a matrix multiply whose row count is the batch size G times (1 + D). A GPU does not size that multiply to your row count; it runs the multiply in fixed tiles, and a tile that is only part-full still costs a whole tile — the general effect the Dimension Alignment step calls tile quantization. NVIDIA's rule follows from it: when attention dominates the step, choose D = 128 / G - 1, so that G × (1 + D) lands exactly on a 128-row tile.
Why a longer draft moves the bottleneck
Decoding one token at a time is memory-bound: the GPU reads the whole weight matrix to produce a single token per request, and the arithmetic is trivial next to that read. Drafting changes the ratio, because the weights are read once no matter how many rows ride along. Every extra drafted row adds arithmetic without adding a weight read, so a longer draft pushes the same step toward the compute-bound side of the roofline. NVIDIA puts a number on the shift for a representative 6144 × 6144 GEMM: a draft length of 7 needs one-eighth the batch size that a draft length of 0 needs before the step turns compute-bound. Draft length and batch size are not two knobs. They are one knob seen from two sides.
Prefill vs Decode on the Roofline
Same GPU, fundamentally different bottlenecks
What a half-empty tile costs
Work the shuttle numbers through, holding the batch at eight concurrent requests and the tile at NVIDIA's 128 rows (the batch and the draft lengths below are illustrative). With a draft length of 3, each request contributes 1 + 3 = 4 rows, the verification pass has 8 × 4 = 32 rows, and the hardware still schedules a 128-row tile: 96 of the 128 rows are padding you pay for. Raise the draft length to 15 and each request contributes 16 rows, so the pass has 8 × 16 = 128 rows and the tile is exactly full — about four times as many token positions verified inside the same tile of work. The catch sits on the other side of the ledger: those twelve extra positions per request are only worth having if the target model accepts them, and acceptance generally falls as the draft runs longer. Tile alignment makes the extra positions cheap to compute; it does not make them likely to be right.
| Draft family | Training cost | What it is |
|---|---|---|
| External draft model | ~10 million to more than 10 trillion tokens, depending on strategy (NVIDIA) | A separate small model, trained and then served alongside the target |
| Learned attached draft | ~1 billion to 10 billion tokens (NVIDIA) | A drafter built into the target model instead of run beside it |
| Suffix / n-gram | None, by construction — the method copies text rather than learns it | A lookup that proposes continuations already seen in the prompt or recent output |
Acceptance is not speedup
The guide's framing is that acceptance quality and end-to-end speed are two different measurements, and that a method can improve the first while the second stays flat. Acceptance tells you how good the drafter is; wall-clock speed also depends on whether the shape of the verification pass fits the hardware and on how loaded the server already is. Read the tile rule the other way and it cuts both ways: a drafter that lifts acceptance but pushes G × (1 + D) just past a tile boundary buys a whole second tile for a handful of rows. That is the co-design part — the draft is sized for the machine it runs on, not in the abstract.
Goes deeper in: LLM Serving → Speculative Decoding → When to Use It