Speculative decoding can raise energy at large batches — Energy per accepted token — What does it mean?
The news. On October 29, 2026, researchers at the Erlangen National High Performance Computing Center posted "Only Accepted Tokens Count", an energy accounting of speculative decoding. They ran 5 drafting schemes on 11 models on H100 and H200 GPUs, at batch sizes up to 64 across 5 text corpora, and paired every speculative run with a non-speculative baseline at the same batch. Read the paper →
Picture the delivery truck. Each decode step is one trip, and the trip costs the same whether the truck carries one parcel or many, because the cost is driving the route: reading every model weight from memory. Batching loads parcels from many customers onto the same trip, so the cost of the trip is split across all of them. Speculation does something different: it loads extra parcels for the same customer, guessed from what the customer will probably want next. If the guess is right, the customer receives several parcels per trip. If the guess is wrong, the parcel rode along for nothing.
That is the paper's central point: only accepted tokens divide the weight read, so what matters is accepted tokens per verification against the cost of every drafted token, not the fraction accepted. A rejected token was still drafted and still carried through verification. At single stream this costs little, because the weight read dominates the step. The paper measures the best scheme per model at a median saving of 54% of decode energy at batch 1. See why decode is memory-bound for the starting point.
Now fill the truck with other customers' parcels. At a larger batch the weight read is already shared, so there is less left for a drafter to recover, and the drafter's extra verified tokens start to displace useful work. The saving decays as batch grows, for the short contexts the paper serves; the authors note that at long contexts the key-value cache keeps large-batch decode memory-bound and the speedup can instead grow with batch.
The end of the saving has a hardware explanation. A decode step runs about 10 operations per byte, far below the H200's balance point near 163. Speculation raises that number by verifying many tokens per weight read, and batch multiplies it. Once the step passes the balance point, every verified token costs arithmetic time the weight read no longer hides, and the saving becomes a loss, as it did for the block drafters the paper measured. The paper reports that the sign flips within one doubling of the batch at which the step reaches the GPU's balance point, and that a tuned draft length does not carry over to a GPU with a different balance point. This is the same logic as the roofline model: the crossover tells you where an optimization stops paying.
| Condition | What the paper reports | Why |
|---|---|---|
| Single stream, best scheme per model | Median saving of 54% of decode energy | The weight read dominates the step, so accepted tokens split it |
| Batch 64, best scheme per model | Still saves 15 to 52% on every model that ran there | The model matters more than the scheme at a fixed batch |
| Batch 64, n-gram drafter | Costs energy on every model reported except one | Too few tokens accepted to pay for verification |
| Batch 64, code-specialised drafter on conversation | Saves 51% on code; costs energy on conversation | Accepts 2.0 to 3.0 tokens per verification on conversation versus 5.9 on code |
| Qwen3-4B, DFlash-16 verifying 16 tokens | Reaches the H100 balance point at batch 32 and the H200's at batch 16 | Arithmetic intensity grows with batch times tokens verified |
Here is the worked example, using only the paper's numbers. Hold three things fixed: Qwen3-4B, the DFlash-16 drafter that verifies 16 tokens per request, and an H200. At batch 16 the step verifies 16 × 16 = 256 tokens at once. The paper reports that DFlash-16 accepts under 3 of the 16 per request, so fewer than 16 × 3 = 48 of those 256 tokens are accepted, and more than 208 are verified but not accepted (computed from the paper's figures; "accepts under 3" means per verification). This is the batch at which the paper finds the DFlash-16 step crossing the H200 balance point, and DFlash-16 stops saving energy on that GPU. By contrast, DSpark-7 verifies fewer than half as many tokens, so on the H100 at batch 64 its step stays below the balance point and it still saves energy.
What an operator should take from this: batching is the cheap energy lever and speculation is the latency lever. For a service without a latency target, raising the batch is the cheaper way to cut energy per token. When the time-between-tokens target is tight, speculation can be the only option that meets it; under a 25 ms per token objective the paper finds the lowest-energy configuration that meets it uses 18 to 69% less energy with speculation on 10 of 11 models, and gemma-4-31B-it meets it only with speculation. Lowering the GPU clock offers little extra: once a speculative step runs at a batch of 16 or more, the paper measured the GPU already running at 1450 to 1700 MHz under the default limit, so a 1500 MHz lock removed little clock and saved no energy. Choose the number of drafted tokens per GPU and per batch, and measure with CUDA graphs on, since the paper finds a capture bias of up to 39 percentage points on Mixture-of-Experts models.
Goes deeper in: LLM Serving → Speculative Decoding → When to Use It
Related explainers
- Token-Energy Law — Active bytes and decode time — the energy-per-token accounting this paper builds on