The news. On July 16, 2026, a paper introduced D-Cut, which targets the case where long speculative drafts help single requests but waste verification compute under high concurrency. Rather than verifying a fixed depth for every request, it prunes draft tokens across the whole batch, reporting that the average speedup under high concurrency improves from 1.26× to 1.65×, and reaching up to 3.0× over autoregressive decoding on MoE models. Read the paper →

Picture an airport shuttle that leaves every ten minutes whether or not anyone is on it. Six passengers hold reservations. There are ten seats, so the driver waves in four standbys from the terminal — it costs the airline nothing, the van was leaving anyway, and if a standby turns out not to belong on this run, no harm done. Now picture the same shuttle at peak hour, with all ten seats reserved. The standbys are still there, still hopeful. But letting one aboard now means a passenger with a reservation gets left at the curb. Nothing about the standby changed. What changed is that the empty seat was never really free — it was just unclaimed.

That is speculative decoding under load, and swapping the van for a GPU changes almost nothing. The trick is well known: a small draft model guesses several tokens ahead, and the big model checks all of the guesses in one verification pass. The reason it works on a quiet server is a roofline fact, not a clever algorithm — this next part is the standard background the paper is written against, not a finding of it. At small batch, decode is memory-bound: the GPU spends its time hauling weights out of HBM, the high-bandwidth memory where the model lives, while its math units sit idle, so checking a handful of candidate tokens costs barely more than checking one. The draft tokens are standbys on an empty van: they ride in the slack.

Fill the van and the arithmetic inverts. As concurrency rises, the batch stops being memory-bound and becomes compute-bound: the math units are now saturated with real work from real requests. A rejected draft token is no longer riding in slack, because there is no slack — it is occupying compute that a paying request needed. The paper's own framing lands on the same place: long drafts help single requests but waste verification compute under high concurrency, and it puts the average speedup there at 1.26×. This is where the fixed-depth baseline shows its age. It picks one static draft length and hands the same allowance to every request in the batch, which means it spends identical verification budget on a request whose draft is nearly certain and one whose draft is a coin flip.

D-Cut's move is to stop treating the draft depth as a per-request setting and start treating it as a batch-wide budget. It ranks the candidate draft tokens across concurrent requests using draft confidence — the draft model's own probability, available before the expensive verification runs — and then spends the verification budget where acceptance is most likely. In shuttle terms: instead of giving every hotel pickup the same three standby slots, run one standby list for the whole van, sorted by who is most likely to be confirmed, and cut it where the seats run out. Some pickups get three seats, some get none. The list, not the quota, decides.

The second half of the mechanism is the part that keeps it from being another hand-tuned constant. How deep you can afford to cut is not a property of the model, it is a property of the machine. A runtime cost model adapts the pruning depth to the GPU architecture and the parallelism strategy, so the policy changes with the deployment environment instead of using one static draft length. The paper reports gains on dense and MoE models and names MoE verification cost as its own concern — and MoE is where it reports the largest gain, up to 3.0× over autoregressive decoding, though the summary does not spell out what makes that gap wider than on a dense model.

Hold the seats fixed and the allocation is easy to see. Take a verification budget of 4 slots and three concurrent requests, each with 3 drafted tokens — nine candidates, four seats (illustrative: the paper does not publish this breakdown). A single fixed depth has no good setting here: depth 1 spends a seat on the coin-flip request while a nearly-certain draft two rows down goes unverified, and depth 2 already asks for 6 seats when only 4 exist. Now rank all nine by draft confidence and cut at 4. Suppose the ordering runs 0.94, 0.91, 0.88, 0.62 above the line and 0.44, 0.31, 0.22, 0.12, 0.09 below it: the survivors are 2 tokens from request 1, 1 from request 2, and 1 from request 3. Nothing stops you writing a 2/1/1 quota by hand — the point is that you would have to write it before seeing any of the confidences, and next step the right split is 1/0/3. A fixed depth has to be chosen in advance; a ranking is chosen after the drafts exist, which is the only time the answer is knowable. The five below the line are dropped unverified, and that is the win: they were the guesses most likely to be thrown away anyway, and now they cost nothing.

The reason this reads as a scheduling result rather than a decoding one is that it moves the decision up a level. A draft length is a knob on a request; a verification budget is a resource the whole batch competes for, and once you see it that way the question stops being "how far ahead should we guess" and becomes "given what we are about to spend, which guesses are worth checking". That reframing is what turns speculation from a single-request trick back into something a busy server can afford.

Goes deeper in: LLM Serving → Speculative Decoding → When to Use It

Related explainers

• Spec-decode latency paper — Load-dependent latency model — the diagnosis this paper is the cure for: why the speculative win shrinks as a server saturates
• DominoTree — Conditional draft tree scoring — spends a single request's draft budget better, where D-Cut spends the batch's
• VIA-SD — Tiered speculative verification — a different lever on the same cost: make verification itself cheaper rather than skipping it