Phrase-Level Speculative Decoding — Drafting in phrases, not tokens — What does it mean?

The news. On November 9, 2026, the arXiv paper Phrase-Level Speculative Decoding on a Frozen Target (2611.12464) reported a 21.94M-parameter phrase head (0.545% of Qwen3-4B) that speeds up greedy decoding by 2.391× on GSM8K and 2.444× on MATH-500 (two maths benchmarks) and 1.544× on MT-Bench (a chat benchmark), on one A5000 GPU at batch 1 against Hugging Face generate(). The served output stays identical to greedy decoding of the target, up to floating-point tie-breaking. The single-author paper frames its claim as parameter efficiency, not raw speed: an 8,000-entry head is 64× smaller than EAGLE-3's drafter, and neither head beats EAGLE-3's largest tree setting. Read the paper →

Picture a phone keyboard whose suggestion bar can only offer one piece of a word at a time. You type "We want", it offers "the", you accept, it offers "number", you accept, and so on. Every tap still saves you work, but each tap can never save more than one piece. That is the position of the drafters the paper compares against: Medusa, EAGLE-3, DFlash and DSpark all draft in the target model's own vocabulary, so each accepted draft slot carries at most one target token. Their only lever is more guesses per pass, through deeper or wider candidate trees or longer draft horizons.

The reason the unit is stuck at one token is the tokenizer, not the drafter. Because the BPE pretokenizer used by Qwen3-4B splits text on whitespace before any merges are learned, no multi-word string can become a single token in its vocabulary. In Qwen3-4B, "United States of America" is four tokens, and the paper notes that no adjacent pair of them exists as one entry. Retraining the tokenizer to allow cross-word tokens (the SuperBPE approach the paper cites) means pretraining the model again from scratch.

The phrase keyboard fixes this without touching the writer. The draft head reads the frozen target's hidden state and picks one entry from its own separate vocabulary, where an entry can be one to three words, such as " the number of". Greedy longest-match expansion then spells that entry back into the target's own tokens, exactly as tapping a chip types out its letters. The target checks N such candidates in one forward pass behind a prefix-tree mask, and keeps a candidate only while it matches the target's own top choice at each position (the standard verification rule for greedy decoding). Because the target decides every kept token, a wrong phrase can cost speed but cannot change the output.

The paper writes the payoff as tokens per pass = 1 + landing × target tokens banked per accepted unit, where the 1 is the token the target produces on its own every pass. An earlier word-level drafter from the same author was accepted 71.1% of the time but banked only 1.049 target tokens per accepted guess, so acceptance was not the limit; the size of the unit was.

In the classic loop the phrase head replaces, each draft slot proposes one target token, and the run stops at the first mismatch. The diagram below shows that token-at-a-time loop.

Here is where the time goes on GSM8K, using the paper's own measurements for the 21.94M head. Plain decoding runs one forward pass per token at 36.06 ms per token. With phrase drafting, each pass checks about 46 candidate positions and costs 41.16 ms, only 5.10 ms more, but banks 2.730 tokens on average. Divide and each token now costs 41.16 ÷ 2.730 = 15.08 ms, and 36.06 ÷ 15.08 gives the headline 2.39× speedup. Plugging the same head's reported acceptance rate on GSM8K (0.777) into the formula above, each accepted phrase banks roughly (2.730 − 1) ÷ 0.777 ≈ 2.2 target tokens, about twice what the earlier word-level drafter got from a similar acceptance rate.

The phrases the head actually learns are mostly not English. Its vocabulary is mined from maths answers the target itself generated, and punctuation and LaTeX entries supply 57–63% of the tokens it banks on maths, while plain-word entries supply only 5.5–7.3%. The drafter is harvesting the predictable formatting skeleton of mathematical writing. That also explains the weaker conversation result: on MT-Bench, which no head was trained for, the acceptance rate falls from 0.817 (on MATH-500) to 0.509 because there is no formatting skeleton to predict.

The draft vocabulary is also the parameter budget. Each entry costs one output row of 257 parameters, so an 8,000-entry head is 3.434M parameters (0.0854% of the target) and keeps 92.3% of the larger head's GSM8K tokens per pass. The table below uses the paper's head-to-head run, where both methods are divided by the same baseline, so its speedups differ from the headline figures above.

The durable idea is that the drafter's unit of guessing is a free design choice, separate from the target's tokenizer, and the paper says it is orthogonal to tree width and depth. The evidence so far is narrow, and the paper lists the limits itself: one GPU, batch 1, greedy decoding only, 40–50 prompts per set, maths-only training, and no confidence intervals. The authors also point to languages where the gain should be larger: Qwen3-4B spends 1.38 tokens per English word but 5.00 per Hindi word and 11.73 per Telugu word, so a word-sized guess carries many more target tokens there. Their own Indic runs used a lower-fertility target, so that larger case is still largely untested.

Goes deeper in: LLM Serving → Speculative Decoding → Draft Model Variants