Adaptive Multi-Token Prediction — acceptance-survival draft horizon — What does it mean?

The news. On October 20, 2026, researchers at the Georgia Institute of Technology posted Adaptive Multi-Token Prediction: How far should we go? on arXiv. It treats the number of tokens an MTP drafter proposes per round as an online control problem. The authors report decoding up to 4.1× faster than standard decoding across five draft–target pairings on AIME 2025 and GSM8K, with throughput that matches or beats the best fixed window in 8 of 10 settings, no retraining, and the exact verifier unchanged. Read the paper →

Back to the conveyor belt. The inspector can only approve parcels from the front of the belt up to the first bad one; everything behind a bad parcel is scrap, however good it looks. That is how exact speculative verification treats a draft: the target model scores every drafted position in one pass, accepts tokens left to right, and at the first rejection emits one correction token and discards the rest. A round that drafts k tokens commits 1 plus the length of the accepted prefix, and when all k are accepted the verifier adds a bonus token.

The loader's problem is the belt length. A longer belt spreads the cost of the inspector's trip over more parcels when the stretch of text is easy, and wastes loading effort when a bad parcel shows up early. The best length therefore changes with what is being written. An autoregressive drafter can watch each token as it is produced and stop early. MTP cannot: its heads all fire in the same forward pass, so the horizon must be chosen before any draft token exists, using only what is known at that point. This is also a correctness rule, not just a limit. The paper gives a two-bit example where keeping a second draft token only when it equals 1 makes the second emitted token equal 1 with probability 3/4 instead of 1/2, even with a perfect drafter. Picking k first, from the committed prefix and the controller's own state, keeps the output distribution exact however badly k is chosen.

For the mechanics of the draft and the check, the draft-verify idea step shows the loop this controller sits on top of.

What should the controller predict? Two things decide the best belt length: how many tokens a given length is expected to commit, and what a round costs. For the first, AMTP uses acceptance survival, the probability that the first j draft tokens are all accepted. Survival is the running product of one minus the rejection hazard at each position, and the expected tokens committed for horizon k are G(k) = 1 + s1 + s2 + … + sk. A single vector of hazards, one per position, therefore scores every horizon at once. The identities do not require the draft positions to be independent, so they hold for correlated heads as well as independent ones.

Before each round the controller turns its predicted hazards into predicted survival, divides the expected gain by a predicted round cost, and picks argmax G(k) / C(k). A running product and a running sum score all K horizons in O(K) time, and nothing assumes that throughput rises and then falls with k. The only learned part is the hazard forecaster, an affine function of features known before drafting.

Learning those hazards has one subtle point. The verifier sees how far the draft and target disagree at each position it reaches, but prefixes that start badly rarely reach later positions, so a plain average of disagreement over all draft prefixes is the wrong target. AMTP weights each position's squared error by the probability the prefix survived to it, and after every round takes one online gradient step on that loss. The paper shows the excess loss equals the hazard error weighted by how often each position is reached, and that any learner's prediction error turns into throughput regret and then into a bound on wall-clock completion time; with online gradient descent the regret is sublinear. With a small probability, set by an exploration rate, the controller drafts the maximum horizon so that late positions keep producing feedback.

Here is the paper's own small case, then a cost made up to show the decision. Use a two-letter vocabulary {a, b}. The first draft head proposes a or b with probability 1/2 each, while the target puts 3/4 of its mass on a, so a proposed a is always accepted, a proposed b is accepted with probability 1/2, and survival after one token is 1/2 × 1 + 1/2 × 1/2 = 3/4. The second head matches the target exactly after a, but after b it puts its mass where the target puts none. Only draft prefixes that started with b are rejected at position 2, and they reach it with weight 1/2 × 1/2 = 1/4, so the true hazard there is (1/4) ÷ (3/4) = 1/3. Survival after two tokens is 3/4 × (1 − 1/3) = 1/2, and the expected tokens for horizon 2 are 1 + 3/4 + 1/2 = 2.25. A plain average of the second head's disagreement, 1/2, would give 1 + 3/4 + 3/8 = 2.125, an underestimate, because it ignores that bad prefixes rarely get there.

Now add a cost (illustrative): a round with horizon 1 costs 1.00 units and a round with horizon 2 costs 1.30. Horizon 1 gives 1.75 ÷ 1.00 = 1.75 tokens per unit and horizon 2 gives 2.25 ÷ 1.30 ≈ 1.73, so the controller stays at horizon 1 by a hair. If a context made the second position cheap, say 1.20 units (illustrative), horizon 2 would give 2.25 ÷ 1.20 = 1.875 and win. Same drafter, same target, a different best belt length.

How much does this help in practice? The paper's Table 1 compares AMTP with the largest fixed window each draft supports. The AIME 2025 rows below show the spread, as speed-up over plain next-token decoding.

The gain concentrates where draft and target are separate models and a longer horizon costs real time. When the draft and target share one forward pass, a longer horizon costs little extra, so there is less to win: the paper reports 2–7% on AIME for the self-distilled 4B and Medusa pairings. What changes everywhere is the acceptance rate. On AIME the self-distilled 4B model's acceptance rose from 18% under the fixed window to 34% under AMTP while tokens committed per round stayed at 3.84 versus 3.83, so AMTP drafts fewer tokens per round, commits about the same number, and wastes less of what it drafts. The chosen horizon is not constant either: for the self-distilled 4B on GSM8K it averaged 11.2 and ranged from 1 to 16.

Read the numbers with their conditions. The runs disable KV caching, so verification recomputes the full context each round, each condition is a single unseeded run, and hardware differs across runs, so only within-model comparisons are meaningful. The Meta MTP 7B regression on AIME is attributed to a base checkpoint that does not reliably stop after answering, which makes every round of a longer generation cost more. The abstract's 4.1× headline is measured against plain decoding, not against the fixed window; the fixed-window comparison is the one in the table.

Goes deeper in: LLM Serving → Speculative Decoding → The Draft-Verify Idea

Related explainers

• NVIDIA tile-aligned draft length — draft length chosen for the GPU's tile size rather than for the context.
• D-Cut cross-request draft pruning — one draft budget shared across a whole batch.
• EDR expected decoding rounds — training the drafter to minimize verification rounds.