LoopSpec drafts the next token while it verifies this one — Pipelined self-speculative decoding — What does it mean?

The news. On September 15, 2026, a paper introduced LoopSpec, a training-free self-speculative decoding framework built for looped Transformers. It extracts draft tokens from early recurrent states and runs them in a pipeline, overlapping draft generation of future tokens with target verification of the current token. It adds a selective second proposal from a deeper recurrent depth to improve draft accuracy without excessive compute overhead, keeps decoding lossless under both greedy and sampling regimes, and derives the optimal proposal depths in closed form, reporting that the prediction matches measurement. The authors report up to 6.83× inference speedup across reasoning and coding benchmarks. Read the paper →

Stay in the print shop for a moment. A looped Transformer buys its small parameter count by owning one press instead of a tall stack of them — and it pays for that at decode time, because the same weights have to be fetched again at every recurrent depth. The paper's own framing is blunt about the trade: looped models decode more slowly than standard Transformers of comparable parameter size, precisely because the shared stack is accessed on every pass. Emitting one token at a time is already the part of inference that leaves the hardware waiting on memory rather than doing arithmetic; a looped model pays that wait once per recurrent depth rather than once per token.

The opening the paper takes is that a half-finished page is not useless. A few passes in, the page is already legible enough to guess what comes next — and the model runs those early passes regardless, so the guess rides along on work it was doing anyway. That is the draft-verify idea turned inward onto the model's own depth: an early recurrent state emits a draft token, the full-depth pass plays the role the target model plays in ordinary speculative decoding, and the verification algorithm decides how many of those drafts survive. Because the draft comes from the same weights, there is no second model to train, host, or keep aligned with the target — which is the standing cost of a separate draft model.

What is actually new is the timing, not the drafting. Ordinary speculative decoding is a relay — draft, then verify, then draft again — so the drafting sits squarely on the critical path, the chain of work that nothing else can run alongside; LoopSpec overlaps the two instead. While the press is finishing the current page, the apprentice is already reading an early pull and guessing the next one: draft generation for future tokens runs against target verification of the current token, in time the machine was spending regardless. The consequence is a different design pressure. A serial scheme has to keep its drafts cheap enough to be worth their own latency, which pushes the draft toward shallow, inaccurate guesses. A pipelined one leans harder on keeping them accurate, because much of their cost is absorbed by a pass that was running anyway.

An early pull can still be too muddy to read, and a rejected guess means discarded work. LoopSpec's answer is a selective second proposal: a second, deeper read taken selectively rather than on every token, so that draft accuracy improves without excessive compute overhead. The abstract describes the second proposal as selective but does not spell out what triggers it. Decoding stays lossless under both greedy and sampling regimes — greedy output is preserved exactly, and under sampling it is the output distribution that is preserved, not any one sampled sequence. And where to read the draft from is not left to guesswork: the authors derive the optimal proposal depths in closed form — an exact formula rather than a search — and report that the closed-form prediction matches what they measure.

Put numbers on the two schedules. The figures here are illustrative — chosen to show how the parts compose, not measured from the paper — and the unit is one application of the shared block stack. Hold three things fixed: recurrent depth 8, so a fully verified token costs 8 applications; drafts read from depth 2; and a round that drafts 2 tokens, both of which verification accepts, yielding 3 tokens (the two drafts plus the token the verification pass produces itself). The work is the same under either schedule: 2 drafted tokens × 2 applications = 4, plus 8 for the verification pass, so 12 applications for 3 tokens. What the pipeline changes is elapsed time. Run it serially and the round takes 12 application-durations — 4 per token, against 8 for plain decoding. Overlap them and the 4 drafting applications for the next round run alongside this round's 8 verification applications, so the round elapses in 8 — 2.67 application-durations per token instead of 4, from exactly the same draft quality and the same total work. The pipeline did not make the guesses better or cheaper; it stopped making you wait for them. That only holds in steady state — it ignores pipeline startup and drain — and only while the drafting work can genuinely fill idle capacity during verification. The paper's headline 6.83× is a benchmark result, and needs far more going right than this one clean round.

Goes deeper in: LLM Serving → Speculative Decoding → The Draft-Verify Idea

Related explainers

• Self-speculative decoding with quantized self-drafters — the other way to draft from the target model itself: quantize the same weights instead of stopping early.
• Looped depth reuse — why models reuse one block stack across depths in the first place, which is the setting LoopSpec optimizes.
• Instance-adaptive draft block sizing — the neighbouring knob: how many tokens to draft per round, rather than how deep to draft from.