What is context-asymmetric speculative decoding?

It is a draft-and-verify serving setup in which the two models are given different amounts of input. In AsymSpec, a lightweight drafter reads the full context while the larger verifier reads a compressed view of it. Because compression drops signals the verifier might need, the drafter influences the verifier's logits through contrastive delta fusion, and a divergence-aware gate decides when the resulting asymmetric draft is still safe to accept. The paper reports about 90% of full-context accuracy on average, 1.3–1.7× throughput, and 0.2–0.3× compute cost on isolated text capabilities.

Why compress the verifier's context rather than the drafter's?

Because the verifier is the expensive model, and in agent pipelines the context is the expensive input. Retrieval results, tool outputs and conversation history are what make the verifier's context costly, and the verifier attends to all of it on every pass. The drafter only has to guess well enough for its guesses to be accepted, so giving it the full input is comparatively cheap — which is why AsymSpec puts the breadth on the small model and the compression on the large one.

How does this relate to ordinary speculative decoding?

Ordinary speculative decoding shows both models the same context and preserves the large model's output distribution exactly, so it reduces latency without any quality argument attached. AsymSpec keeps the draft-verify structure but breaks that symmetry, and does not claim that exactness — its headline is an accuracy ratio, about 90% of full-context accuracy on average, rather than a preservation guarantee. In exchange it reduces what the large model reads, which is the one cost standard speculative decoding cannot touch.

AsymSpec gives the small drafter the full context and the large verifier a compressed one — Context-asymmetric speculative decoding

TL;DR

What is it: The news anchor is AsymSpec, a paper accepted to the EMNLP 2026 main conference; the concept it makes concrete is context-asymmetric speculative decoding — a draft-and-verify setup where the small drafter reads the full input and the large verifier reads a compressed view of it.
Why it’s needed: In agent pipelines the context is mostly retrieval results, tool outputs and conversation history, and the model that has to read all of it is the large one — so the bill is set less by how many tokens you generate than by how many the expensive model has to attend to.
vs previous: Ordinary speculative decoding shows both models the same context and preserves the big model’s output distribution exactly, so it buys latency but never reduces what the big model reads; AsymSpec does not claim that guarantee — its headline is a ratio instead, about 90% of full-context accuracy on average — in exchange for a verifier that attends to far less.

Think of it as

a junior who read the whole case file, briefing a partner who read the summary

the case file = the agent's full context — retrieval results, tool outputs, the whole conversation so far
the junior associate = the small drafter model, fast and cheap, who still reads every page
the partner = the large verifier model, slow and expensive, whose signature is the only one that counts
the summary = the compressed view the partner reads instead of the file, because partner-hours are what the case costs
the junior's margin notes = contrastive delta fusion, nudging the partner toward what the summary left out
the partner's disagreement rule = the divergence-aware gate — when the notes and the summary point different ways, stop trusting the draft
a signed-off block of paragraphs = the run of draft tokens one verifier pass accepts at once

Jargon

Speculative decoding: A serving technique where a small model guesses several tokens ahead and a large model checks them all in one forward pass instead of one pass per token. The saving comes from the check being parallel, not from the guessing being clever. See the draft-verify idea →
Drafter: The small, cheap model that produces the guesses. In AsymSpec it is described as lightweight, and it is the model that reads the full input — which is the inversion the paper is built on.
Verifier: The large model whose output you actually want. Everything speculative decoding does is arranged so that the verifier runs as few times as possible, because it is the expensive one. See the verification algorithm →
Context compression: Replacing a long input with a shorter stand-in — a summary, a selection, a learned encoding — before a model reads it. It is cheap and it is lossy by construction; the interesting question is always which signals it drops.
Contrastive delta fusion: AsymSpec’s mechanism for putting the dropped signals back. The drafter, which read everything, influences the verifier’s logits — the raw scores the model assigns each next token before they are turned into probabilities — rather than the verifier having to attend to the full context itself.
Divergence-aware acceptance gating: The safety valve. A gate decides when the asymmetric draft is still safe to accept, so the system can notice that the compressed view and the full-context drafter have diverged, instead of accepting the draft blindly.
Lossless (in speculative decoding): The property standard speculative decoding is normally sold on: the accepted output is distributed exactly as the large model alone would have produced it. AsymSpec does not claim it: the record reports an accuracy ratio — about 90% of full-context accuracy on average — rather than a preservation guarantee, so treat exactness as not on offer here.
Compute cost vs throughput: Two different savings that are easy to conflate. Compute is total work, and it is what a GPU-hour bill is made of; throughput is finished requests per second, and it is what a user feels. AsymSpec reports a much larger reduction in the first than in the second. See throughput vs goodput →

The news. On August 26, 2026 a paper introducing AsymSpec was posted, aimed at agent pipelines whose retrieval results, tool outputs and conversation history make the verifier’s context expensive. A lightweight drafter reads the full input while the larger verifier reads a compressed view; the drafter influences the verifier’s logits through contrastive delta fusion, and a divergence-aware gate decides when the asymmetric draft is still safe to accept. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, the paper reports about 90% of full-context accuracy on average, 1.3–1.7× throughput, and 0.2–0.3× compute cost on isolated text capabilities. It is accepted to the EMNLP 2026 main conference. Read the paper →

Picture the law office. A case has arrived as four boxes of documents, and there are two people who could read them: a junior associate whose hours are cheap, and a partner whose hours are what the client is actually paying for. The obvious arrangement is that both read the file and the partner signs. The arrangement AsymSpec proposes is the one a real office would land on: the junior reads all four boxes, the partner reads a summary, and the junior sits in the room to say what the summary left out. The partner still signs — that has not changed, and it is the whole point of having a partner — but the partner is no longer being paid to turn pages.

Speculative decoding, in its ordinary form, is the arrangement where both of them read the file. A small drafter proposes a run of tokens, the large verifier checks the whole run in one forward pass instead of one pass per token, and the accepted output is distributed exactly as the big model alone would have produced it. That exactness is why the technique is easy to adopt: it is a pure latency trick with no quality argument attached. But it never touches the size of the thing the big model reads — the verifier still attends to every token of context, once per pass, forever.

That is a fine trade when the context is a prompt. It stops being fine when the context is an agent's working memory. Retrieved documents, tool outputs and turn-by-turn history are exactly the sort of input that grows without anyone deciding it should, which is why context is the scarce resource in agent systems rather than an afterthought. AsymSpec's move is to notice that the two models in a draft-verify pair do not need the same input, because they are not doing the same job — the drafter needs breadth to guess well, and the verifier needs to be right about the next few tokens.

So the compression goes on the expensive side, and two mechanisms hold the quality up. Contrastive delta fusion lets the drafter — the model that did read everything — push on the verifier's logits, so a signal the summary dropped can still reach the decision without the large model attending to the full context to find it. And a divergence-aware gate watches for the case where that repair is not working: when the full-context drafter and the compressed-context verifier disagree too much, the gate stops treating the draft as safe. Neither mechanism is described as restoring a preservation guarantee. What the gate buys is narrower and more useful: an asymmetric draft that has stopped being safe gets noticed rather than accepted silently.

Approach	What it varies	Do both models see the same context?	Big model’s output distribution preserved exactly?
Standard draft-verify	How many tokens are guessed before each check	Yes	Yes
Tree drafting	The shape of the guess — many candidate branches, not one line	Yes	Yes, given an exactness-preserving verification rule
Tiered verification	Which verifier handles a close call — a slimmer one, or the full model	Yes	Depends on the tiering scheme
AsymSpec	How much context each model reads	No — full for the drafter, compressed for the verifier	Not claimed — the reported headline is a ratio, about 90% of full-context accuracy on average

Now price it, because the two headline numbers describe different things and reading them as one number is the easiest mistake here. Hold the workload fixed at 1,000 agent requests (illustrative: the paper reports ratios, not a request count) and price that same fixed workload twice. Priced in compute, at the reported 0.2–0.3× on isolated text capabilities, those 1,000 requests cost what 200 to 300 of them used to — the bill falls by roughly three to five times. Priced in wall clock, at the reported 1.3–1.7× throughput, the same 1,000 requests take about 590 to 770 time units instead of 1,000 — roughly 23% to 41% less aggregate time. The compute saving is several times larger than the latency saving — the record reports both ratios without explaining the gap, and the plain reading is that shrinking the verifier's context removes work the verifier was doing, while the steps the pipeline still has to walk through in order are untouched. And the accuracy figure is a ratio too: 90% of full-context accuracy on average is relative, so a task the full-context system gets right 70% of the time lands near 63%, not 60% — worth spelling out, because the two readings differ by enough to change whether you would ship it.

Which leaves the honest summary of the trade. This is not the version of speculative decoding you adopt because it is free — that version exists, it is lossless, and it is already in your serving stack. This is the version you reach for when the verifier's context has become the dominant cost and you have decided that roughly a tenth of full-context accuracy, on average, is a price worth paying for it, which is a product decision before it is an engineering one. Read the reported scope carefully before you assume it transfers: the accuracy figure is given across the whole evaluation — four agentic capabilities and two end-to-end agent benchmarks — while the largest compute win, the 0.2–0.3×, is scoped to isolated text capabilities. The place to check whether it applies to you is when to use it — and the question to ask first is not how fast the drafter is, but how much of what you are paying for is the big model re-reading the same tool output for the thousandth time.

Goes deeper in: LLM Serving → Speculative Decoding → The Verification Algorithm

Related explainers

Spec-decode latency under load — Little's Law decomposition — why a speculative-decoding win measured on an idle server shrinks as the server saturates
VIA-SD — Tiered speculative verification — the other way to make verification cheaper: a slimmer verifier, rather than a shorter context
D-Cut — Cross-request draft pruning — cutting draft cost across a whole batch instead of inside one request

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based