What is a copy-intent probe?

It is a small classifier trained on a target model's internal representations, which outputs whether the model is about to reproduce text from its context. SwitchSD uses that output to pick the drafter for the next speculation, and reports **AUC above 0.99** for the classification.

Why is n-gram overlap a bad trigger for copy-based drafting?

Because prose repeats itself accidentally. Common phrases and boilerplate produce a surface match that does not continue, so the copier proposes a long span that is wrong after a token or two. Each of those false positives still costs a full verification pass, which is how a copy-based drafter can end up slower than the neural one it replaced.

How does SwitchSD relate to EAGLE3?

EAGLE3 is one of the two machines, not a competitor to the idea. SwitchSD keeps neural drafting as the robust default and switches to context copying only when the probe says the repetition is genuine, reporting **up to 15% throughput over EAGLE3** on Llama and Qwen families.

Does switching drafting modes change the model's output?

No. The target model still verifies every drafted token and corrects the first wrong one, and that step is unchanged by which drafter proposed the span. The drafting mode is a throughput decision, not a quality one — see [the verification algorithm](/tracks/llm-serving/speculative-decoding?step=verification).

LLM· 2026-09-20

SwitchSD reads copy intent from the model's own states — Copy-intent probe drafting-mode switch — What does it mean?

TL;DR

What is it: The SwitchSD paper (arXiv 2609.20186, "To Copy or Not to Copy") adds a copy-intent probe to speculative decoding: a small classifier that reads the target model's own internal representations and decides, at each step, whether the tokens coming next will be copied out of the context or genuinely written.
Why it’s needed: That decision picks which drafter runs, and copying is the cheapest draft available: it proposes a long span without running a draft model at all — so choosing it wrongly is not merely neutral, because a rejected speculation still costs a full verification pass.
vs previous: Earlier copy-based drafting triggers on surface-level n-gram overlap, which fires on accidental repetition; SwitchSD replaces that heuristic with an intrinsic model signal and reports AUC above 0.99 at separating real copy intent from coincidence.

Jargon

Speculative decoding (SD): A serving technique where a cheap drafter guesses several tokens ahead and the real model — the target model — checks them all in one pass. Covered step by step in the draft-verify loop.
Neural drafting: Drafting with a small trained model or extra head, such as EAGLE3, that predicts the next few tokens from the target model's own features. Robust across ordinary prose, but it costs a forward pass of its own per speculation.
Context-based copying: Drafting by looking backwards in the text already generated or prompted and proposing the continuation that followed last time. It runs no model at all, so a long repeated span can be proposed almost for free.
Copy intent: The paper's term for the model actually being about to reproduce earlier text, as opposed to a few words happening to match. The distinction is the whole point: only the first kind pays off.
n-gram overlap: A match between the last few tokens and an earlier stretch of the same text. It is the surface signal older copy-based drafters trigger on, and it cannot tell a real repetition from a coincidence.
Probe: A small classifier trained on a model's internal activations to read off some property of them. Here it is trained on the target model's representations to output a copy-or-not decision.
AUC: A single score for how well a classifier separates two classes across every decision threshold: 1.0 is perfect and 0.5 is a coin flip. SwitchSD reports above 0.99 for copy intent.
Acceptance length: How many drafted tokens survive verification per step, on average. It is the number speculative decoding lives or dies by, because one verification pass costs one full target forward either way.

The news. On September 17, 2026, a paper titled "To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals" introduced SwitchSD. It starts from a tradeoff the field already knew about: neural drafters such as EAGLE3 behave well on any text, while copy-based drafters win big on repetitive text by proposing long spans cheaply. The paper's contribution is the switch between them. Rather than triggering on n-gram overlap, SwitchSD trains lightweight probes on the target model's internal representations and reports AUC above 0.99 at detecting genuine copy intent, with throughput gains of up to 15% over EAGLE3 across Llama and Qwen families. Read the paper →

Picture the counter of a print shop. Two machines sit behind it. The photocopier is unbeatable when the job is a reprint: feed it the original and a hundred identical pages come back in the time the typist needs for one line. The typist is slower, but the typist can handle a page nobody has written yet. Every job that arrives has to be sent to one of them, and the shop cannot ask the customer, because the customer does not know either.

That is the position a speculative decoder is in at every step. A drafter is not judged on being right; it is judged on being cheap enough that being wrong is affordable — and those two machines fail in opposite directions. Neural drafting spends a small forward pass to produce a few plausible tokens, which is a steady, modest win on almost anything. Context-based copying spends essentially nothing and can propose a very long span, but only if the text really is repeating.

The older way to choose was to look at the page itself: if the last few tokens match something earlier in the document, assume a reprint and run the copier. That is the n-gram heuristic, and its problem is that prose repeats itself by accident all the time. Common phrases, list scaffolding and boilerplate all produce overlap that goes nowhere. Send those to the copier and it confidently prints the wrong pages.

Draft Model (1B) — generates K=5 tokens

Paris.Itis

↓ verify all at once↓

Target Model (70B) — one forward pass

✓ Par✓ is✓ .✗ It→The— is

accepted

rejected → corrected

discarded

3 accepted + 1 corrected = 4 tokens from 2 forward passes

↻ repeat until done

The diagram is the loop both machines feed into. Whatever the drafter proposes, the target model checks the whole span in a single forward pass, keeps the longest correct prefix and corrects the first wrong token. That verification step is why speculation is safe: the text you get out is the text the target model would have produced on its own. It is also why a bad draft costs real money — the verification pass is paid in full whether ten tokens survive it or one.

SwitchSD's claim is that the shop does not have to guess at the job, because the author has already written the answer in the margin: a probe on the target model's internal representations picks the drafter, so the choice is made from the model's own state rather than from the text on the page. The paper calls this treating "copying as a latent control signal of the LLM." It describes those probes as lightweight, and the switch is dynamic rather than decided once per request — the copier when the copy signal is strong, the typist when it is weak.

The difference between that and the n-gram trigger is where the evidence comes from. An n-gram match is evidence about the text, which is why coincidence fools it. A probe reading the internal activations is evidence about the model, which is the thing that actually decides what comes next. The paper reports that separation as near-perfect, which is what makes the switch worth having: a gate that is only occasionally right would hand the copier just enough bad jobs to erase its advantage.

Drafting policy	What proposes the tokens	Strongest when	How it fails
Neural drafting (EAGLE3)	a small trained head runs its own forward pass	ordinary prose, where nothing repeats	the draft costs a forward pass every step; the abstract frames it as robust across settings rather than a large win in any one
Context-based copying (n-gram triggered)	a span lifted from earlier text, no model run	copy-intensive text with long repeated spans	fires on accidental overlap, and each false positive still pays a full verification pass
SwitchSD (probe-gated switch)	either one, chosen by a probe on the target model's hidden states	mixed traffic, where both regimes appear in one request	reported AUC above 0.99, up to 15% over EAGLE3 on Llama and Qwen (source)

Here is why a false positive is worse than it sounds. Hold two things fixed: each speculation proposes 8 tokens, and each verification costs one full target forward pass no matter how many of those eight survive. Say a quarter of the steps in a request are genuine repetition (illustrative).

On a genuine copy span the copier is close to perfect — call it 8 of 8 accepted, an acceptance length of 8. On an accidental n-gram match the copied span is simply wrong after the first token or two — call it 1 of 8. A blind n-gram trigger that fires on both cases therefore averages 0.25 × 8 + 0.75 × 1 = 2.75 tokens per verification. The neural drafter, left alone, accepts perhaps 3 of 8 on the same traffic — 3.00 tokens per verification. The heuristic copier is not merely failing to help here; it is running below the drafter it replaced. That is the shape of the problem SwitchSD names.

Now gate the copier on a probe that routes every step correctly, the idealized case. The copy path is taken only on the quarter of steps that are real, and the other three quarters go to the typist: 0.25 × 8 + 0.75 × 3 = 4.25 tokens per verification (illustrative) — above both machines running alone. The numbers here are made up to show the composition; the paper's measured result is the up to 15% throughput gain over EAGLE3. The mechanism behind both is the same: the value of a switch is bounded by how often it is right, because every wrong switch is charged the full price of a verification.

Two limits are worth naming. First, the probe is trained on a target model's internal representations, and the paper's abstract does not say whether such a probe transfers to another model, or what training one costs — so treat portability as an open question rather than a settled one. Second, this is repetition inside a single request. Repetition across requests — the same system prompt on every call — is a different lever entirely, and it is already handled upstream by prefix caching, which skips the computation rather than guessing at it.

What generalizes past this one paper is the move itself. Serving systems are full of choices that are currently made from surface statistics: how long a draft to run, which expert to prefetch, when to stop thinking. The broader bet SwitchSD makes is that the model's own activations are a better place to look for those answers than the text it has produced so far — the signal is upstream of the output, so it arrives before the cost is committed. Whether a given workload is worth speculating on at all is the next question in the same family.

Continue in trackSpeculative Decoding — the draft model variants SwitchSD switches between

Goes deeper in: LLM Serving → Speculative Decoding → Draft Model Variants

Related explainers

DeLS-Spec adds short-context heads to block drafting — another way to improve the draft itself, by fusing two drafters instead of choosing between them.
BlockPilot gives diffusion speculative decoding 4.2x — the same adaptive instinct applied to a different knob: how many tokens to draft per step.
NVIDIA aligns draft length to tensor-core tiles — what happens when the draft-length choice is made by the hardware instead of the text.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based