HyFlash is a compact parallel drafter for on-device speculative decoding. Each layer combines local attention over recent target features and draft positions with a recurrent memory that compresses the committed history into a fixed-size state.

Why not just use full attention in the drafter?

Full attention needs a KV cache that grows with the prefix, which a memory-limited device cannot spare, and the paper reports that its acceptance falls sharply beyond the 8K training length. A sliding window bounds memory but loses distant information.

Why does the memory update only from committed features?

Rejected draft tokens are wrong by definition, so writing them into the memory would pollute the history. Updating only after verification keeps the state a clean summary of what the target accepted, and drafting reads it without changing it.

How does HyFlash relate to ReaDy?

Both give a drafter a fixed-size recurrent memory of the prefix. ReaDy reads the recurrent state of a hybrid linear-attention target, while HyFlash builds its own recurrent memory from committed target features, so it needs no special target.

LLM· 2026-11-07

HyFlash lifts on-device speculative decoding 16.2% — Prefix-memory drafting — What does it mean?

TL;DR

What is it: The HyFlash paper (arXiv, November 5, 2026) gives a tiny on-device speculative-decoding drafter a bounded memory: a short local attention window for recent tokens plus one fixed-size recurrent memory that compresses the history of features the target model has already confirmed.
Why it’s needed: On a 16 GB-class device the target model's weights and KV cache leave the drafter very little room, so a drafter that must keep a growing KV cache either runs out of space or goes blind to distant context; HyFlash gets long-range context for a constant 5 MiB in the paper's Phi-4 configuration.
vs previous: A full-attention drafter stores a KV cache that grows with the prefix and, in the paper's test, collapses past its 8K training length, while a sliding-window drafter forgets everything outside the window; HyFlash keeps the window and adds the recurrent memory, so acceptance stays up as the prefix grows (Phi-4-reasoning, LongBench).

Jargon

Speculative decoding: A small, fast drafter model proposes several tokens and the big target model checks them all in one forward pass, keeping the correct prefix. Primer →
Parallel block drafting: The drafter predicts several future positions in a single pass instead of one token at a time. In the paper the block is 7 positions.
Acceptance length: The average number of tokens the target commits per verification round. Longer is better, because each round costs about the same.
Committed target features: The target model's hidden vectors for tokens it has verified and kept. HyFlash conditions on these, and rejected draft tokens never contribute.
Sliding-window attention (SWA): Attention that looks only at the most recent entries, so older KV entries can be thrown away and storage stays bounded, at the price of losing distant context.
Recurrent memory (Mamba-2): A state-space layer that folds each new input into a fixed-size state using an input-dependent step size and a learned decay. The paper uses Mamba-2 for this.
KV cache: The stored keys and values of every earlier token, which grows by one entry per token. Memory cost →
Prefix-memory prior: The paper's name for the design assumption that history belongs in recurrent memory while recent context and interactions among draft positions belong in local attention.

The news. On November 5, 2026, two researchers at Microsoft's Applied Sciences Group posted HyFlash (arXiv:2611.07263), a compact parallel drafter built for devices with tight memory. Against the DSpark drafter it builds on, it raises average decoding speedup from 1.95× to 2.15× on Qwen3-4B and from 2.22× to 2.58× on Phi-4-reasoning, with a constant-size context. Read the paper →

Picture a sous-chef who must plate the next seven dishes before the head chef tastes any of them. Two things help. A rail of the last few order tickets shows what was just served, and a single tally sheet records, in compressed form, everything the head chef has approved so far. The rail is small because old tickets are thrown away; the tally sheet never grows, because new approvals are folded into the same sheet. The sous-chef may read both while plating, but may write on neither, and the sheet is updated only after the head chef approves a dish, so a rejected guess never contaminates it.

That is HyFlash. The drafter is a two-layer network in which every layer has two branches. Local attention looks at the last committed target features and at all seven draft positions, in both directions, which handles recent context and the interactions among the draft slots. A recurrent branch, instantiated with Mamba-2, holds the compressed history of committed target features and is only queried during drafting. After verification, only the newly committed features update that state, so unverified and rejected proposals are excluded. The paper calls this split the prefix-memory prior.

The reason to split the jobs is the memory budget. The paper takes a 16 GB-class device with an illustrative 12 GiB inference budget. With Phi-4-reasoning as the geometry reference, the target's KV cache alone is 6.25 GiB at 32K context, and even with ideal 2-bit target weights at most 0.84 GiB remains for the drafter and all other overhead. A drafter that wants to use the whole KV cache cannot afford it, and a shallow one cannot model long-range context and slot-to-slot interactions at the same time. Giving each job to the structure that suits it is what lets two layers do the work of more. In the ablations, restricting the recurrent history to the last 512 features leaves nearby acceptance unchanged at 6.32 but drops the distant-evidence probe from 5.62 to 3.09, while switching off attention among draft slots hurts the nearby probe, 6.32 down to 5.68, even with full recurrent history.

Context length (Phi-4-reasoning, HotpotQA)	Full-context attention	Sliding window	HyFlash
2K–4K tokens	2.78	2.72	3.02
4K–8K tokens	3.23	2.57	3.25
8K–16K tokens	1.24	2.25	3.11
16K–32K tokens	1.02	2.13	2.72

The table lists the average acceptance length from the paper's LongBench test. Full-context attention is competitive up to the 8K length it was trained at, then collapses to 1.24 and 1.02 tokens per round on HotpotQA. The sliding window degrades gently but cannot recover distant information. On HotpotQA, HyFlash stays at 2.72 or above across the whole range. It is also the highest in every cell of the paper's QASPER and GovReport results, with the largest gaps at 8K and beyond.

Where it earns its keep

Take the paper's Phi-4-reasoning numbers. The drafter's context is 2.5 MiB of local KV plus 2.5 MiB of FP32 recurrent state, a constant 5 MiB, matching a two-layer windowed drafter. The target's KV cache at 32K is 6.25 GiB, which is 6,400 MiB, so the target's cache is about 1,280 times larger than the drafter's context (derived from the paper's figures). Quantized to 4 bits with GPTQ, the drafter's weights are 465.9 MiB, so weights plus context come to about 471 MiB, under the 0.84 GiB (about 860 MiB) that the paper says is left even with 2-bit target weights (derived; the paper notes these estimates exclude temporary buffers, and the remainder must also cover other overhead). The 4-bit drafter keeps 99.2% of the BF16 acceptance length and runs 4.3% faster.

Now the speed. On Phi-4-reasoning, DSpark averages 3.28 tokens committed per round at a 2.22× speedup, and HyFlash averages 3.79 at 2.58×. Acceptance rises 15.5% and speedup rises 16.2%, two gains of similar size, which suggests the extra recurrent computation costs little per round (a rough reading only: the paper reports speedup and acceptance as different averages across benchmarks, so this does not prove the per-round cost). On depth, a one-layer HyFlash is competitive with a two-layer full-attention drafter, and a two-layer HyFlash approaches five-layer full-attention drafters at about matched parameter counts.

Three limits are worth knowing. The tests use two targets (Qwen3-4B and Phi-4-reasoning), two-layer drafters, batch-one decoding on one NVIDIA RTX PRO 6000 Blackwell GPU and temperature 1.0, so the numbers say little about large batches or other devices. Speedup falls at longer contexts for a reason outside the drafter: on AIME25, target verification latency nearly doubles from 44.76 ms at 4K to 87.80 ms at 16K, while the drafter backbone stays near 3 ms (2.95 vs 3.02 ms). And the idea is not tied to one component: swapping Mamba-2 for Gated DeltaNet-2 gives comparable results, with Mamba-2 slightly ahead on average.

Goes deeper in: LLM Serving → Speculative Decoding → Draft Model Variants

Continue in trackSpeculative Decoding: the drafter variants HyFlash builds on

Related explainers

ReaDy: recurrent-state speculative drafting — the mirror image: the drafter reads the target's own recurrent state instead of keeping its own
JetSpec: parallel tree drafting — another way to spend a small draft budget: shape the draft as a tree
Low-bit recurrent states — what a fixed-size recurrent state costs to store

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based