LLM·

Olifant memory-based drafter — Draft cost vs verifier cost — What does it mean?

The news. On November 12, 2026, the paper Towards Green Text Generation: Memory-Based Speculative Decoding was posted to arXiv. It introduces Olifant and measures throughput and whole-system energy on one consumer machine (a Mac Studio with an Apple M1 Max) with a Mistral-7B-v0.1 verifier, in two regimes: a verifier with no KV cache and a verifier with one. Olifant is compared with plain decoding, an in-session 4-gram model, and a TinyLlama-1.1B neural draft model on general, medical and legal text. Read the paper →

Picture a senior editor who must approve every word of a document, and a helper who writes guesses for the next few words so the editor can approve several at once. If the editor rereads the whole page before each approval, an approval is slow, so even a helper who takes a while to write a guess is worth having. Give the editor a perfect memory of the page and each approval becomes quick; now a slow helper is the bottleneck, and only a helper who answers instantly from a phrasebook still saves time. In the paper, the editor's memory is the KV cache, the slow helper is a neural draft model, and the phrasebook is Olifant.

In speculative decoding, each step costs draft time plus verify time, and it produces the accepted draft tokens plus one token from the verifier. A step is a win only when the extra tokens it produces are worth more than the drafting time it adds. Without a KV cache, every verifier pass attends over the whole context, so adding a few draft positions to a pass costs little, and nearly every tested drafter sped up; the one exception was a domain-mismatched, low-acceptance case at 0.96×. With a KV cache, a plain decode step is a cheap one-token pass: plain decoding in the paper went from 2.3–2.8 to 16.3–16.9 tokens per second. The TinyLlama drafter needs its own sequential forward pass for every guess, on the same accelerator as the verifier, so in the cached regime each speculative step spent more time drafting than the verifier needed to check, and throughput fell to 0.47–0.29× of plain decoding.

Draft Model (1B) — generates K=5 tokens
Paris.Itis
↓ verify all at once↓
Target Model (70B) — one forward pass
✓ Par✓ is✓ .✗ It→The— is
accepted
rejected → corrected
discarded
3 accepted + 1 corrected = 4 tokens from 2 forward passes
↻ repeat until done

Olifant sits at the other end of the cost scale. It maps a four-token window to a next-token guess in 0.16–0.25 ms on the CPU, uses no accelerator memory, and needs no backpropagation; TinyLlama took 254 ms per token on the same CPU and, hosted next to the verifier, took 2 GB of accelerator memory. Olifant's acceptance is modest: 38.1% on EU legal text for the variant trained on legal text, the same in both regimes. The track's rule of thumb that acceptance below about 50% usually makes speculation slower assumes a drafter with a real cost. When drafting is nearly free, 38% acceptance is enough: Olifant gave up to 2.43× throughput without a KV cache and still 1.25× with one. The paper places the cached break-even point near 20–25% acceptance of the first drafted token; drafters trained on the wrong domain, at 5–20% acceptance, landed at 0.9–1.1×.

DrafterCost per guessSpeedup, no KV cacheSpeedup, KV cacheEnergy, KV cacheSource
None (plain decoding)—1.00× (2.3–2.8 tok/s)1.00× (16.3–16.9 tok/s)96–106 J per 100 tokenspaper
TinyLlama-1.1B (neural)one 1.1B forward pass; 254 ms on CPU1.15–1.20×0.47–0.29×243–382 J per 100 tokenspaper
Olifant, legal-trained, on legal text0.16–0.25 ms on CPUup to 2.43×1.25× at k = 393 J per 100 tokens at k = 3paper

A worked example: the drafting budget. Hold the draft length at 3 tokens and use the paper's measured verifier speeds. Without a KV cache the verifier produced 2.3–2.8 tokens per second, about 400 ms per step; with one, 16.3–16.9 tokens per second, about 60 ms per step. Three Olifant guesses cost under 3 × 0.3 = 0.9 ms: about 0.2% of an uncached step and 1.5% of a cached one, so the drafter stays nearly free in both regimes. Now take a neural drafter at 30 ms per guess (illustrative; the paper does not report TinyLlama's per-token time on the accelerator). Three guesses cost 90 ms: about 22% of an uncached step, repaid by a single accepted token, but 150% of a cached step. Counting the check as one ordinary 60 ms step, that drafter needs more than 1.5 of its 3 guesses accepted on every step just to break even. The same drafter goes from a small tax to costing more than the verification it is meant to save, which is the shape of the TinyLlama result.

Once the verifier is cached, speed and energy also stop moving together. Without a KV cache, mean system power stayed in a 16–31 W band whatever the drafter did, so energy per token followed wall-clock time. With one, power rose with draft length, from 16.4 W for plain decoding to 22.1 W for Olifant at 8 drafted tokens, because checking several tokens at once works the accelerator harder than a one-token step. At k = 8 Olifant was 1.16× faster than plain decoding yet used 1.18× more energy; at k = 3 it was 1.25× faster and 1.09× more energy-efficient, at 93 J per 100 tokens, the lowest in the study. The energy-optimal draft length is shorter than the speed-optimal one, so a battery-bound device should draft fewer tokens than a latency-bound one.

Two limits keep this in proportion. The cache itself is still the largest lever: cached plain decoding cost 96–106 J per 100 tokens, about one tenth of the uncached 1,005–1,326 J, and the authors say no draft model comes close to that saving. And every number comes from one Apple M1 Max and one 7B verifier; the authors list other verifier sizes and server-class GPUs as future work, where faster verification should squeeze the drafting budget further. A separate result on speculative decoding energy under batching shows the same budget shrinking from another direction: batching, not caching, makes the verifier cheaper per token.

Goes deeper in: LLM Serving → Speculative Decoding → When to Use It

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based