What is the Context-Generation Substitution Law?

It is the paper's name for a trade between the two halves of a request: explicit context placed in the prompt can substitute for reasoning tokens the model would otherwise generate, provided the retrieved memory is relevant to the question. It matters because the two are not priced alike — prompt tokens are processed together during prefill, while generated tokens are produced one per forward pass during decode. The relevance condition is the load-bearing part: an irrelevant memory is simply a longer prompt, and it saves nothing.

Why does moving reasoning into the prompt make it cheaper?

Because prefill and decode have very different shapes on the hardware — background from the serving curriculum, not a measurement in this paper. Prefill processes every prompt token in one parallel pass, so it has enough arithmetic per byte of weights it reads to keep the GPU busy. Decode produces one token per pass, re-reading the model's weights for each one, which leaves it bound by memory bandwidth rather than compute. A chain of thought is generated, so all of it lands on the decode side; a retrieved memory is prompt text, so it lands on the prefill side.

How does this relate to retrieval-augmented generation?

The machinery is the same — index something, retrieve the relevant piece, put it in the prompt — but what gets retrieved is different. RAG retrieves facts the model does not know, to fix knowledge gaps. Memory-Augmented Compression retrieves reasoning methods the model could have derived itself, to avoid paying for the derivation. It should therefore inherit the failure mode retrieval commonly has — the wrong memory comes back — though the paper states only the relevance condition and does not evaluate retrieval fixes.

Compress CoT reasoning with reusable memory scaffolds — Context-Generation Substitution Law

TL;DR

What is it: The news anchor is an arXiv paper on Memory-Augmented Compression, which retrieves compact reasoning memories into the prompt instead of letting the model generate a full chain of thought; the concept it makes concrete is the Context-Generation Substitution Law — explicit context can stand in for generated reasoning tokens, as long as the retrieved memory is relevant.
Why it’s needed: The two halves of a request do not cost the same: prefill reads the whole prompt in one parallel pass, while decode emits one token at a time, each token its own trip through the weights — so a long reasoning trace is the most expensive way to get the same information into the model.
vs previous: Standard chain-of-thought makes the model rediscover the same method on each new question of a familiar type, and Chain-of-Draft only shortens what it writes; this work instead moves the reasoning to the cheap side of the request by retrieving it, and states the trade as a rule you can plan around rather than a prompt-length tweak.

Jargon

Chain-of-Thought (CoT): Prompting the model to write its reasoning out step by step before answering. It raises accuracy on multi-step problems, and every one of those steps is a generated token the user waits for.
Chain-of-Draft (CoD): A shorter variant of the same idea: keep the reasoning steps but write them as terse drafts rather than full sentences. It is the paper’s accuracy baseline — the thing to beat once you are already writing less.
Prefill and decode: The two phases of serving a request: prefill processes every prompt token together in one pass, while decode produces the answer one token per pass. That asymmetry, taught in the curriculum rather than claimed by this paper, is what makes the substitution worth making. Walk prefill vs decode →
Reasoning memory: A compact record distilled from past reasoning traces — the patterns, constraints, and operations that recurred — stored so it can be retrieved later instead of re-derived. Not the answer to a past question; the method that solved it.
Prefill-side scaffold: The retrieved memory once it has been placed in the prompt. It is called a scaffold because it does not answer the question — it gives the model the structure to answer with, so the trace it still has to write can be short.
Context-Generation Substitution Law: The paper’s name for the trade: explicit context can substitute for generated reasoning tokens when the retrieved memory is relevant. The relevance clause is load-bearing — it is what stops this from being “stuff more text into the prompt”.
GSM8K, MATH, BBH, MMLU-Sci: The four reasoning benchmarks the results are reported on: grade-school word problems, competition mathematics, BIG-Bench Hard, and the science slice of MMLU. All four are used here as reasoning benchmarks, the regime where a trace is long enough to be worth replacing.

The news. On August 21, 2026, an arXiv paper reframed verbose chain-of-thought as an inference-cost problem and proposed Memory-Augmented Compression: build reusable reasoning memories out of past traces, retrieve the relevant ones into the prompt, and let the model emit a much shorter Chain-of-Draft trace instead of working the whole method out again. The authors name the underlying trade the Context-Generation Substitution Law — explicit context can substitute for generated reasoning tokens when the retrieved memory is relevant. Against Chain-of-Draft they report +21.4 points on GSM8K, +28.0 on MATH, +29.5 on BBH and +6.61 on MMLU-Sci, alongside a 1.14–1.49× latency speedup over standard CoT. Read the paper →

Picture the exam — an illustration, not anything the paper describes. Two students get the same multi-step question. The first derives everything on the answer sheet — restates the problem, sets up the equation, works it line by line, checks it — and hands in three pages of scratch work with the answer at the bottom. The second brought the one index card the rules allow, prepared the night before out of the last twenty problems of this shape: the setup that always works, the constraint people always forget, the two operations that finish it. She glances at the card, writes four lines, and is done. Neither student knew less; one of them had the method in front of her instead of in her head.

That is the whole proposal, and the reason it is a serving result and not a prompting tip is where each student's work lands in the request. The scratch work is generated tokens — decode, one token per forward pass, each pass dragging the model's weights out of memory again for a single token of output. The index card is prompt text — prefill, where the GPU takes the entire card in one parallel pass because every token is already known. The same information costs wildly different amounts depending on which side of the request it arrives on, and a chain of thought puts all of it on the expensive side.

Prefill vs Decode on the Roofline

Same GPU, fundamentally different bottlenecks

None of that hardware story is a claim from this paper — it is the standard serving picture the curriculum already teaches, and it is why the paper's trade is worth making. The roofline picture makes the asymmetry concrete: prefill sits high, with enough parallel work per byte moved to keep the arithmetic units busy, while decode sits down in the memory-bandwidth-bound corner, doing a tiny amount of math per byte it reads. Decode is not slow because the model is thinking hard — it is slow because one token at a time cannot fill a GPU. So every reasoning step you can hand the model as text instead of asking it to produce is a step that moves from the flat part of the curve to the steep part.

The memories themselves come from historical reasoning traces: the paper summarizes them into patterns, constraints, and operations — the parts that can stand in for decode-time reasoning. Reading that as method rather than answer is my gloss, not the paper's wording, but it is what makes reuse plausible: a stored answer can only help if the same question comes back, whereas a stored method can help on the next question of that shape. At inference the relevant ones are retrieved as prefill-side scaffolds, and the model still writes a trace, just a Chain-of-Draft-sized one. The law's condition is the retrieval quality, not the compression ratio: an irrelevant memory is a longer prompt that saves nothing, which is the same failure mode retrieve-then-generate has always had.

Hold the two rates fixed and the exchange rate falls out (illustrative — this arithmetic is mine, not the paper's). Take a server decoding at 50 tokens/second — 20 ms per generated token — and prefilling at 5,000 tokens/second, or 0.2 ms per prompt token. A question that takes 400 tokens of chain-of-thought costs 400 × 20 ms = 8.0 s. Now swap it: retrieve a 300-token memory and let the model write a 120-token draft. The card costs 300 × 0.2 ms = 60 ms; the draft costs 120 × 20 ms = 2.4 s; the total is 2.46 s, or 3.3× faster. Put the other way, at these rates one decode token buys 100 prefill tokens, so the 280 reasoning tokens you did not generate were worth 28,000 prefill tokens — and you spent 300. That ratio is why the trade is worth making at all, and also why the measured result is far more modest: the paper reports 1.14–1.49× over standard CoT, because retrieval costs time, a longer prompt grows the KV cache you then decode against, and real traces do not shrink to 30% on every task.

Getting multi-step reasoning into the answer	Where the reasoning lives	Reported result
Standard chain-of-thought	generated at decode, every question, from scratch	the latency baseline the 1.14–1.49× speedup is measured against
Chain-of-Draft	still generated at decode, written more tersely	the accuracy baseline: +21.4 GSM8K, +28.0 MATH, +29.5 BBH, +6.61 MMLU-Sci are gains over it
Memory-Augmented Compression	retrieved into prefill as a scaffold; only a short draft is generated	both of the above — higher accuracy than CoD, lower latency than CoT
What has to be true for it to work	the retrieved memory matches the question’s shape	~the relevance condition stated in the law itself (qualitative, no reported number)

The part worth carrying past this paper is a design implication rather than a reported result: what the substitution does to the failure mode. When reasoning is generated, a wrong answer is a reasoning failure, and you fix it by training or prompting the model to think better. When reasoning is retrieved, a wrong answer can just as easily be a retrieval failure — the right method existed in the store and the wrong card came back — and you fix that with chunking, ranking, and eviction, not with the model at all. Taken seriously, the substitution law moves multi-step reasoning out of model behaviour and into context engineering, where the prompt is a budget you spend deliberately and every token you place has to earn the decode tokens it saves. The paper states the law and reports the accuracy and latency numbers above; which retrieval fixes work best is not something it evaluates.

Goes deeper in: LLM Internals → KV Cache → Prefill vs Decode

Related explainers

EvoArena + EvoMem — Patch-based agent memory — the other half of the problem: how a reusable memory store gets built and kept current
Proactive memory agent — Memory-grounded reminder injection — the same move at the harness level, re-injecting a memory into the prompt at the moment it is needed
Compute Where It Counts — Per-token compute controller — the other lever on the same bill: make each generated token cheaper instead of generating fewer of them

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based