What is Prefix Sliding?

A rule for deciding which tokens of a model's reasoning trace stay in the KV cache. A token is kept if it is in the prompt's fixed prefix — the instructions and tool descriptions — or in the window of the last few thousand tokens; everything between the two regions is discarded. Because both kept regions are a fixed size, total memory stops growing with the length of the reasoning.

Why does capping reasoning memory matter?

Test-time scaling makes models better at hard problems by letting them think for longer, but under full attention each extra thinking token adds another KV cache entry. Cost therefore rises with difficulty, which is the wrong way round. A cap turns a per-request cost that grows without limit into a fixed number a scheduler can plan a batch around.

How is Prefix Sliding different from a normal sliding window?

A plain sliding window keeps only the most recent tokens, so it eventually slides past the instructions and the tool descriptions and the model loses track of its own task. Prefix Sliding exempts that prefix from the window. The paper reports this is what makes the difference: its ablations show the method beating both the plain window and summarizing the discarded middle.

Does Prefix Sliding require retraining the model?

No. The paper reports roughly a 3x speedup on existing models with no training at all, because the rule is purely about which cached tokens are kept. Training with the same pattern using reinforcement learning is optional and buys something extra: the model adapts to the truncated history and can reason past a hundred thousand tokens.

LLM· 2026-09-05

A 3x speedup for long reasoning, with no retraining — Prefix Sliding — What does it mean?

Jargon

Test-time scaling: Spending more compute at inference to get a better answer — most often by letting the model reason for longer before it replies. The extra compute is the point; the memory it drags along is the problem.
Reasoning trace: The chain of intermediate tokens a model generates while working toward an answer. It can run to tens or hundreds of thousands of tokens on a hard problem.
KV cache: The stored key and value vectors for every token the model has already processed, so it does not recompute them each step. It is what makes generation fast and what makes long contexts expensive. How the cache works
Full attention: The default: every new token attends to every token before it, so the cache must hold all of them. Cost grows with the length of the sequence.
Sliding-window attention: Restricting each token to a fixed window of recent tokens instead of the whole past, which caps the cache. The plain version slides over the prompt too. A related use of the idea
Prefix: In this paper, the fixed opening region of the prompt: the instructions and the descriptions of the tools the model can call. Prefix Sliding exempts it from the window.
Ablation: An experiment that removes or swaps one part of a method to show that part is what produces the result. The paper's ablations compare Prefix Sliding against a plain window and against summarization.
Reinforcement learning (RL): Training a model by rewarding whole outputs that turn out well, rather than copying a reference answer token by token. Here it is used to train the model with the sliding rule already in place, so it learns to reason with a truncated history.

The news. On August 26, 2026, a team led by Niklas Muennighoff posted "Prefix Sliding for efficient test-time scaling" (arXiv 2608.26070), with co-authors including Percy Liang, Jason Wei, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi and Mike Lewis. The claim is narrow and practical: when a model reasons for a long time, most of the tokens in the middle of that reasoning stop mattering, so you can discard them. Doing so makes existing models 3x faster with no retraining, and training with the same rule using reinforcement learning lets a model reason past a hundred thousand tokens. Read the paper →

Picture that whiteboard again, three hours into the meeting. The agenda card is still pinned in the corner where it started. The last few lines of working are still legible on the right. Everything in between — the arithmetic from an hour ago, the branch that was abandoned, the sub-result already folded into the current line — has been wiped to make room. Nobody in the room misses it, because the line they are working on now already carries whatever those steps were worth.

That is the observation the paper starts from: most intermediate reasoning tokens lose importance as the model keeps reasoning. Under full attention — every token attending to every token that came before it — the model has no way to act on that. Each new thinking token adds another entry to the KV cache, and that cache grows in a straight line with the length of the trace. The harder the problem, the longer the model thinks, and the more expensive the request becomes. That is exactly backwards from what you want.

Prefix Sliding makes the discard rule positional instead of clever. There is no scorer, no learned predictor, no summarizer deciding what deserves to stay. A token survives if it sits in the prefix — the instructions and the tool descriptions at the top of the prompt — or in the window of the last few thousand tokens. Everything else is dropped. Because both surviving regions are fixed in size, the total is fixed too, however long the reasoning runs.

The pinned card is the part that is easy to get wrong. A plain sliding window slides over everything, instructions included, so a model deep into a problem eventually forgets what it was asked to do and which tools it has. In the whiteboard picture, that is the eraser reaching the agenda card. Prefix Sliding's only structural addition is the exemption: the prefix stays pinned while the window slides past it.

The paper reports that this beats both the plain window and the obvious alternative, which is to summarize the discarded middle into a shorter note. Summarizing does keep information the positional rule throws away — but it costs a generation pass every time the window fills, and the summary then has to be trusted for the rest of the run.

Approach	What it discards	Cache size vs. trace length	Training needed
Full attention	nothing	grows in a straight line	none — this is the default
Summarize the middle	the middle, rewritten into a shorter note	grows slowly	none, but a generation pass each time the window fills
Vanilla sliding window	everything outside the recent window, prefix included	flat	none
Prefix Sliding	the middle only — prefix and recent window are kept	flat	none for the reported 3x speedup; RL training to reach 100K+ token traces

Put numbers on it. The diagram above is the per-token cost of a KV cache, written as a chain of multipliers: two vectors per token, one cache per layer, one entry per attention head, so many numbers per entry, so many bytes per number. Fill it in with 32 layers, a head dimension of 128, FP16, and 8 key-value heads rather than the diagram's 32 — most current models share one K/V pair across several query heads — and it gives 131,072 bytes, or 128 KB, of KV cache per token. Hold those four numbers fixed and vary only how long the model thinks.

Under full attention, a hundred-thousand-token reasoning trace — the length the paper reaches after training with Prefix Sliding — costs 100,000 x 128 KB, or about 13 GB of KV cache for one request. With Prefix Sliding pinning a 1,000-token prefix and holding a 4,000-token window, that same request holds 5,000 tokens no matter how far it reasons: 5,000 x 128 KB, or about 0.65 GB. That is roughly 20x less memory (illustrative) — and the figure does not move when the trace gets longer, because it is a ceiling rather than a slope. Memory saved this way turns directly into time saved, because decoding is bound by how much cache the GPU has to read back on every step — which is why the paper measures the same rule as 3x faster wall-clock on models it never retrained.

What this does to a batch is where it earns its keep. Serving throughput is set by how many requests fit in GPU memory at once, and under full attention a long-reasoning request is one whose footprint keeps growing while it runs, so the scheduler cannot plan around it — it does not know when the thinking will stop. A capped per-request footprint turns an open-ended cost into a number you can multiply by the batch size. The diagram above is the usual form of that tradeoff: many short conversations, or few long ones. Prefix Sliding moves a long-reasoning request into the left-hand column, because its cache stops growing after the first few thousand tokens.

It is worth being clear about what it does not replace. Grouped-query attention shrinks the cache along the head axis. KV quantization shrinks it along the bit axis. Prefix Sliding shrinks it along the token axis, and it is the only one of the three whose saving grows with how long the model thinks. They are independent, and a production system can run all three at once.

Continue in trackKV Cache — why the cache grows, and what every fix has to give up

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

Reference Sliding Window Attention — the same positional idea applied to document pages instead of reasoning traces, with a reference region standing in for the prefix.
InfoKV's Forward Influence — the other school of thought: score every cached token and keep the important ones, rather than deciding by position.
Per-head KV cache budgets — shrinking the same cache along the head axis instead of the token axis.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based