What is the physical cost of context?

It is the work a model runtime actually executes for a call: the logical prompt minus the longest prefix it can reuse from cache, plus KV residency, time and energy. It differs from the nominal prompt-token count, which only measures how long the prompt looks.

Why can a shorter prompt cost more?

A prefix cache reuses tokens only up to the first changed position. If a policy deletes or rewrites history near the start, the cached prefix ends early and most of the shorter prompt is prefilled again. In the paper, Mask-5 cut prompt volume by 33.3% yet raised uncached prefill by 608.5%.

How does this relate to prefix caching?

Prefix caching is what makes the logical prompt and the executed work different. With it on, the paper's Eager policy cut uncached prefill by 67.7% while prompt volume stayed about the same.

What should I measure in a production agent?

Measure uncached prefill (logical prompt minus reused prefix) alongside call count and prompt tokens, and apply a task-quality gate before comparing policies. A policy that fails the gate is not a saving, whatever its token count.

Agent· 2026-10-28

The Physical Cost of Context — Nominal tokens vs physical prefill — What does it mean?

Jargon

Prefill: The first phase of an LLM call, where the model reads the whole prompt and builds its key-value (KV) cache before it writes any output token.
KV cache: The stored attention keys and values for tokens the model has already read. Reusing them lets the model skip reading those tokens again.
Prefix caching: A serving feature that keeps the KV cache of a prompt's beginning so a later call that starts with the same tokens can reuse it. How it works.
Uncached prefill: The tokens that are actually prefilled on a call: the logical prompt minus the longest prefix found in cache.
KV residency: How much KV cache a request keeps in GPU memory, and for how long. The paper measures it next to time and energy.
Quality gate: A rule applied before comparing costs: a context policy only counts as a saving if its task success stays within a set margin (here, five points) of the baseline.
Observation masking: A context policy that hides older tool outputs from the prompt. The paper's Mask-5 keeps only the last five observations.

The news. In late October 2026, a paper on coding agents proposed a quality-gated way to count the physical work of a context policy: calls, uncached prefill, KV residency, time and an energy proxy. Its headline result is that one history-deleting policy cut prompt volume by a third and still multiplied uncached prefill. Read the paper →

Picture the typesetter again. A coding agent sends the whole manuscript on every call: the task, the tool results so far, and the new step. If the typesetter kept the opening pages already set in type, only the pages after the last unchanged one need new work. That is what a caching runtime does: it prefills only the tokens after the longest cached prefix, so the work on a call is the logical prompt minus the reused prefix.

This is why a prompt-token count is the wrong meter. A call can carry 90,000 tokens and still be cheap, if 87,000 of them are the same opening pages as last time. Prefix caching rewards one thing above all: an unchanged beginning. The paper measured this directly. Turning prefix caching on left the prompt volume of its Eager policy about the same (+8.3%, a change whose 95% confidence interval spans zero) but cut its uncached prefill by 67.7%.

Now tear a page out of the middle. A context policy that deletes old observations makes the prompt shorter, which looks like a saving. But a prefix cache matches from the first token onward, so the first edited position is where reuse stops, and every page after that edit must be set in type again. The paper tested this with a policy called Mask-5 on 100 SWE-bench Verified tasks: it kept only the last five observations. It cut prompt volume by 33.3% but lost 135.9K reusable tokens per task, raised uncached prefill by 608.5% and the energy proxy by 125.6%, and it missed the quality gate.

The lesson is not that trimming context is bad. The lesson is that the unit of cost is the executed prefill, not the prompt. Policies that shorten the trajectory itself save real work: the paper found that preloading ten retrieved blocks (Eager) cut model calls by 35.7% and the GPU-energy proxy by 33.2% against searching on demand (Grep). Policies that edit already-cached history can multiply the work. Context engineering (see the four fixes) needs this second check.

Policy / change	What the paper reports	Why it moves physical work	Source
Eager vs on-demand search (Grep)	Model calls −35.7%, GPU-energy proxy −33.2%	Fewer calls means fewer resent prompts	arXiv 2610.34763
Eager, prefix caching on vs off	Prompt volume +8.3% (CI spans zero), uncached prefill −67.7%	Reused prefix is not prefilled again	arXiv 2610.34763
Anchored compaction (Compact)	Passes the quality gate, saves nothing further over Eager	Edits history, so little net work is saved	arXiv 2610.34763
Mask-5 (last five observations only)	Prompt volume −33.3%, uncached prefill +608.5%, energy proxy +125.6%, fails the quality gate	Rewrites the cached prefix, so most of the prompt is prefilled again	arXiv 2610.34763

Worked example (illustrative numbers, not from the paper). An agent is on step 40. Its logical prompt is 90,000 tokens, and the last call already cached all but the newest 3,000. With history kept, uncached prefill is 3,000 tokens. Now a policy deletes old tool output and the prompt drops by a third to 60,000 tokens, but the first deleted block started at token 10,000. The cached prefix now ends at 10,000, so uncached prefill is 60,000 − 10,000 = 50,000 tokens. The prompt fell 33%, and the executed prefill went from 3,000 to 50,000 tokens, about 17 times more. Hold the same three numbers fixed (prompt length, reused prefix, new tokens) and the arithmetic always runs this way: shrinking the prompt only helps if it does not move the point where the prefix stops matching.

For a production agent, the practical change is small: log uncached prefill next to prompt tokens, and gate every context policy on task quality before comparing cost. Cost tools that only show prompt tokens will rate a history-editing policy as a win. The prompt caching step covers prompt caching at the agent layer, the same prefix-reuse idea seen from the application side. The paper's authors measured an energy proxy rather than a metered figure, so treat the energy percentages as relative, not absolute.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → When It Helps, When It Fails

Related explainers

KV-streams: cache-preserving compaction — a way to shrink history without forcing a re-prefill.
AutoCompact: learned compaction timing — when to compact, which decides how often the prefix is rewritten.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based