A serving study traces agent irreproducibility to prefix-cache state — Cache-state divergence — What does it mean?

The news. On September 4, 2026 a paper appeared on arXiv measuring how reproducible agent serving actually is. The authors ran an 80-episode multi-turn tool-use workload serially at batch size one (one request in flight at a time, so batching cannot be the culprit), holding the model, the decoding parameters, the seed and the request order fixed — the full set of knobs an eval harness normally pins. With prefix caching enabled, 36.2% of trajectories still diverged at 16-bit precision, and 75.0% diverged at 4-bit. With caching disabled, 0 of 800 episodes diverged. Controlled cache settings, execution-order tests and restored cache snapshots all pointed the same way: each path is deterministic given the cache state, and the request never carries that state. Read the paper

The jar is an input, and nobody wrote it down

Two bakers work from the same recipe card. Same flour, same oven, same timer, same order of steps. One loaf comes out open and springy, the other tight. Nothing on the card explains it, because the thing that differed was the starter jar on the shelf — how recently it had been fed, what the last bake took out of it. The card is a complete description of the instructions, and a partial description of the bake.

A request to a prefix-cached inference server is exactly that card: complete about the instructions, silent about the state. When the server processes a prompt it keeps the attention state for the prefix, so the next request that opens with the same tokens can skip recomputing it. That is the whole point of the feature, and it works — reuse is why serving stacks offer prefix caching at all. But reuse is not free of consequences. The paper does not isolate why a reused block gives a slightly different answer than a freshly computed one; the standard explanation, and an inference rather than a result here, is that floating-point arithmetic is not associative, so recombining cached blocks sums the same quantities in a different order. Either way, what the study measures is the outcome: the result is numerically close, not numerically identical.

For a single next-token choice, "numerically close" is almost always enough — the top token wins by a comfortable margin and a tiny difference changes nothing. An agent episode is not a single choice. It is dozens of model turns, each feeding a tool call whose result feeds the next turn. One flipped token early — a different argument to a search call, a different file path — and the two runs are reading different things from step three onward. That is what divergence means here, and it is why the paper's framing is sharper than "inference is a bit noisy": the noise is not the interesting part, the amplification is. The study's own summary of the situation is blunt: cached serving is deterministic given cache state, and irreproducible in practice because that state is absent from the request.

The control that proves it is the boring one. Turn caching off, and 800 episodes produce 0 divergences — the seed and the decoding settings really were doing their job. Restore a saved cache snapshot before a rerun, and the run repeats. The cache is not adding randomness; it is adding an input that your harness forgot to record.

Why 4-bit makes it worse

The jump from 36.2% to 75.0% when the weights drop to 4 bits is the result most worth sitting with, and it is worth being careful about what the paper does and does not say. It reports the doubling; the Mechanism it describes is the measurement setup, not an isolated cause. The standard explanation — a reasonable inference, not a claim the paper proves — is about margins. Quantization snaps each weight to one of a small set of representable values, and at 4 bits that set has 16 members rather than the thousands a 16-bit format offers. The plausible reading is that coarser weights can leave a next-token decision more sensitive to a small numerical difference, so the same nudge is more often enough to flip which token wins. Whether that is the actual cause here, and how far it carries beyond this model and this workload, the study does not say.

The number line below is the same idea one layer down: watch how many distinct values survive as the precision drops from FP32 to INT4.

Working the numbers on one eval suite

Illustrative projection, not reported episode counts. The paper reports two divergence percentages, an 80-episode workload, and a count of 800 episodes for its caching-disabled arm. The arithmetic below applies those percentages to a suite of 800 so the numbers are easier to feel; treat the derived counts as a sense of scale, not as results the paper published.

Take a suite of 800 episodes — the count the paper reports for its caching-disabled arm — and hold the model, the seed, the decoding settings and the request order fixed. Change one thing at a time.

Caching off, the paper's own figure: 0 of 800. Every rerun repeats, which is what your harness was designed to assume.

Caching on, the paper reports 36.2% of trajectories diverging at 16-bit — on a suite this size that would be about 290 episodes. At 4-bit it reports 75.0%, which would be about 600. Changing precision alone would move roughly 310 more episodes into the divergent pile, more than a third of a suite that size changing behaviour because of a serving flag and a weight format.

The last figure needs no projection at all. A single prompt-cache setting shifts divergence by 37.5 percentage points — on a suite of 800 that would be about 300 episodes' worth of behaviour riding on a configuration line nobody puts in the run manifest. If your A/B harness compares an arm captured on Monday against an arm captured on Friday, and the cache was warm on one and cold on the other, a chunk of your measured effect is that.

What this changes about running evals

The practical reading is not "turn prefix caching off." Prefix caching can cut a lot of repeated compute on workloads with reusable prefixes, and the prefix-caching module's closing step already treats it as a feature with a shape — good for shared system prompts and long tool schemas, weak when prefixes rarely repeat. This paper adds a second axis to that judgement: caching is also the difference between an eval you can rerun and one you cannot.

So treat cache state the way you already treat the seed — as something the run manifest records, not something the server happens to be in the mood for. That means one of three things depending on what you are measuring. For an offline regression suite, run it with caching disabled and accept the cost, since the point of the suite is comparability across weeks. For a staged rollout where the numbers must reflect production, keep caching on but give every arm the same starting point: reset the cache before each arm, or restore every arm from one saved snapshot. Taking a fresh snapshot per arm does not achieve this — two snapshots captured at different moments are two different starting states. For the 4-bit configuration this study evaluated, the rate was high enough that a single run tells you very little. Whether your own quantized setup behaves the same way is something to measure per model and workload rather than assume.

There is a mirror-image lesson for the Online vs Offline Evals step. An offline eval's whole claim is that it is a controlled reproduction of the online system. A control you did not know existed is not a control — and this one went unnoticed until someone checked.

Goes deeper in: LLM Serving → Prefix Caching → When It Helps, When It Fails

Related explainers

• ReCache reuses tool-schema KV blocks across agent calls — the block-composition side of the same machinery: what has to hold for a reused block to be safe.
• CacheWeaver reorders RAG evidence for prefix-cache reuse — reordering inputs to raise hit rate, which is exactly the kind of change this study says to log.
• CacheRoute sends repeated prefixes to the server that still holds them — when the cache lives on one replica, which server answered becomes part of the hidden state.