The news. On August 21, 2026, researchers posted CacheRoute, a routing layer for large-scale LLM serving built on the observation that repeated prefixes only save prefill work if the next request lands on a server that still holds the prefix KV. It periodically builds a routing plan from observed high-rate prefix keys, admits them into a stable warm set, and assigns destinations by expected load. In experiments on Llama-3.3-70B in fp8 across 60 H100 GPUs, it sustains 176 ± 11 QPS (queries per second) at a 3.5-second p99 SLO — reported as 2.3× the strongest of five baselines — and lifts the served KV-cache hit rate from 64.1 ± 1.3% to 93.2 ± 0.5% against cache-blind balancing. Read the paper →

Picture a chain of walk-in clinics. You are a regular, and at one branch a clerk has already pulled your file and left it open on the desk — your history is done, and the visit can start at the actual question. Now the chain installs a dispatcher whose only rule is send the patient to the shortest queue. It is a perfectly sensible rule, and on a busy morning it will route you to a branch that has never seen you, where a clerk retakes your entire history from scratch before anyone gets to the question. The file was ready. It was just in the wrong building.

That is exactly the shape of the problem in a serving fleet. A prefix cache is not a property of the deployment; it belongs to the one replica that computed it. The blocks holding a shared system prompt or a hot document live in the HBM of the specific replica that computed them, so a router that only balances queue depth will happily send the next request that shares that prefix to a different replica — where the cache is cold and the engine has to prefill the entire shared opening again. This is why the module on prefix caching spends a step on when it helps and when it fails: the mechanism is real, and a cache-blind scheduler in front of it can throw most of the benefit away.

The naive repair is to make the dispatcher sticky — hash each prefix key to one branch, permanently. That fixes locality and buys the opposite failure. Perfect affinity simply trades a cache miss for a queue: the most popular prefix now pins all of its traffic onto one replica, which backs up while its neighbours idle. In serving terms that is server skew, and it hurts the tail badly, because tail latency is set by the busiest replica rather than the average one.

CacheRoute's answer is to stop making this a per-request choice at all, and make it a plan. It watches traffic, identifies the high-rate prefix keys, and admits them into what the paper calls a stable warm set. The load-bearing word there is admits: the plan covers the prefixes that actually repeat, not every prompt the fleet sees. (The published summary does not spell out what "stable" guarantees, how large the set is, or how traffic outside it is routed.) It then assigns those keys to destinations by expected load, and here is the release valve: a hot key is allowed more than one destination when its load demands it. In the clinic, that is photocopying the one file half the neighbourhood shares to a second branch — you pay to duplicate it, and in exchange the queue at the first branch stops being the ceiling. The plan is therefore an explicit balance of prefix affinity against server skew, not a commitment to either.

Because the number that matters is a p99 under an SLO rather than an average, the paper is careful about how a plan reaches production: it recommends shadow replay — running the candidate plan against recorded traffic to measure the hit rate and tail it would have produced — before enabling it live. That is the operational half of the idea, and it is the half most easily skipped.

Here is where it earns its keep. Hold the workload fixed: say each request carries a 2,000-token shared prefix plus a short unique tail (illustrative — the paper does not publish a per-request token split). The hit rate decides what fraction of those prefixes must be recomputed. At the cache-blind baseline's 64.1%, 35.9% of requests miss, so per 1,000 requests the fleet re-prefills 359 × 2,000 = 718,000 prefix tokens. Under CacheRoute's 93.2%, only 6.8% miss: 68 × 2,000 = 136,000 tokens. Same traffic, same model, same hardware — the wasted prefill drops about 5.3×, and that reclaimed capacity is what shows up as 2.3× the QPS of the strongest baseline at the same 3.5-second p99 bound.

The reason this is worth noticing is where the win comes from. Nothing about the model changed, nothing about the cache implementation changed, and no memory was saved. What changed is that a decision made one layer above the engine — which replica gets this request — stopped being taken in ignorance of the engine's state. Once the fleet is more than one machine, cache reuse is a placement problem, and the router is the thing doing the placing.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → When It Helps, When It Fails

Related explainers

• CacheWeaver — prefix-cache-aware evidence reordering — the same cache, attacked from the other side: reorder the tokens inside one prompt so more of its opening matches, rather than choosing where to send it.
• Attention Once Is All You Need — persistent KV cache across queries — what happens when the prefix cache stops being per-request and becomes a long-lived object a session keeps advancing.
• SGLang v0.5.14 — LPLB expert-parallel load balancing — the same affinity-versus-skew tension one level down, balancing MoE expert load across GPUs inside a single model.