What is cross-recurrence KV cache sharing?

It lets every recurrence of a looped language model read one KV cache, the one the final recurrence produces, instead of each recurrence keeping its own. BeNice is a training method that makes this safe for quality.

Why does a looped model need a bigger KV cache?

Each recurrence reuses the same weights but still computes its own keys and values for every token. With 4 recurrences the cache is about 4 times the size of a single pass, so it grows linearly with the loop count.

How does BeNice make sharing work?

Pretraining uses stochastic depth to make recurrences converge toward a common limit. Post-training then broadcasts the final recurrence's KV cache into the first recurrence and trains compatible residuals while preserving the model's quality.

LLM· 2026-10-25

BeNice shares one KV cache across a looped model's recurrences — Cross-recurrence KV sharing — What does it mean?

Q: Can BeNice be combined with KV cache quantization?

The paper reports that KV sharing composes with MLA, Kimi Delta Attention and fp8, mxfp8 and nvfp4 cache quantization. The abstract does not state the combined memory savings.

Jargon

Looped model: A transformer that runs one block of layers several times on the same activations instead of stacking different layers. It spends more compute per token without adding parameters.
Recurrence: One pass through the reused block. A model with 4 recurrences runs the block 4 times for each token.
KV cache: The stored keys and values of earlier tokens, so each new token does not recompute them. Full lesson
KV-shareability: The property that earlier recurrences can use the final recurrence's KV cache without hurting quality. The paper shows it appears in some looped models but not reliably.
Stochastic depth: A training technique that randomly drops part of the model's depth on each step. BeNice uses it in pretraining to induce contractive residual dynamics.
Contractive residual dynamics: Each recurrence changes the representation by less than the one before, so the sequence settles toward a single final value, called a limit.
p90 TTFT: The 90th-percentile time-to-first-token: nine in ten requests see their first output token at or before this time. Percentiles and tail latency
MLA: Multi-head latent attention, an attention variant that stores a compressed latent vector instead of full keys and values.

The news. In October 2026 a paper on arXiv introduced BeNice, a two-stage method that makes looped language models shareable across recurrences: one KV cache, not one per pass. The authors report that it matches standard fine-tuning's unshared validation perplexity, and that in a simulated low-memory, long-context setting it raises cache hit rate by up to 31.4%, raises throughput by up to 23.6% and cuts p90 time-to-first-token by up to 83.6%. Read the paper →

Picture an editor who polishes one draft in four passes, using the same checklist every time. A looped model works this way: it runs one block of layers again and again on the same activations, so quality can grow without adding parameters. The cost is the notebook. On every pass the editor jots the notes that later words will look back at, which are the keys and values of attention, and the default arrangement keeps a separate notebook for each pass. Four passes mean four notebooks, and the KV cache is already what limits how many documents a GPU can hold at once. A looped model with 4 recurrences therefore pays the cache bill four times, even though its weights were paid for once.

The obvious shortcut is to let every pass read one notebook, the last pass's. The hard part is that this only works if the passes write notes that are close enough to interchangeable. The paper calls that property KV-shareability and reports that it shows up in some looped models but not reliably, even among identically trained ones. BeNice builds the property in two stages. Stage one is pretraining with stochastic depth, which induces contractive residual dynamics, so representations are driven toward one common limit across recurrences, like drafts converging on a final text. Stage two is post-training: the abstract describes it as broadcasting the final recurrence's KV cache into the first recurrence and training that recurrence to produce compatible residuals, while preserving the original model's quality. The result is one cache that serves all recurrences.

Approach	What is stored	Cache growth with recurrences
Per-recurrence cache (default)	Full keys and values for every recurrence	Linear: 4 recurrences cost 4 caches
Quantize each recurrence separately	Every recurrence, at fewer bits	Still linear, with a smaller constant
Store inter-recurrence residuals (ResidualQuant)	One reference plus a low-bit difference per other recurrence	Still one entry per recurrence, each much smaller
Share the final cache (BeNice)	One cache, read by every recurrence	Flat: the same cache at any recurrence count

Here is the arithmetic with every variable held fixed (illustrative numbers, not from the paper). Take a reused block of 12 layers with 8 KV heads of dimension 128, stored in fp16 (2 bytes). One layer stores 2 × 8 × 128 × 2 = 4,096 bytes per token, so the block stores 12 × 4,096 = 48 KiB per token per recurrence. With 4 recurrences that is 192 KiB per token, and a 32,768-token context needs 192 KiB × 32,768 = 6 GiB per sequence. Sharing keeps one recurrence's worth, 1.5 GiB per sequence. If 40 GiB of GPU memory is left for caches, the server holds floor(40 / 6) = 6 sequences before and floor(40 / 1.5) = 26 after: 6 → 26 concurrent sequences from the same hardware. Storing the shared cache in fp8 instead of fp16 would halve it again to 0.75 GiB, or 53 sequences, because the two savings act on different axes: sharing removes copies, quantization shrinks each entry.

The headline numbers come from a simulated low-memory, long-context setting, and each is an upper bound ('up to'). Cache hit rate rises by up to 31.4%, throughput by up to 23.6%, and p90 time-to-first-token falls by up to 83.6%. One plausible route, which is our inference and not a mechanism the paper states, is that a smaller cache per request keeps more requests resident in memory. Two cautions follow. BeNice is a training recipe, not a serving switch: the abstract places stochastic depth in pretraining and the broadcast step in post-training, so it is not something to flip on at serving time. And the paper reports composition with MLA, Kimi Delta Attention and fp8, mxfp8 and nvfp4 cache quantization, but the abstract does not give the combined savings.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

ResidualQuant stores looped-Transformer KV caches as 2-bit residuals — the other way to shrink a looped model's cache: keep every loop's entry, but store only the difference.
SMELT loops the middle half of an MoE transformer twice — what depth reuse buys in the first place.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based