What is value cache amortisation?

It replaces most of a head's stored value vectors with a small network that predicts each value from the token's key. The network is fitted to the current context during prefill, and exact **residuals** are kept only for the tokens it predicts worst, so those tokens are reconstructed exactly.

Why does LowRAM predict values from keys and not keys from values?

Both are needed at decode, and the keys must stay because they set the attention weights and feed the predictor. The authors suggest key errors are likely more harmful: a key error can shift which tokens the head attends to, while a value error only changes what is mixed in after the weights are fixed. In the paper's comparison, predicting current-layer values from current-layer keys preserved the most accuracy, so keys get low-rank compression instead.

When does LowRAM pay off?

On long contexts. The predictor is a fixed memory cost that the paper finds is only amortised after a few thousand tokens, and fitting it adds prefill time, from **1.3×** time to first token with a meta-learned start up to **26.3×** with 150 fitting steps. It suits long documents and shared prefixes that serve many queries.

How does LowRAM relate to KV cache quantization and eviction?

It uses a different redundancy, so it stacks with both. On Mistral-7B, adding quantization to 4× LowRAM raised compression to **6.4×** and the LongBench score from 97.9% to 99.4%; adding eviction too reached **26.9×** at 94.5% on contexts over 10K tokens.

LLM· 2026-11-10

LowRAM KV cache compression — Predicting the value cache from keys — What does it mean?

TL;DR

What is it: LowRAM (arXiv 2611.07025) is a KV cache compression method built on value cache amortisation: inside each attention head it predicts every token's value vector from its key with a small network fitted to the current context, and stores exact corrections only for the tokens it predicts worst.
Why it’s needed: The KV cache grows with every token of context and every request in a batch, so at long context it is the memory that caps batch size and decode throughput; a new way to shrink it, one that stacks with the existing ones, raises that cap.
vs previous: Earlier post-training methods drop tokens (eviction), store fewer bits (quantization), or factor the cache into a smaller basis (low-rank), and they largely leave the redundancy between a token's key and its value unused; LowRAM predicts values from keys and compresses the keys separately.

Jargon

KV cache: The stored key and value vectors of every earlier token, kept so each new token does not recompute them. How it works
Key and value: Two vectors each token produces per attention head: the key is matched against the current query to set attention weights, the value is what gets mixed into the output with those weights.
Attention head: One of several parallel attention units in a layer, each with its own key and value projections. LowRAM fits one predictor per head.
GQA: Grouped-query attention: several query heads share one key/value head, which makes the cache smaller and the key projection narrower than the hidden state. GQA step
Residual: The difference between the true value and the predicted one. Adding it back gives the exact value for that token.
Prefill / decode: Prefill processes the whole prompt in one pass and fills the cache; decode then generates one token at a time, reading the full cache each step.
Meta-learned initialisation: Starting weights trained (here with First-Order MAML) so that a predictor needs only a few gradient steps to fit a new context.
RULER / LongBench: Long-context benchmarks. Results here are reported as the percentage of the uncompressed model's score that survives compression.

The news. On November 4, 2026, researchers from Huawei Noah's Ark Lab (London) and University College London posted LowRAM (arXiv 2611.07025). The paper reports 4× near-lossless compression of the value cache on its own, up to 5× higher peak decode throughput at 4× KV compression on Llama-3.1-8B at a 262K-token context, and 26.9× compression at 94.5% of baseline performance on long LongBench contexts when combined with eviction and quantization. Read the paper →

Picture a bakery that takes 30,000 orders in an evening and keeps two things: a shelf with every finished cake, and the order ticket for each one. The shelf is the expensive part, and the baker finds that one recipe can recreate most cakes from their tickets. So the baker reads tonight's tickets, writes one recipe that turns a ticket into a cake, and checks the recipe against the real cakes. For the few orders where the recipe gets it badly wrong, the baker staples a correction note to the ticket. Then the shelf is emptied. When a customer comes back, the baker bakes their cake from the ticket and applies the note if there is one.

In a transformer, each cached token has a key and a value in every attention head (the three roles). Both are computed from the same hidden state, so the key already carries much of the information in the value. LowRAM keeps the tickets (the keys), learns the recipe (a small two-layer network per head, fitted during prefill to this exact context), and stores correction notes (exact residuals) only for the tokens the network reconstructs worst. At decode time the values are rebuilt as predicted value + residual where one exists and fed into ordinary attention.

Why can a key predict a value at all? Both come from one hidden state x: the key is x·W_K and the value is x·W_V. When the full key projection (all heads together) is square and invertible, earlier work called Slim Attention pointed out that values can be recovered from keys exactly with one matrix multiply. With grouped-query attention the key projection is narrower than the hidden state, so some information is lost (why GQA shrinks the cache). LowRAM therefore starts its predictor from the best linear guess the weights allow (a pseudoinverse) and lets a small nonlinear branch learn the rest from the current context.

Three details make the recipe work. The map is head-specific: sharing one predictor across all heads in a layer cost almost 20 points in the paper's ablation. Corrections must go to the worst tokens: at a fixed residual budget, ranking tokens by error recovered almost all of the accuracy, while picking tokens at random lost most of it. And the error ranking itself is not the goal: the authors found that reconstruction error predicts downstream accuracy only weakly, so they chose the design by benchmark results instead.

Values get a predictor and keys do not: in the paper's tests, predicting keys recovered less accuracy, and the authors suggest key errors are likely more harmful. A wrong key perturbs the attention scores and may shift the head onto the wrong tokens; a wrong value enters only after the weights are fixed, so it changes what is mixed in rather than where the head looks (scores to output). In the paper's comparison, predicting values from current-layer keys recovered the most downstream performance of every source and target it tried. The keys still have to be stored, both for attention and as the predictor's input, so LowRAM compresses them a different way: a low-rank basis shared across groups of adjacent layers (a method called xKV), which works because key energy sits in few directions.

Method	Redundancy it removes	Mistral-7B on LongBench: compression → score kept	Source
Eviction	tokens unlikely to be attended	2× → 99.4%	Table 3
Quantization (TurboQuant)	bits per stored number	5.1× → 98.9%	Table 3
LowRAM	low-rank keys across layers + values predicted from keys	4× → 97.9%	Table 3
LowRAM + quantization	both, with 2-bit residuals	6.4× → 99.4%	Table 3
LowRAM + eviction + quantization	all three axes	26.9× → 94.5% (contexts over 10K tokens only)	Table 4

Where the memory actually goes (illustrative, one head, 128-number vectors, a predictor with hidden width equal to the head size, a residual index counted as two numbers). At 32,768 tokens the value cache for that head holds 32,768 × 128 = 4,194,304 numbers. A 4× target leaves 1,048,576. The predictor's three 128 × 128 weight matrices plus biases take 49,408 of them, a fixed cost. The remaining 999,168 buy 999,168 ÷ 130 ≈ 7,686 exact residuals, about 23% of tokens, so roughly three cakes in four come from the recipe alone. Run the same numbers at 4,096 tokens: the budget is 131,072, the predictor still takes 49,408, and only ≈ 628 residuals fit, about 15% of tokens. At 2,048 tokens the predictor eats three quarters of the budget. The recipe is a fixed cost that only pays off once the context is long, which matches the paper's report that the parameters dominate below roughly 2–3K tokens at 4× and that LowRAM loses more accuracy on 4K-token RULER than on longer LongBench inputs (how the cache grows with tokens).

The recipe also has to be written, and that cost lands on prefill. Fitting the predictor with 150 gradient steps per context raised time to first token to 26.3× the uncompressed baseline. A meta-learned starting point cut adaptation to 2 steps and the overhead to 1.3×, at a cost of 1.4–4.4 points on RULER and 2.4 on long LongBench for Llama-3.1-8B, compared with a full 300-step fit. The paper's rule of thumb follows: use the quick fit when a context is queried once, and the full fit when one long document or shared prefix serves many queries, because the bakery writes the recipe once and bakes from it all night. On the decode side, rebuilding only the chunks the current query selects let a 4×-compressed Llama-3.1-8B run a batch of 28 and reach roughly 5× the baseline's peak decode throughput at 262K tokens — decode is memory-bound, so a smaller cache turns directly into a bigger batch (why decode is bandwidth-bound). The authors note they did not evaluate long generations: newly generated tokens are stored exactly, so a long output would need periodic re-compression.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

iS-KV: basis-synchronized low-rank KV compression — the low-rank axis LowRAM uses for keys, applied to both keys and values
DepthGraft: cross-layer KV reconstruction — rebuilding one layer's cache from another layer instead of from the keys of the same head
FOSCAR: affine folding of low-bit KV dequantization — the quantization axis, and why fewer bits alone did not make decode faster

Continue in trackKV Cache: why the cache grows with every token, and what that costs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based