Compress context up to 266× with one ratio-adaptive model — Matryoshka memory budgets — What does it mean?

The news. On September 10, 2026, a paper introducing FlexComp appeared on arXiv. It targets a practical annoyance in soft context compression: existing compressors are trained for one ratio, so serving both a mild and an aggressive setting means training and hosting two models. FlexComp samples the memory-token budget during training so that prefixes of the learned representation stay usable across the supported budgets, then selects a budget per input at inference using either a confidence cascade or a small learned predictor. The reported results are more than 98% of the mildest ratio's accuracy at up to 266× average compression for the cascade and, for the learned budget predictor, a 50% smaller context KV cache with 47% higher decoding throughput at serving-scale batch sizes. Read the paper →

Picture a news desk. The story is written once, in the old inverted-pyramid style: the first paragraph carries the whole event, the second adds the numbers, the third adds the background. An editor with four column-inches prints the first paragraph, an editor with a full page prints all twelve, and neither of them has to commission a different writer. That single property — any prefix is still a usable version of the story — is what FlexComp trains into a context compressor.

Without it, you are back in the world of the left-hand panel of the illustration: one writer per column length. A compressor trained at 4× produces memory tokens that only mean what they mean at 4×; ask it for a quarter of them and you get a quarter of a sentence, not a shorter sentence. So a serving stack that wants a cheap path for easy requests and a careful path for hard ones ends up training, validating and holding two separate compressors in GPU memory, which is exactly the overhead the cheap path was supposed to save.

FlexComp's training loop makes the budget itself a random variable. Each training instance draws its own memory-token budget K, so the compressor never learns that any one length is the real one; it learns to front-load, putting the load-bearing content in the earliest memory tokens because those are the ones present in every draw. The result is one set of weights and one forward pass, from which you read as many memory tokens as the current request deserves.

That last phrase is where the second half of the paper lives. Front-loading gives you the freedom to cut anywhere; it does not tell you where to cut. FlexComp offers two ways to decide, and they differ in what they spend.

The confidence cascade is the honest, slow option: run the smallest budget, look at how confident the model is in its own answer, and re-run with a longer prefix when it is not. It spends extra forward passes on the hard inputs to buy accuracy, and it is the configuration credited with keeping more than 98% of the mildest ratio's accuracy at up to 266× average compression. The learned K predictor is the fast option: a small model guesses the budget from the input alone, in one shot, and lands at a reported 158–236× compression within 0.7 F1 of the reference. One pays in latency, the other in a little accuracy, and both reuse the same compressor.

The word doing quiet work in all of those numbers is "average". A ratio reported as 266× is not a promise about any single request — it is a statement about the mix, and the mix is what the memory manager actually has to fit.

Work it through with round numbers (illustrative — the split below is a plausible mix, not a figure from the paper). Hold the workload fixed: 100 requests, each carrying 8,192 tokens of context, which is 819,200 context tokens in total. Say the cascade settles 80 of them at the smallest budget of 16 memory tokens, and escalates the remaining 20 to 128 memory tokens. The compressed side is then 80 × 16 + 20 × 128 = 1,280 + 2,560 = 3,840 memory tokens, so the workload-wide compression ratio — total context tokens divided by total memory tokens — is 819,200 ÷ 3,840 = 213×. Average the per-request ratios instead, and the very same mix reads as 422× (512× on eighty requests, 64× on twenty), which is why a reported ratio is meaningless until someone says which average it is. Now serve the same 100 requests with a single fixed compressor conservative enough for the hard ones — 128 memory tokens for everybody — and you store 100 × 128 = 12,800 memory tokens, 3.3× more cached context for the same request mix. That gap is the whole argument, and it also exposes the catch: the 20 escalated requests still each occupy 128 memory tokens of KV cache, so the hard requests set the largest per-request context cache while the easy ones set the flattering average — and total server memory then depends on how many of each are in flight at once, a distinction that matters the moment you size a machine rather than a benchmark.

It is worth being precise about which memory FlexComp touches, because the KV cache has two very different occupants. The keys and values for the generated tokens are produced one at a time during decoding and cannot be compressed away by anything that happens to the prompt. The keys and values for the context — the documents, the history, the retrieved passages — are the part a context compressor replaces, and that is the part the paper's learned-predictor result — a 50% smaller context KV cache — refers to. The paper separately reports 47% higher decoding throughput at serving-scale batch sizes, and the standard reason a smaller per-request cache raises decoding throughput is worth holding onto: more requests fit in the batch at once, and decoding is memory-bandwidth bound, so a fuller batch turns into more tokens per second.

This puts adaptive compression in the same family as the architectural tricks that shrink the cache at the source, such as grouped-query attention, but it arrives from the other end. GQA changes what a token costs to store; FlexComp changes how many tokens there are to store, per request, at serving time. They compose, and neither of them is visible in a latency number until you look at goodput rather than raw throughput — a cascade that quietly re-runs a slice of its requests — a fifth of them, in the illustration above — is doing more work than its average ratio suggests.

The honest limits are the ones the paper does not close. The reported ratios are averages over specific benchmark mixes, so a workload whose inputs are uniformly hard will sit near the expensive end of the range and see far less of the headline number. The cascade's escalations cost real forward passes, which a throughput figure measured at a fixed batch size will not show you. And a compressor trained to front-load is making a bet about what matters early — an input whose decisive detail sits in the last paragraph is precisely the case where a short prefix is confidently wrong rather than usefully uncertain. The mechanism worth taking away is the training trick, not the 266×: make the budget a variable during training, and a single model stops being a single operating point.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

• Encoder-decoder prompt compression — the fixed-ratio soft compressor FlexComp is reacting to; read it first if "memory tokens" is new.
• Entropy-aware KV-cache compression — the other end of the same problem: compressing the cache after it exists, rather than the context before it does.
• Cost-aware LLM cascades — the same cheap-first, escalate-if-unsure pattern applied to model choice instead of compression ratio.