Latent Compaction (arXiv 2610.20582) — Training a document-specific KV cache — What does it mean?
The news. On October 14, 2026, a Tsinghua University group posted "Latent Compaction with Test-Time Training". With model weights frozen, it trains a compact cache for one document and reports 96.7% accuracy on RULER at 32× compression, against 40.0% for KVZip+ at the same budget. The gains held on Qwen3 and MiniCPM5. Read the paper →
Picture a student who may bring one page into an exam on a 400-page textbook. The exam questions are not known yet. The usual approach is the highlighter: keep the most important lines and cross out the rest. That is what eviction does to a KV cache: it scores every stored entry and deletes the lowest-scoring ones. The page you carry is a strict subset of the book.
Latent Compaction asks a different question: why must the page contain only lines from the book? A student can also rewrite the page. So the method starts the sheet with copied lines, then drills with practice questions and edits the sheet until it answers the way the whole book would. The model is the student's brain and it never changes. Only the sheet does.
The compact cache has three kinds of entry, and each one has a different job. Preservers are the highest-scoring original key-value pairs under KVZip+, chosen across all layers and heads together, so some heads get more of them than others. Anchors are original entries that the full model attends to heavily, plus the first token, which often acts as an attention sink. Diversifiers are chosen by farthest-first traversal: each new key is the one farthest from every key already picked, which spreads the cache across the key space.
Training then runs in two stages. Stage 1 works one layer at a time: with preservers frozen, it adjusts the anchors and diversifiers so the compact cache reproduces the full cache's attention output and its log-normalizer on queries recorded from the frozen model. Stage 2 unfreezes everything and distills end to end: the compact-cache model must match the full-cache model's next-token probabilities, scored on its top 20 tokens plus one lumped tail, plus a standard language-modeling loss on the teacher's greedy token.
The training signal never contains the real questions. The paper builds 480 generic prompts (24 document requests, each paired with 20 response instructions), lets the full-cache model answer each one greedily for at most 24 tokens, and uses those answers as the teacher. That is the "drill with practice questions" step: the sheet is shaped by questions about the document in general, then frozen and reused for every later question.
One detail makes this work with a real question attached. Question tokens and generated tokens are appended as ordinary entries after the compact cache, and the attention over both is combined with their normalizers, which equals one softmax over everything. That is why matching the log-normalizer in stage 1 matters: it sets how loudly the document speaks relative to the question. Averaging the two outputs equally instead, with no normalizer weighting, dropped Phonebook Lookup accuracy to at most 0.25%.
| Approach | What the small cache is made of | Are entries optimized? |
|---|---|---|
| Eviction (H2O, SnapKV, PyramidKV, KVZip+) | A subset of the original key-value pairs | No, entries are kept or deleted |
| Text summary | A shorter text that is prefilled again | No, the cache is rebuilt from text |
| Cartridges | A compact cache trained per corpus with self-study and context distillation | Yes |
| Attention Matching | Selected keys with learned biases and values that fit full-cache attention outputs and mass | Yes, to match attention only |
| Latent Compaction | Preservers, anchors and diversifiers with learned biases | Yes: attention fitting, then end-to-end distillation |
Take one 4,096-token document at 32× compression. The average budget is 4,096 / 32 = 128 entries per layer and KV head (the paper allocates budgets as averages, so single heads can differ). A quarter of it, M = 128 / 4 = 32, is reserved for anchors and diversifiers, and a quarter of those 32 are anchors: 8 anchors and 24 diversifiers. The remaining 96 slots are preservers on average (the paper reports 96 and 32 per head). So on average, each head goes from 4,096 entries to 128, but only 32 of the 128 are free to be rewritten in stage 1; stage 2 then lets all 128 move.
The ablations on Phonebook Lookup (Qwen3-1.7B, 16× compression, average over 4K, 8K and 16K contexts) show that each part carries weight. The full method scores 80.67%. Removing stage 1 drops it to 13.92%, making the preservers trainable in stage 1 drops it to 33.42%, and removing the log-normalizer loss drops it to 36.50%. Removing stage 2 gives 70.67%.
Treat the 32× figure as a statement about entries, not about memory or speed. The paper says its budgets are logical KV-pair counts, and it reports no measured memory savings or inference speedups. It also lists three limits: per-document fitting is costly, text summarization beats the method on RULER at the most aggressive ratios, and all compressed methods fall close to zero on Phonebook Lookup at 64× and beyond. The cost is paid once per document, so the idea suits long documents that many questions will reuse.
The design choice to remember: a KV cache does not have to be a subset of what the model computed. It is a set of vectors the next tokens attend to, so it can be optimized like any other small parameter block.
Goes deeper in: LLM Internals → KV Cache → Memory Cost
Related explainers
- KV-streams — Keeping the KV cache through context compaction — compaction that keeps the cache valid instead of re-prefilling it
- KV-COBRA — Per-head rank vs bit-width allocation — a different way to shrink the same cache, by spending a budget unevenly across heads