Attic-KV — Query-conditioned rehearsal vs full-context rereading — What does it mean?
The news. On October 8, 2026, Zhiyun Shi of Nanyang Technological University posted "Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read" (arXiv 2610.12133). The paper argues that rehearsal-scoring KV compressors fail at tight budgets because of what they rehearse, not how they score it. Its fix changes only the rehearsal text, needs no training, and plugs into existing compressors. On RULER-4K with Qwen3-8B at a 3% keep ratio it scores 73.4, against 31.5 for full-context rereading (KVzip+). Read the paper →
Picture the night before an exam you may bring one index card into. Rereading the whole chapter feels thorough, but the card fills with whatever caught your eye, which is mostly prose. Writing practice questions and answering them from the book puts the facts an examiner would ask on the card instead. In the paper's words, a cache remembers more by testing itself than by rereading.
In KV-cache terms, the chapter is a document the model has already prefilled into keys and values. Rehearsal scoring, introduced by KVzip, compresses that cache before any question arrives: the model reads a rehearsal text after the document, every cached entry gets a score equal to the strongest attention weight a rehearsal token puts on it, and only the top few percent of entries survive. The rehearsal stands in for the unknown queries, and KVzip and its successors use the whole document as that stand-in.
The paper's central finding is that a cache keeps what it rehearses. On the RULER-4K development split with Qwen3-8B at a 5% keep ratio, rehearsed positions covering about 5% of a document drew 20.6–32.9% of the kept budget. Rereading the whole document singles nothing out, so the budget spreads over fluent prose. Capacity is not the problem: when the rehearsal was the question plus its gold answer, the same 5% budget scored 93.5, against 94.7 uncompressed.
How Attic-KV builds the rehearsal
Attic-KV changes only the rehearsal text and leaves the host compressor's scorer alone. The rehearsal has three parts:
- Self-generated question-answer pairs. With the full cache, the model writes the questions a reader is likely to ask and answers each by quoting the document. The answer matters: rehearsing questions alone scored 47.0, against 93.5 with answers, because attention lands on a fact's value only while the answer is being produced.
- Anchors. Tokens whose keys stand out from the chunk's average key decide what else to rehearse.
- A content-adaptive amount. Each anchor expands to the length of its sentence, so a prose chunk is rehearsed briefly and a dense list of facts almost in full. At an equal average of about 35% rehearsed, this beat KV²'s fixed fraction 78.0 to 74.6 on the development split.
Because only the rehearsal changes, the same text plugs into other compressors: added to the gradient-based KVgrad it raised scores by up to 17.1 points, and added to the trained RestoreKV+ by up to 28.1 points. It is also faster than rereading, because it rehearses about 42% of each context on RULER-4K rather than all of it: compressing a document from LooGLE (a long-document question-answering benchmark) and answering all of its questions took 126.0 s, against 154.8 s for KVzip+.
| Method | What it rehearses | Rehearsal length (share of context) | RULER-4K, 5% kept | RULER-4K, 3% kept | Source |
|---|---|---|---|---|---|
| Compactor | nothing (scores without rehearsal) | 0 | 35.3 | 25.5 | Table 1 |
| KVzip+ | the whole context | 1.00 | 58.6 | 31.5 | Table 1 |
| KV² | a fixed share of anchors | 0.35 | 76.8 | 67.0 | Table 1 |
| Attic-KV | self-quiz Q+A plus adaptive anchors | 0.42 | 82.1 | 73.4 | Table 1 |
| Full cache (reference) | not compressed | not applicable | 96.5 | 96.5 | Table 1 |
Worked example: what a 5% cache buys, and where its budget goes
Hold three things fixed: Qwen3-8B, the 4K-token RULER tasks, and a 5% keep ratio. Memory first: the paper notes that a 5% ratio fits 20× as many caches in the same memory and a 30% ratio only about 3×, so a GPU budget that holds 10 full document caches would hold about 200 caches at 5% but only about 33 at 30% (illustrative count). Now follow one answer through compression. A cache that kept entries at random would keep about 5% of the answer's KV entries. On the development split, full-context rereading kept 5.7–7.1% of the answer's entries on seven of eight needle tasks, so the answer was almost as likely to be dropped as any filler word, while Attic-KV kept 15.0–17.0% on the five needle tasks it solved, roughly 3× chance. On the test split the score moved accordingly, from 58.6 for rereading to 82.1 for Attic-KV. At a 3% keep ratio the gap widens to 73.4 versus 31.5, a 41.9-point lead.
Where it helps and where it does not
The gain is largest exactly where compression pays off: tight budgets and fact-dense text. On LongBench's natural-text tasks at 5%, Attic-KV leads KV² by only 40.6 to 39.1, and on LooGLE's long-dependency questions it scores slightly below KV² on ROUGE-L, a text-overlap score (22.9 vs 23.6). Two RULER tasks built from long key-value lists (MK2 and MK3) stay unsolved, because the self-quiz misses most of their facts and the cache keeps only what the quiz reads. The paper names shared prompt prefixes as one target, the same setting where prefix caching reuses one stored cache across many requests. For other ways of choosing which KV entries to keep, see InfoKV and risk-controlled KV eviction.
Goes deeper in: LLM Internals → KV Cache → Memory Cost