iS-KV is an online **low-rank KV cache compression** method for long chain-of-thought decoding. It keeps recent tokens exact, stores older keys and values as short coefficient vectors over a shared basis, and updates that basis and all old coefficients together with block-incremental SVD.

Why not just evict unimportant tokens?

Because tokens that look unimportant now often matter later: in the paper's AIME traces about 30% of the least-attended tokens got strong attention again within 8K tokens. At matched memory on MATH-500 with Qwen3-8B, eviction with R-KV scored 68.8% while iS-KV scored 89.2%.

What is basis synchronization?

When new tokens update the low-rank basis, the old tokens' coefficients were computed for the previous basis. Keeping them unchanged rebuilds different vectors, which the paper measures as large drift; synchronization rewrites the old coefficients in the new basis at the same time.

How does iS-KV relate to GQA and KV quantization?

They shrink different parts of the cache. [GQA](/tracks/llm-internals/kv-cache?step=gqa) reduces the number of KV heads, quantization reduces the bits per stored number, and iS-KV reduces the number of coordinates stored per old token. The paper does not test combining iS-KV with low-bit quantization.

LLM· 2026-10-06

iS-KV — Basis-synchronized online low-rank KV compression — What does it mean?

TL;DR

What is it: The iS-KV paper (arXiv, October 2026) introduces basis-synchronized online low-rank KV compression: during long reasoning it stores each older token's key and value as a short list of coefficients over a shared basis instead of deleting the token, and rewrites those coefficients every time the basis changes.
Why it’s needed: Long chain-of-thought traces add one KV cache entry per generated token, and a token that looks unimportant now can be needed again thousands of tokens later, so the cache has to shrink without losing positions for good.
vs previous: Token eviction methods such as R-KV and SnapKV delete low-scoring tokens and can never bring them back, and a naive online low-rank update that changes the basis but leaves old coefficients alone makes the stored history drift away from the states it was meant to hold.

Jargon

KV cache: The stored key and value vectors for every token the model has already processed, reused at each decode step so they are not recomputed. It grows by one entry per generated token. KV Cache → Memory Cost
Token eviction: Shrinking the KV cache by deleting some token positions, chosen by attention scores, recency or redundancy. R-KV and SnapKV are the eviction methods this paper compares against.
Low-rank: A matrix is low-rank when a few directions explain most of its variation, so each row can be written as a short combination of those directions with little error.
Basis and coefficients: The basis is the small shared set of directions; a token's coefficients are the amounts of each direction that rebuild its vector. The same coefficients mean a different vector if the basis changes.
SVD: Singular value decomposition: the standard way to find the best low-rank approximation of a matrix. Truncated SVD keeps only the strongest r directions, where r is the rank.
Block-incremental SVD: A classical method (Brand, 2006) that updates an existing SVD when new rows arrive, without refactoring the whole history. iS-KV applies it to each 64-row block of tokens.
Pre-RoPE keys: Key vectors taken before rotary position embedding rotates them by position. The paper reports they are more strongly low-rank than rotated keys, so iS-KV compresses keys at that point and re-applies RoPE when it rebuilds them.
Persistent-KV compression: Full KV bytes divided by the bytes iS-KV keeps between decode steps: the low-rank factors plus the exact windows and protected prompt rows. It excludes model weights and temporary rebuild buffers, so it is not a peak GPU memory ratio.

The news. On October 2, 2026, researchers from HKUST (Guangzhou), the Shenzhen Institutes of Advanced Technology, the University of Macau and OPPO posted iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD on arXiv. They test it on DeepSeek-R1-Distill-Llama-8B and Qwen3-8B, mainly on the MATH-500 math benchmark with a 16K-token limit, and compare it with the eviction methods R-KV and SnapKV at matched memory. At all six matched-memory settings, iS-KV scored higher than both eviction baselines. Read the paper →

Picture a paint shop that must be able to remake every color it has ever mixed, but has counter space for only a few real swatches. One option is to throw old swatches away. That is token eviction, and a customer who asks for a thrown-away color is out of luck. The other option is to keep a small set of base paints and write each old color down as a recipe: so much of paint one, so much of paint two, so much of paint three. iS-KV takes the second option for the KV cache: every token position stays, but older keys and values are stored as short coefficient vectors over a shared basis instead of as full vectors. The newest tokens, a recent window of 128 in the paper's main runs, stay exact like the swatches on the counter, and a set of protected prompt positions also stays exact.

Why not just bin the quiet colors? In long reasoning, a token the model ignores now is often one it needs later. The authors marked the 75% of tokens with the lowest recent attention at decoding step 1,024, removed nothing, and kept watching: on the AIME math problems about 30% of those tokens received strong attention again within the next 8K tokens, and about 60% did by 30K tokens. Every new query scores every earlier key, so a deleted key is something the model can no longer look at. Keeping old tokens in compressed form is possible because generated keys and values are strongly low-rank: a handful of directions explain most of their variation, and keys show this most clearly before RoPE rotates them by position. The diagram below shows the eviction policies that iS-KV replaces.

Remove earliest tokens first

Now the hard part of the paint shop. New colors keep arriving: every time 64 tokens leave the recent window, they are folded into the compressed history as one block. The best base paints for the old colors are not the best base paints for old plus new, so the basis is updated. If you change the base paints but keep the old recipes, every old recipe now mixes a different color, and the stored history drifts even though nothing touched it. The paper measures this. At step 256, updating only the basis gave a drift of 1.55 for keys and 1.40 for values on the paper's drift measure; updating the basis and the old coefficients together cut it to 0.023 and 0.051. Drift here is the change in the stored history caused by one update, not the error against the original uncompressed cache. iS-KV does the joint update with block-incremental SVD, a classical method by Brand (2006) that rewrites the basis and every stored coefficient in one step. The decomposition itself is solved on a small core matrix whose size does not depend on the history length, although rewriting the stored coefficients still costs more as the history grows.

A worked example from the paper, holding three things fixed: Qwen3-8B, the MATH-500 benchmark, and a persistent KV budget of about 17.7% of the full cache (5.64× compression), counted in bytes the same way for every method. Uncompressed, the model answers 94.4% correctly. R-KV, which keeps a selected subset of tokens inside that budget, drops to 68.8%. iS-KV at rank 96, which keeps every position in exact or low-rank form, reaches 89.2%. Same memory budget, and keeping every token approximately beats keeping some tokens exactly by 20.4 points. The matching counts the low-rank factors, the exact rows and R-KV's 128-token staging buffer, and excludes model weights and temporary rebuild buffers.

Model (MATH-500, matched memory)	Compression	Full cache	R-KV	SnapKV	iS-KV	Source
DeepSeek-R1-Distill-Llama-8B	4.06×	83.6%	78.2%	70.1%	82.6% (rank 128)	Table 1
DeepSeek-R1-Distill-Llama-8B	5.92×	83.6%	72.6%	64.3%	78.2%	Table 1
Qwen3-8B	5.64×	94.4%	68.8%	73.17% (reported)	89.2% (rank 96)	Table 1
Qwen3-8B	≈7.1×	94.4%	60.8%	65.04% (reported)	87.8% (rank 64)	Table 1

The trade is that memory still grows and generation gets slower. Each compressed token is cheaper, but each one is still stored, so iS-KV slows cache growth rather than capping it. In a single-run timing test on one NVIDIA A40 (DeepSeek-R1-Distill-Llama-8B, rank 128, 1,024 decode steps after the input), KV memory fell from 4.125 GiB to 1.808 GiB with a 32K input, a 56.2% saving that shrinks to 33.7% with an 8K input, while total generation time rose by 8.8% to 22.2% across the 8K to 32K inputs. The authors call these single runs rather than a stable latency measurement, and they did not test low rank combined with low-bit quantization or inside a serving engine. Low rank is a third lever next to two the curriculum already covers: GQA cuts the number of KV heads, quantization cuts the bits per number, and iS-KV cuts the number of coordinates stored per old token.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

KV-COBRA: rank vs bit-width allocation per head — a fixed low-rank projection whose rank is chosen per head; iS-KV instead moves its basis online during decoding.
Risk-controlled KV cache eviction — the eviction side of the trade: how much cache to drop, certified against a failure target.
WorldKV: evict-and-reinsert KV memory — another answer to "old tokens come back", which stores evicted chunks and reinserts them.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based