KV-COBRA (Co-Optimized Bit-Rank Allocation) is a KV-cache compression method from a [September 2026 arXiv paper](https://arxiv.org/abs/2609.24298). It keeps the standard low-rank projection plus scalar quantization, but chooses the rank and the bit width **separately for every attention head**, then moves budget between heads so each sits at the same marginal error.

Why does per-head allocation matter?

Heads differ a lot: in LLaMA-3.1-8B at a uniform 2 bits per dimension, the paper measures a **100× spread** in per-head error across 256 KV heads. One global rank and bit width wastes bits on concentrated heads and starves diffuse ones, and at very low bit rates that mismatch dominates the accuracy loss.

How is it different from KVarN or TurboQuant?

KVarN and TurboQuant improve the quantizer itself, mainly by rotating the cache so outliers spread out, and use one bit width for every head. KV-COBRA uses the same rotate-and-quantize kernel and changes only **how the bit budget is split** across heads and between rank and precision.

Does KV-COBRA slow down inference?

The paper reports **no per-token overhead**: the allocation is computed once at calibration, in under 200 ms, and the rotation is fused into the projection matrix. It reports kernel-level attention speedups, but not end-to-end serving latency.

LLM· 2026-09-23

KV-COBRA splits the KV-cache budget per head — Rank vs bit-width allocation — What does it mean?

TL;DR

What is it: KV-COBRA is a KV-cache compression method from a September 2026 paper that picks, for each attention head separately, how many directions to keep (the rank) and how many bits to store each one with (the bit width).
Why it’s needed: The KV cache is the memory wall of long-context inference — every decode step reads all of it — so squeezing it below 1 bit per value decides how much context and batch fits on a GPU, and at those rates where the bits go matters more than how they are quantized.
vs previous: Earlier compressors such as TurboQuant, SVDq and KQ-SVD give every head the same bit width and never choose rank and bits together, which wastes bits on heads that need few and starves heads that need many — KV-COBRA keeps the same kernel and only re-allocates the budget.

Jargon

KV cache: The stored keys and values of every earlier token, re-read on each decode step. At long context it can outgrow the model weights.
bpd (bits per dimension): The average number of bits spent per stored number in the cache. FP16 is 16 bpd; KV-COBRA is tested from 0.5 to 4 bpd.
Rank / low-rank projection: Keeping only the top-r directions of a head's key vectors (found with an SVD) and dropping the rest. Fewer directions means fewer numbers to store.
Scalar quantization: Rounding each stored number to one of 2^b levels, where b is the bit width. Each extra bit cuts the rounding error by about 4×.
Hadamard rotation: A fixed ±1 orthogonal transform that mixes channels so their variances even out, letting one quantizer setting fit every channel. Full explainer
Attention-KL: How far the compressed cache pushes the attention weights away from the uncompressed ones. KV-COBRA uses it to rank directions by how much the query actually uses them.
Calibration: A one-time pass over sample text, before serving, that measures each head's statistics and freezes the allocation. The paper reports it takes under 200 ms.

The news. On September 21, 2026, researchers at POSTECH posted KV-COBRA (Co-Optimized Bit-Rank Allocation) to arXiv. The paper argues that at extreme compression rates the limit is not the compression scheme but how its budget is allocated across attention heads. It keeps the standard low-rank-plus-quantize kernel and adds a two-level allocator, tested on LLaMA-3.1-8B, Mistral-7B-v0.3 and Qwen2.5-7B-Instruct from 0.5 to 4 bits per dimension. Read the paper →

Picture one phone whose storage is shared by several photo albums. For each album you make two choices: how many photos to keep, and how sharp to save each one. A holiday album with three great moments should keep those three at full resolution. A stock-inventory album where every shelf matters should keep all of its photos, even if each one is a bit blurry. Give both albums the same rule — say, 32 photos at medium resolution — and the first wastes space on blurry filler while the second loses most of its shelves. The storage total is fixed; the only question is how it is split.

In a transformer, each album is one attention head and the photos are the directions of its key vectors. A head is concentrated when a handful of directions carry almost all of its variance, and diffuse when variance spreads over many. The paper measures how different heads really are: in LLaMA-3.1-8B at a uniform 2 bpd, the per-head error spans 100× across its 256 KV heads. One shared setting cannot be right for both ends of that range, yet current rotate-and-quantize compressors give every head the same bit width, and none choose rank and bits together.

KV-COBRA splits the decision into two levels. Inside one head (step C1), it trades two kinds of loss against each other: dropping a direction loses that direction's variance outright, while quantizing the kept directions adds rounding noise that falls about 4× per extra bit. C1 picks the rank and bit width where two more directions would buy back exactly as much as they cost in thinner bits. Across heads (step C2), it moves budget until every head sits at the same marginal error — the classic water-filling rule from information theory: a bit goes wherever it removes the most error. The paper reports C2 converges within five rounds, and at 2 bpd the chosen settings range from roughly rank 16 to 100 and 2 to 5 bits across heads.

Two supporting pieces make this work with a plain kernel. A Hadamard rotation, fused into the same projection matrix at no extra latency, flattens the steeply uneven variances of the kept directions so one uniform b-bit quantizer fits all of them — the same outlier-spreading trick other low-bit caches use. And instead of ranking directions by variance alone, the KL variant reorders them by how much the attention output depends on them, so a low-variance direction the query attends to strongly is kept rather than cut. Everything is decided once at calibration; at inference each key is projected, rotated and quantized with frozen per-head settings, so there is no per-token overhead.

Where the bits actually go

Hold three things fixed (illustrative numbers, not from the paper): a head dimension of 128, a budget of 1 bpd, and two heads — so each head gets 128 bits per token and the pair gets 256. Head A is concentrated: its top 16 directions hold 99% of its variance. Head B is diffuse: its variance is spread evenly over all 128 directions. Uniform setting, rank 32 at 4 bits for both (32 × 4 = 128 bits each): head A loses about 1%, but head B keeps only 32 of its 128 directions and throws away 75% of its variance before any rounding happens. Per-head setting: head A needs only rank 16 at 4 bits (64 bits) to keep the same 99%. The 64 bits it frees move to head B, which now has 192 bits and can take rank 96 at 2 bits. Head B's lost variance drops from 75% to 25%, at the price of the extra rounding noise that 2-bit storage adds — the trade C1 checks explicitly. Same 256 bits, same kernel; only the split changed.

Method	Basis the cache is stored in	Rank per head	Bits per head
KIVI, KVQuant, GEAR	original channels	no rank reduction	fixed hyperparameter (paper, Table 1)
TurboQuant	random rotation	no rank reduction	fixed hyperparameter (paper, Table 1)
SVDq	per-layer SVD	fixed hyperparameter	fixed hyperparameter (paper, Table 1)
KQ-SVD	query-aware SVD	adaptive, set by an error threshold	fixed per head, optimized per layer (paper, Table 1)
KV-COBRA	per-head SVD + Hadamard	optimized per head	optimized per head and per layer (paper, Table 1)

The gains are concentrated at the low end. The paper states that at 2 bpd and above the choice of allocator barely matters; below that, KV-COBRA degrades most gracefully, and at 1 bpd its KL variant nearly doubles the F1 score of the integer-only baselines on LongBench, a long-document question-answering benchmark. The scope has limits worth stating: the main results compress keys only, taken before rotary position encoding, because the value spectrum is flatter and gains little from redistribution (the paper's joint key-plus-value sweep is in its Tables 4 and 5); the allocation is calibrated once on Wikitext-2 samples; and the paper reports kernel-level attention speed, not end-to-end serving latency. It also composes with, rather than replaces, token eviction and grouped-query attention, which shrink the cache along other axes.

Goes deeper in: LLM Internals → Quantization → Modern Methods

Related explainers

KVarN — Hadamard-rotated 2-bit KV cache — the rotation step KV-COBRA fuses into its projection
Tangram — per-head KV cache budgets — per-head budgets on the token axis, deciding which tokens each head keeps
SigmaScale — learned scaling for truncated SVD — low-rank truncation as an alternative to fewer bits, applied to weights
D-Quant — entropy-coded KV cache — a different lever: change the quantizer itself instead of its budget

Continue in trackQuantization: Modern Methods

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based