D-Quant drifts variable-length KV codes into fixed-size token streams — Entropy-coded KV cache — What does it mean?

The news. On 17 September 2026, a paper titled D-Quant: Driftable Entropy Coding for KV Cache Quantization was posted to arXiv. It argues that fixed-width KV-cache quantization is the wrong shape for the data: a b-bit code offers only 2^b levels, and that count shrinks exponentially as the bit width drops, so low-bit fixed-width caches lose information quickly. The authors report that after rotation and normalization — a standard pre-step that spreads outlier channels out and puts values on a common scale — KV values approximately follow a normal distribution — most values clustered near the centre, a small fraction out in the tails — yet fixed-width coding spends the same bits on both. Their answer is entropy coding plus a drift mechanism that converts each token's representation into a fixed-size bitstream, restoring regular memory access and parallel dequantization. Read the paper →

Picture the car park. Every vehicle has to be stored, and you have to paint the bays before you know what will arrive. The safe choice is to paint every bay truck-sized: nothing ever fails to fit, and bay 40 is always forty bay-widths from the entrance. That is fixed-width quantization. It is also why the row runs so long — a fixed-width code spends the same number of bits on a value the model stores constantly and one it almost never stores.

The waste is not hypothetical, and it follows from the shape of the data. D-Quant reports that once KV values have been rotated and normalized (the same spread-and-rescale pre-step), they land in roughly a normal distribution: a dense cluster near the centre, thin tails at the edges. Most of what arrives is a compact car. Painting for trucks means most bays sit mostly empty. (In the car park, a vehicle's length stands for the length of its codeword, which is set by how often that value occurs — not by how large the number itself is.)

The second problem is that the bays cannot simply be made narrower. With b bits you get 2^b levels and no more, and each bit you remove halves the number of levels available, so the grid coarsens fast. At the low bit widths that make a KV cache genuinely cheap, there are few enough levels that distinct values start rounding onto the same one.

Entropy coding attacks the waste directly rather than the level count. Give the values near the centre of the distribution short codewords and the rare tail values long ones, and the average cost per value falls below what a fixed-width code over the same set of levels can reach, because the code now matches how often each value actually appears — the paper describes this as substantially reducing the average number of bits needed. In the car park, every vehicle takes exactly the length it needs and the row gets shorter.

And then the bay lines are gone. Entropy coding fixes the waste and breaks the layout in the same move — once codewords have different lengths, a token's record has no predictable size, so token 40 no longer starts at a predictable offset. Without a separate index recording where each record begins, finding it means walking everything before it.

This matters because of where the data is read. The KV cache is consumed inside attention kernels, and the paper's framing is that efficient dequantization and computation there rely on regular memory layouts and fixed-stride access: thousands of threads each work out an address arithmetically and load from it in the same instant. Variable-length records break that pattern — an index can still record where each one starts, but the single multiply that located token 40 becomes a lookup and an irregular read. That is the tension D-Quant is built around: store fewer bits without giving up an efficient way to read and decode them back out of HBM, the high-bandwidth memory a GPU keeps its working data in.

Work it through with numbers. The paper reports no figures in its abstract, so these are (illustrative) — they show how the parts compose, not what D-Quant measured. Hold three things fixed: a 4-bit budget, one 128-dimensional key vector for a single token at a single layer and head, and a distribution where 60% of values fall in the 2 levels nearest the centre, 20% in the next 2, and the remaining 20% spread across the outer 12.

Fixed-width first. Four bits gives 2⁴ = 16 levels, and every one of the 128 values costs 4 bits: 128 × 4 = 512 bits for that token, always. Now assign codewords by frequency — 2 bits for the 2 centre levels, 3 bits for the next 2, 6 bits for the outer 12. The average cost is 0.60 × 2 + 0.20 × 3 + 0.20 × 6 = 3.0 bits per value, so the same token averages 128 × 3.0 = 384 bits — a 25% saving (illustrative).

Now look at what that average hides. A token made entirely of centre values would need 128 × 2 = 256 bits; one full of tail values would need 128 × 6 = 768 bits. The saving is real, and the addresses are lost in the same step — token records now range from 256 to 768 bits, so no fixed stride can locate where token 40 begins. D-Quant's drift mechanism repacks each token's variable-length codewords into a fixed-size bitstream, which puts token 40 back at a computable offset and lets dequantization run in parallel inside the kernel. How drift handles a token whose codewords overrun that fixed size is not documented in the abstract.

Goes deeper in: LLM Internals → Quantization → How Numbers Shrink

Related explainers

• KVarN: Hadamard rotation for 2-bit KV caches — the other way to attack the same problem: keep fixed-width coding and reshape the distribution so it fits.
• InfoKV: entropy-aware KV-cache compression — decides which tokens to keep rather than how many bits each value costs.
• ThriftAttention: importance-aware FP4 — varies precision per attention block instead of per value.