LLM·

Storing the Gemma 4 K=V cache once — Overlapping strided K and V views — What does it mean?

The news. On November 17, 2026, a single-author paper from San José State University (arXiv 2611.23035) took a saving that the Gemma 4 report already states, up to 37.5% less global KV cache, and made it real inside vLLM without touching the attention kernel. In a prototype with the 16-bit Gemma 4 31B model, the steady decode KV memory per request fell by 1.34× to 1.40× at 32K to 64K tokens, outputs were bit-identical to an emulation of the same arithmetic, and 2 of 432 final answers differed from the stock engine, which the author reports is within its run-to-run noise. Read the paper →

Picture the shelf. A stock paged engine keeps two shelves per token and KV head: a 512-book key shelf and a 512-book value shelf, because its attention kernel expects separate key and value caches. But in Gemma 4's global layers, which the model configuration marks attention_k_eq_v, there is no value projection at all. The value is the RMS-normalized key projection, and the key is that same value multiplied by a fixed per-layer weight, with RoPE applied to 128 of its 512 dimensions. So on 384 of the 512 dimensions, the key is the value times a fixed weight: the key shelf holds the value's books, each one merely tinted, and a compact layout can reuse the value bytes.

The fix has two moves. First, the fixed weight moves out of the cache and into the query: an attention score is q · k, and on the unrotated dimensions q · (w ⊙ v) equals (q ⊙ w) · v, where ⊙ means element-by-element multiplication. Multiply the query by the key-norm weight once per step, and the key cache can hold the plain value numbers on those 384 dimensions. That is the reader's tinted glasses: tint the reader, not every book. Second, one row of 640 values is laid out as rotated key (128), shared value (384), rest of the value (128). The K view starts at slot 0 and is 512 wide; the V view starts at slot 128 and is 512 wide. Both are strided views with the same block, token and head strides, so a kernel that takes key and value caches with strides reads them unchanged. The output comes back with its dimensions in the V view's order, and a fixed permutation puts them back.

Why not go all the way to 512? Because the rotated key is a function of the value and the token position, the whole key could be rebuilt from the value alone, for a 50% saving. But the rotation angle is different for every cached token, so it cannot be folded into the query once; a 512-value row needs a new kernel that rotates while it reads. The paper keeps 128 extra values per row to buy zero kernel changes, which is the trade that lets the method drop into vLLM's existing block allocator.

K and V
Two vectors stored per token (Key + Value)
× 2
Layers
Each layer has its own cache (like 32 filing cabinets)
× 32
Heads
Each attention head stores its own K/V pair
× 32
Head size
Each K or V vector has 128 numbers (d_head)
× 128
Bytes per number
FP16 = 2 bytes per number (half precision)
× 2
Per token (Llama-2 7B):2 × 32 × 32 × 128 × 2 = 524,288 bytes ≈ 512 KB

The diagram above is the usual KV formula, and its leading factor of 2 (one key, one value) is the term this paper shrinks: for these layers, 2 × 512 = 1,024 values becomes 640, a factor of 1.25 per head instead of 2. It applies only to the global layers; the sliding-window layers keep their normal key and value caches.

Layout per global KV head and tokenValues storedKernel changeSource
Stock engine: full key + full value1,024nonepaper §1
Compact row, two overlapping views640 (37.5% fewer)none; query fold + output permutationpaper §3
Value only, rebuild the rotated key512 (50% fewer)new kernel that applies RoPE per cached tokenpaper §3
8-bit (FP8) KV cachesame count, half the bytes, lossynot shown to stack with the compact row at the default scalepaper §1, App. D

Worked example (from the paper's model dimensions, 32K-token request, 16-bit). Hold the model fixed at Gemma 4 31B and the context at 32,768 tokens. A global layer stores 4 KV heads × 1,024 values × 2 bytes = 8,192 bytes per token; across 10 global layers that is about 2.5 GiB. The 50 sliding layers keep only the last 1,024 tokens, at 16 heads × 256 dims × 2 (key and value) × 2 bytes = 16,384 bytes per token, about 0.78 GiB, and they do not change. The compact row needs 4 × 640 × 2 = 5,120 bytes, but the page must match the sliding page size, so the block grows to 48 tokens and is padded, giving 5,461 bytes per token and about 1.67 GiB. The totals, ≈3.28 GiB → ≈2.45 GiB, match the measured 3.29 → 2.46 GiB steady decode footprint, a factor of 1.34×. In a 15 GiB pool that is the difference between 3 and 4 concurrent 32K requests, and decode rose from 56.0 to 69.7 tokens per second (1.24×).

The gain is capacity, not a faster step. The author's byte count shows that model weights are about 85% of the bytes a decode step reads at these batch sizes, so a smaller cache barely speeds up one step; the prototype step in eager mode (kernels launched one by one from Python, without CUDA graphs) was even 4% to 9% slower at equal batch. The gain turns into throughput only where one more request fits: at 64K tokens both layouts held 2 requests and the compact build was 4% slower. On a two-A100 prefill/decode pair with NIXL KV transfer and a full decode pool, the compact decode engine ran 1.38× the stock decode tokens per step for 16K-token prompts and 1.24× for 8K-token prompts. The general lesson is older than Gemma 4: when the model's algebra makes two cached tensors overlap, store the overlap once and let views, not copies, give each consumer the shape it expects.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based