DepthGraft is a Transformer design from a November 2026 arXiv paper. It splits the layers into shallow, middle and deep thirds, and each deep layer **rebuilds** its keys and values from one shallow and one middle layer instead of storing its own.

Why does DepthGraft shrink the KV cache by about one third?

Only the shallow and middle thirds write KV to the cache. The deep third stores nothing and recomputes its keys and values from those stored tensors with a cheap element-wise blend, so roughly two thirds of the layers keep a cache.

How is DepthGraft different from other cross-layer KV sharing?

CLA and MLKV reuse a neighbouring layer, and YOCO and FusedKV feed all deep layers from one or two anchors. DepthGraft pairs each deep layer with a shallow and a middle layer at the same relative position, which the paper says spreads the training signal more evenly.

Does DepthGraft slow down decoding?

The paper's measurements show almost no change: on the 16B GQA model, time per output token at 128K context went from 7.79 ms to 7.84 ms, while time to first token fell from 2,655.90 ms to 1,891.80 ms.

LLM· 2026-11-04

DepthGraft rebuilds deep-layer KV from shallow and middle layers — Cross-layer KV reconstruction — What does it mean?

Jargon

KV cache: The keys and values every layer saves for each earlier token, so decoding does not recompute them. It grows with context length. KV Cache module
GQA: Grouped-query attention: several query heads share one set of key and value heads inside a layer, which shrinks the cache. How GQA shrinks it
MLA: Multi-head latent attention: keys and values are stored as a compressed latent vector and expanded when needed. It is a second attention variant the paper tests.
Pre-LN: A Transformer layout where normalization comes before each sub-layer and the residual stream carries the running sum. Pre-norm vs post-norm
Residual stream: The single running vector each layer adds its output to. Early-layer features can get diluted in it as the stack grows deeper. Residual connections
Prefill and TTFT: Prefill processes the whole prompt before the first output token; time to first token (TTFT) is how long that takes. Prefill vs decode
RoPE: Rotary position embedding: it encodes token position by rotating pairs of channels in the query and key vectors.
Gradient path: The route backpropagation takes to send a training signal from a loss back to earlier parameters. Reusing a layer's KV adds one such route.

The news. On November 2, 2026, researchers from Shanghai Jiao Tong University, Xiaohongshu's Dots Studio and the Shanghai Innovation Institute posted DepthGraft on arXiv. They report that a hierarchical KV-sharing design cuts the KV cache by roughly one third across dense and mixture-of-experts models from 0.5B to 16B parameters, with a 7.0-point MMLU-Pro gain on the 16B MoE. Read the paper →

Picture a tower where every floor keeps a filing cabinet of everything it has read so far. That is the KV cache: one cabinet per layer, and every cabinet grows with each token of context. DepthGraft splits the tower into three equal blocks — shallow, middle and deep — and removes the cabinets from the top block. A deep floor still needs something to look things up in, so it builds a temporary cabinet by blending two from below: the shallow-block floor and the middle-block floor at the same relative position in their blocks.

The blend uses a learned mixing dial for every channel, so a deep floor can lean on the shallow cabinet for some columns and the middle cabinet for others. The paper's reasoning is that shallow memory keeps lower-level, token-local patterns while middle-layer memory carries more abstract states, so each deep layer gets two levels to choose from. For keys the dials are tied across rotary-position pairs, so applying RoPE before or after the blend gives the same result and the standard cache logic still works. Because the deep third stores nothing, the cache shrinks by about one third. GQA already shares keys and values across heads inside one layer; this shares across layers, and the paper tests it on top of both GQA and MLA.

Prefill gets cheaper too. Normally every prompt token passes through every layer, because each layer has to write its cabinet. Here, once the shallow and middle blocks have processed the prompt, every cabinet that decoding will ever read already exists. The deep block then runs only the last prompt token, reading those stored cabinets, to produce the first output token. Skipping full-prompt work in a third of the layers caps the saving near one third, and the measured time to first token approaches that as context grows: on the 16B GQA model it fell from 2,655.90 ms to 1,891.80 ms at 128K context (28.8% lower) but only 17.4% lower at 8K, while time per output token moved from 7.79 ms to 7.84 ms.

The more surprising claim is about training, not serving. In a Pre-LN Transformer the residual stream is one running sum, and the paper argues that features from early layers get diluted in it, so extra depth is used poorly. When a deep layer reads a shallow layer's keys and values, backpropagation also flows through that read. The shallow layer's K and V projections then receive a training signal shaped by a distant layer's queries. The paper's analysis, supported by its training-dynamics measurements, says this sharpens the key directions used for retrieval and enriches the value content. Whether that happens depends on which layers share, and the paper's diagnosis of the earlier designs is in the table.

Design	Who reuses what	The paper's diagnosis
Adjacent groups (CLA, MLKV)	Groups of 2–3 consecutive layers reuse the first layer's KV	Neighbouring layers see nearly the same hidden states, so the reuse adds little training pressure
Global anchors (YOCO, FusedKV)	One or two early layers feed every deep layer	Gradient from all deep layers piles onto the anchors, an overloaded compromise
DepthGraft	Each deep layer blends one shallow and one middle layer at the same relative position	Spreads gradient over many sources and keeps all KV ready before the deep stage

Here is the cache arithmetic (illustrative numbers, not from the paper). Take a 36-layer model with 8 KV heads of dimension 128, stored in bf16, serving a 128K-token prompt. Each layer stores 2 (K and V) × 8 × 128 × 2 bytes = 4,096 bytes per token, so one layer holds 4,096 × 131,072 = 512 MiB, and 36 layers hold 18 GiB per sequence. With DepthGraft the 12 deep layers store nothing, leaving 24 layers: 12 GiB, a saving of 6 GiB per sequence. A GPU with a fixed cache budget then fits 18 / 12 = 1.5× as many 128K sequences. In prefill, 12 of 36 layers skip the full prompt, so if every layer costs about the same the ceiling is one third; the 28.8% measured at 128K on the 16B model sits just under it.

The evidence is one paper, with models trained from scratch. Its ablation on the 16B MoE (which uses its own average over the benchmark suite) isolates the gradient path: copying the shallow layer's KV but blocking gradients scored 42.86 against a 42.38 baseline, switching the gradients on lifted the same copy to 45.30, and the full shallow-plus-middle blend reached 47.56. That ordering is what supports the claim that the training signal, not only the memory saving, drives the gain. The gains are not uniform: MMLU-Pro fell from 7.43 to 6.29 on the 3B MoE and from 20.43 to 19.71 on the 9B GQA MoE, while it rose from 23.86 to 30.86 (+7.0 points) on the 16B MoE. The largest model tested has 16B total parameters, so behaviour at frontier scale is not shown.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

BeNice shares one KV cache across a looped model's recurrences — sharing KV across repeated passes of the same layers, not across different layers.
Hy4 IndexCache cross-layer index reuse — reuses which tokens attention selects across layers, not the KV tensors themselves.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based