Adaptive DHiSparse — Load-adaptive hierarchical sparse KV cache — What does it mean?

The news. On October 9, 2026, Naji Alazhar and Wassel Alazhar of Umans AI posted Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy to arXiv. Built in SGLang on top of HiSparse, it adds GPU residency that follows load and support for the draft-model speculative decoders DSpark and DFlash2. On B300 GPUs serving GLM-5.3 at 100K-token context with 128 concurrent requests, the paper reports 793 output tokens/s per decode GPU, 1.45× HiSparse. Read the paper →

Picture a workshop where each worker needs a few parts at every step of a job, and a job sheet names exactly which ones. The bench is fast to reach but small; the warehouse out back holds every part, but each trip takes time. A sparse-attention model is that worker: at each decode step its indexer picks only the top-k history rows the query will read, so the rest of the history does not have to sit on the GPU. In the model, the bench is the GPU's high-bandwidth memory and the warehouse is host DRAM, reached over PCIe.

Conventional serving keeps every request's full history on the bench anyway. That is the memory cost of the KV cache: it grows with context length times the number of requests, so at long context the bench fills and the server stops admitting requests while the GPU's compute is still underused. Hierarchical KV caching moves the full history to host memory and keeps on the GPU only what the next steps need.

HiSparse, the earlier design, gives each worker one fixed toolbox: a hot cache of recently useful rows per request, replaced least-recently-used first, and one fused kernel that finds hits, picks victims and fetches misses from host. The toolbox has the same size whether the shop is empty or packed. A small one admits more workers but makes needless warehouse trips while the bench is mostly empty; a large one saves trips but blocks new workers when the shop is busy.

Adaptive DHiSparse splits the bench into a part that is never lent and a part that follows the load. Each request keeps its small fixed hot cache and its newest rows (the tail). On top of that, while GPU memory is spare, the runtime leaves the request's earlier history on the GPU as optional history, held in whole pages of the KV pool, much like the blocks of PagedAttention. When a new request needs room, the runtime picks whole requests, makes sure their rows have a copy in host memory, and takes their optional pages back; when load falls, a background scheduler refills pages a few at a time. Because every row is always either ready on the GPU or backed up in host memory, attention still reads exactly the rows the model selected; only their location changes.

The second addition is speculative decoding. A draft proposes several tokens and the target model checks them in one pass, which means several queries in one step, and those queries often select the same history rows. Adaptive DHiSparse fetches a row that several verification queries share only once, keeps the drafter's own KV in a ring bounded by the drafter's visible window, and reuses verification staging buffers across layer groups. The paper reports that this adds 0.060 GiB of required GPU state per GLM-5.3 session, and that DSpark and DFlash2 keep their proposal and acceptance rules unchanged.

Walk the paper's main result, holding three things fixed: the model (GLM-5.3), the context (100K tokens) and the load (128 concurrent requests) on B300 GPUs. HiSparse delivers 548 tokens/s per decode GPU. Step 1 adds load-adaptive residency and launches the row-copy kernel over more GPU thread blocks, without speculation: 622 tokens/s, or 1.14×. Step 2 adds DFlash2: each round verifies 4 queries and emits 2.88 tokens in 2.2× the time of a plain decode step, so each session runs about 2.88 ÷ 2.2 ≈ 1.3× faster, and output rises to 793 tokens/s, or 1.27×. The two steps multiply: 1.14 × 1.27 ≈ 1.45× HiSparse. Shared fetches matter for the same reason. In the paper's kernel test, the time of the resolver (the kernel that finds each selected row and copies in the missing ones) grows with the bytes copied from host, at about 49.4 GB/s: when 8 verification queries over a 32-request call pick disjoint rows, the call moves 302 MB in 6.23 ms, and 8 queries that pick the same rows cost about 1/7.9 of that.

The paper also states where the design stops paying. At 256 concurrent requests on GLM-5.3, speculation no longer adds output: Adaptive DHiSparse with DFlash2 and the same system without a drafter both land at about 1.13× HiSparse, mean TTFT is higher with speculation (64.6 s against 34.4 s), and fewer requests decode at once (about 138 against 202 on average). Optional residency has its own cost when load jumps: pages must be backed up and returned before new requests can use them, which is the minute spent packing parts back into the warehouse before new workers start. The rule the design follows is to use spare GPU memory while it is spare, and to give it back in an order that never leaves a row without a copy.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

• Partial loads from a lower cache tier — another way a serving engine uses a slower memory tier under the GPU, here for cached prefixes rather than sparse selections
• IndexCache cross-layer index reuse — the indexer side of sparse attention, which produces the top-k list this cache serves
• Query-cluster KV residency — a different answer to which KV rows to keep close: predict revisits instead of following load