FlashOffloading offloads all KV cache — rolling prefill buffers plus a hot decode cache — What does it mean?

The news. On November 5, 2026, a Huawei team posted FlashOffloading (arXiv 2611.07542, an MLSys paper). It stores the full KV cache in host memory in both prefill and decode. On GLM-5.2 with MTP3 on an Ascend 910C cluster with separate prefill and decode nodes, it reports up to 1.38× prefill throughput and up to 1.91× decode throughput under a 30 ms TPOT constraint, and it makes 512k and 1M-token requests servable. Read the paper →

Picture a line cook. The prep counter is tiny, but every dish needs ingredients from a huge walk-in pantry. In a model server the counter is the accelerator's HBM and the pantry is the host's DRAM. The KV cache stores a key and a value for every past token in every layer, so it grows with context length and with the number of requests served at once. The paper cites 10 to 100 GB for one 1M-token sequence, so long requests fill the counter and limit how many cooks can work together.

Moving the cache to the pantry frees the counter, but every trip costs time, and so does keeping track of what is where. The paper's claim is that the trips can be hidden if each phase gets its own counter layout. Prefill reads every token once, layer by layer. Decode reads only a small chosen set of tokens per step. The pantry is the single shared record for both, and in the disaggregated setup the hand-off from prefill to decode writes straight into the decode node's host memory, not onto the counter.

In prefill, the counter holds only four layer-sized trays. A transformer reads one layer's KV at a time, so layers 1 to 77 do not need counter space while layer 0 is being computed. Tray B0 stays bound to the first layer for the whole pass. Trays B1 to B3 rotate: one is being computed on, one is loading the next layer from the pantry, and one is writing a finished layer back. Keeping layer 0 resident lets compute start at once, and that compute time gives the loader its first window. A map named onload_layer_map sets how many layers ahead to load, so a slow link can start earlier.

The exposed delay for a layer is about the transfer time minus the compute time available while it moves, never below zero. So the transfer is free whenever the compute window is at least as long as the copy. Model depth only makes the pantry larger, not the number of trays. For a 78-layer model the paper estimates the four trays hold about 4/78, or 5.1%, of the K/V data that a fully resident design needs.

In decode, the cook keeps a small hot shelf and fetches only the misses. Each step the sparse indexer names the tokens it wants. Tokens already on the shelf need no host fetch. The others are loaded from the pantry and take the places of evicted tokens, so the shelf size stays fixed. With MTP3, the four per-position miss lists are merged and deduplicated before loading.

Two details keep the bookkeeping off the critical path. First, a reverse index called cache_slots maps each token to its shelf slot, or to -1 if it is absent, so a hit-or-miss check is one lookup and not a scan of the shelf. That separates management time from shelf size, and the paper ties growing the shelf from 1/8 to 1/4 of the context to nearly 4× larger batches. Second, the check is folded into the top-k itself: each candidate's token index and slot value are packed into one int32 before selection, so the winners already say hit or miss (a fused kernel the paper calls LIM). A further fused kernel, CopySFA, fetches the missed tokens from host memory inside the sparse-attention kernel while the hit tiles are already computing. The paper reports about 144 microseconds of cache management per indexer layer (roughly 3 ms across 21 layers per step) and a 90% to 98% overlap of miss loads at batch size 16 or more.

Here is how the pieces compose, using the paper's 128k-token row of Table 1 with 8,192 output tokens. With prefill offloading only, 2 requests fit per NPU at 16.6 ms per output token; with decode offloading added, 6 fit at 26.0 ms. Per NPU, throughput is roughly requests divided by time per token: 2 / 16.6 ms is about 120 tokens per second, and 6 / 26.0 ms is about 231. Tripling the batch while each stream runs at 16.6 / 26.0, about 64%, of its old speed gives 3 × 0.64, about 1.9×, which matches the reported system-wide output rise from 1,928 to 3,692 tokens per second (1.91×). The price is visible too: each token is 56.6% slower.

Read the result with its limits. It is one model (GLM-5.2), one platform (Ascend 910C) and the authors' own measurements. Decode offloading relies on the model's native sparse indexer, because the system can only fetch what the model selects; the layer-by-layer prefill offloading also applies to dense attention. Prefill gains grow with length: input throughput rises 8.6%, 5.0% and 37.8% at 64k, 128k and 256k, and the 1.38× figure is the 256k point. At the same chunk size, prefill offloading actually raises time to first token, because the reused prefix is reloaded on every forward pass. The win comes from the freed HBM, which lets the chunk grow from 16,384 to 131,072 tokens. Per-token latency also worsens by 42% to 79% in the decode runs. The paper states that offloading changes only where data lives, not the computed results.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

• SparPR: copy versus recompute for offloaded KV blocks — another offloading answer: choose per block between copying over PCIe and recomputing on a consumer GPU.