What is a query-dependent execution view?

It is the slice of stored KV cache that a system materializes into the model's context window for one decoding step, chosen by what that step's query is likely to attend to. The stored history stays fixed; the view is rebuilt for every query, so the same 1M-token workspace produces a different in-window slice depending on what is being asked.

How is this different from retrieval-augmented generation?

RAG retrieves text and re-prefills it, so it pays attention's setup cost again on every step and matches on text similarity. KVMem retrieves already-computed KV blocks and scores them in attention space, which is closer to asking which stored keys this query would actually attend to.

Why does it need NVMe at all?

Because the full history does not fit in GPU memory. At the reported setup a 1M-token workspace is far larger than the 24 GB on the GPU, so blocks are racked by access speed — GPU, host RAM, NVMe — and pulled forward only when the index selects them.

How does this relate to PagedAttention?

PagedAttention introduced fixed-size KV blocks and a block table so one request's cache need not be contiguous in GPU memory. KVMem takes the same paging idea past the edge of the GPU, across host RAM and NVMe, and adds an index that decides which blocks are worth paging in for this particular query.

What is the failure mode?

A cache miss with no error message. If the index does not select a block the step needed, the model answers from an incomplete desk and sounds just as confident. The block is still stored, so a later step can still reach it, but the quality of the whole system rests on how often the selected view actually contains what the step required.

LLM· 2026-09-08

Virtualize million-token agent workspaces across GPU, RAM, and NVMe — Query-dependent execution view over paged KV memory — What does it mean?

TL;DR

What is it: The KVMem paper describes a query-dependent execution view over paged KV memory: an agent's overflowed history is kept as KV blocks spread across GPU memory, host RAM and NVMe, and only the blocks a given decoding step needs are assembled into the model's native context window.
Why it’s needed: A long-running agent overflows its window in hours, and the usual repair is to throw history away — summarize the old turns, or re-prefill retrieved text every step — which discards exactly the detail a later step turns out to need; KVMem keeps the state and changes what the model looks at instead, which is what lets a 24 GB laptop GPU drive a 1M-token workspace at about 50 tokens/s.
vs previous: Against compaction — summarizing old turns down to a paragraph — and against retrieval with re-prefill, which re-pays prefill compute on the same text every step, KVMem indexes the stored KV in attention space and pays for a block transfer plus an index lookup, lifting task success on DeepSWE — a software-engineering agent benchmark — from 43.8% to 48.4% over compaction-only management.

Jargon

KV cache: The keys and values every past token contributed to attention, stored so the model does not recompute them each step. It is the thing that grows with conversation length. Walk through it
KV block: A fixed-size chunk of that cache — say 16 tokens' worth — treated as one allocatable unit, the same way an operating system treats a memory page. Blocks are what KVMem moves between tiers.
Prefill: The first pass over a prompt, which computes KV for every token at once. It is compute-heavy, so re-prefilling the same old text on every step is the cost that retrieval-based memory keeps paying.
Compaction: Summarizing or truncating older turns to make room in the window. Cheap and lossy: the discarded detail cannot be recovered later, which is the failure mode KVMem is aimed at.
Attention-space index: A lightweight, model-native lookup that scores stored KV blocks by how much this step's query would attend to them — rather than by text similarity, which is what ordinary retrieval uses.
Execution view: The subset of blocks actually materialized into the model's context for one step. It is rebuilt per query, so the same stored history yields a different view depending on what is being asked.
NVFP4: A 4-bit NVIDIA floating-point format. Running the model at NVFP4 is what frees enough of the 24 GB GPU for a workspace this large.
MTP: Multi-Token Prediction — the model proposes several tokens per forward pass instead of one. It is part of the reported setup; the source does not separate out how much of the throughput it accounts for.

The news. On September 4, 2026, a paper introduced KVMem, a system that treats an agent's overflowed history as pageable KV state rather than text to be summarized. Running Qwen3.6/3.8-27B at NVFP4 with MTP on a single 24 GB RTX 5090 Laptop GPU, it reports a 1M-token addressable workspace — four times the model's native 256K window — at about 50 tokens/s, and a move from 43.8% to 48.4% task success on DeepSWE, a software-engineering agent benchmark, against compaction-only context management. Read the paper →

Picture the desk. It is small, and it is the only surface the model can read from — everything the model knows at this instant is what fits on the desk, which for this setup is 256K tokens. Behind it stands a warehouse holding every file the agent has ever touched. The old answer to a full desk was to shred: take the last four hours of work, boil it down to a paragraph, and put the paragraph on the desk. That clears space, and it is also the moment the run quietly becomes unrecoverable — the detail you shredded is the detail step ninety needed.

KVMem refuses the shredder. Nothing leaves the warehouse; what changes is which boxes get carried to the desk. The files stay in the form the model can actually read — KV cache blocks, not text — racked across three shelves by how fast you can reach them: GPU memory in arm's reach, host RAM a few steps away, NVMe at the back. That is virtual memory, and it is the same trick vLLM plays inside a single request: the model addresses one contiguous history, while the bytes are scattered across whatever blocks happen to be free.

The part that is new is the clerk. Before each step a lightweight, model-native index reads the incoming query and picks the blocks it will need, scoring them in attention space rather than by text similarity. That distinction is the whole design: ordinary retrieval asks "which passage looks like this question?", while the index asks "which stored keys would this query actually attend to?" How the index answers that cheaply — without doing the attention it is trying to avoid — is the part the paper summary does not spell out, and the part worth watching. The stack the clerk hands over is the execution view, chosen fresh for each query. Ask the same agent two different questions about the same 1M-token history and two different desks come back.

OS Virtual Memory — 4 programs sharing physical RAM

What each program sees (virtual)

Chrome

VS Code

Slack

Spotify

Each app sees pages 0, 1, 2 — contiguous

Physical RAM (actual)

Pages scattered — C0 at slot 2, C1 at slot 7...

Page table translates:Chrome v0 → p2, VS Code v0 → p0, Slack v0 → p3

Where does the memory actually go? Hold the workspace fixed at 1M tokens and the window fixed at 256K. At any single decoding step the model can see roughly a quarter of its own history — 256K in, about 744K out. Compaction answers that gap by deleting the 744K down to a summary; KVMem answers it by keeping all 1M as blocks and re-choosing which 256K enter the view. The size of what is being kept is what forces the tiering — (illustrative) if one token's KV runs about 40 KB at this quantization, 1M tokens is roughly 40 GB of state, well past the 24 GB on an RTX 5090 Laptop GPU, while the 256K actually resident is about 10 GB, which fits alongside the weights. The bill arrives as throughput: the paper reports about 50 tokens/s, so a 4,000-token agent turn lands near 80 seconds. What that buys is measured on DeepSWE, where task success moves from 43.8% to 48.4% — +4.6 points, about 10.5% relative — against compaction-only management.

Approach	What it stores	What it costs	Where it breaks
Compaction / summarization	A short summary of older turns	Almost nothing — the window stays small	The discarded detail is gone for good; a later step cannot recover it
Retrieval + re-prefill	Raw text, re-prefilled each step	Re-pays prefill compute on the same text, repeatedly	Retrieval matches on text similarity, not on what the query would attend to
KVMem (paged KV + index)	KV blocks across GPU, host RAM and NVMe	A block transfer plus an index lookup per step	Bounded by tier bandwidth — ~50 tokens/s reported, single local session, setup-dependent

The catch is the one every cache has: the index can be wrong. A block the query needed but the clerk did not fetch is invisible to that step — not wrong, just absent — and the model will answer confidently from the desk it was given. That is a softer failure than compaction's, because the block is still on the shelf and a later step can still ask for it, but it means KVMem trades a permanent loss for a recoverable miss rather than removing the loss. It also means the interesting engineering number is not the 1M headline; it is how often the view contains what the step needed, which is the same question eviction policies have always been judged on. If you have been treating context as the scarce resource in an agent, this reframes it: the window is scarce, the memory behind it need not be.

Goes deeper in: LLM Internals → Paged Attention → Virtual Memory for GPUs

Related explainers

vLLM 0.28 tiered KV offload — the same three tiers inside a serving engine, moving bytes without changing what the model addresses
TF-Engram SSD prefetch — how prefetching hides the latency of the slowest shelf
Tangram per-head KV budgets — the other way to fit more history: shrink what each attention head keeps

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based