What is LLM-wait latency hiding?

It is scheduling expensive background work — in AgentZip's case, compressing an agent sandbox's memory — into the period when the sandbox is blocked waiting for the model to generate its next tool call. The work still costs CPU, but most of that cost stops being visible, because the time it uses was already being spent idle. Not all of it: the paper reports a residual 1.40× on the foreground, not zero.

Compressing agent sandbox memory is the obvious answer to fan-out memory pressure, and it is normally rejected because it slows tool calls down. The paper reports that scheduling plus restore-time prefetching cuts that slowdown from as high as 3.1× to 1.40×, which is the difference between a technique you cannot use in production and one you can.

How is this different from the way Linux already compresses memory?

zswap and zram compress when memory runs short. That trigger is blind to what the program is doing, so the compression frequently lands during a tool call and competes with it for CPU. AgentZip triggers on the agent's execution phase instead, and decompresses pages ahead of the accesses that will need them.

When does this stop helping?

When there is no window. A fast local model that replies in a few hundred milliseconds, or an agent that chains tool calls without a model turn between them, leaves too little idle time to hide the work in. The technique scales with how slow and how expensive the model turn is.

Agent· 2026-09-12

AgentZip compresses agent sandboxes 8.7× — LLM-wait latency hiding — What does it mean?

TL;DR

What is it: AgentZip is a research memory system for the sandboxes that agent tool calls run inside, and LLM-wait latency hiding is its central move: run the expensive compression work only in the window where the sandbox is already sitting idle, waiting for the model to write the next tool call.
Why it’s needed: Every agent in a fan-out needs its own sandbox — a whole Linux process tree — so the memory bill grows in a place no token counter shows, and the obvious fix, compressing memory harder, steals CPU from the very tool calls the user is waiting on.
vs previous: The previous approach is the stock Linux configuration the paper measures against — memory compression driven by pressure rather than by what the agent is doing, the way Linux's own zswap and zram work; AgentZip instead schedules compression into the model's generation window and prefetches pages back before they are touched, which the authors report cuts aggressive compression's slowdown from as high as 3.1× down to 1.40×.

Jargon

sandbox: An isolated environment — its own process tree and file system — that an agent runs tool calls inside, so a bad command cannot reach the host. One per agent, not one per task.
high-fanout: One task that spawns many agents at once, so memory demand grows with the width of the fan-out rather than its length. Agent Teams covers when that width earns its cost.
page: The unit an operating system manages memory in, usually 4 KB. A page is cold when nothing has touched it for a while, which is what makes it a safe candidate to compress — and compression happens page by page, so a system can pack part of a sandbox and leave the rest alone.
zswap / zram: Linux's built-in compressed memory. It keeps a compressed pool in RAM and pushes cold pages into it when memory runs short — a decision driven by pressure, never by what the program is currently doing.
foreground slowdown: How much longer real work takes because compression is competing with it for CPU. A 3.1× foreground slowdown means a tool call that took one second now takes 3.1 seconds.
restore-time prefetching: Decompressing pages ahead of the accesses that will need them, rather than one at a time as the program faults on each — which is what turns the restore side from a source of foreground stalls into a much smaller one.
latency hiding: Moving work into a window where you are already blocked on something else, so most of its cost stops showing up in wall-clock time — rarely all of it, since AgentZip still reports a residual 1.40×. The same idea as overlapping a memory transfer with compute on a GPU.

The news. On September 10, 2026, a paper posted to arXiv described AgentZip, a memory system for the sandboxes that agent tool calls run inside. The authors report sandbox-owned memory reduced by up to 8.7×, against 2.1× for the Linux configuration they compare against. The part this article is about is the second claim: restore prefetching plus execution-aware scheduling cut the slowdown from aggressive compression from as high as 3.1× to 1.40×. Read the paper →

Picture the stage between two scenes. The curtain is down, the house lights are up, and for ninety seconds nothing is being performed. That is when the crew strikes the old set and stacks the flats against the back wall — not because the theatre has suddenly run out of room, but because it is the one window in the evening where moving heavy things costs the audience nothing.

An agent's sandbox has exactly such a window, and it is the larger part of the evening. A tool call runs for a few hundred milliseconds; the model then spends seconds writing the next one. For most of an agent's wall-clock life the sandbox is a fully provisioned Linux process tree doing absolutely nothing — the curtain is down and the stage is free.

The crew that gets this wrong is the operating system's own. The paper's comparison point is a stock Linux configuration, and a Linux box compresses memory the way zswap and zram do — it starts packing pages when free memory runs short. Memory pressure is not a stage direction: it arrives in the middle of a scene, and the packing then competes for CPU with the shell command the agent is waiting on. The paper's contrast is not about how expensive the compression is, but about when it runs: pressure-driven packing is not aligned with the agent's execution phase, so it lands where it shows.

That is why the other half of the design matters. Striking the set is only mostly free, and only if the next scene's props are already waiting in the wings, so AgentZip pairs the scheduling with restore-time prefetching: when the sandbox starts touching pages again, the system decompresses ahead of the accesses that will need them, rather than stalling the tool call on each page in turn.

Tool call runssandbox CPU busy

Model generatessandbox idle — the window

Compress herecold pages packed, free CPU

Prefetch backpages restored before touch

Next tool callback to near full speed

The scheduling also changes what can be compressed, not only when. A memory system that must stay out of the foreground's way can only touch pages it is confident are cold, because every mistake costs a visible stall. Moving the work out of the foreground widens the set of pages worth packing at all, because packing one no longer risks stalling a tool call that is running right now — the paper describes this as broadening the compressible set rather than merely rescheduling the same work.

Design	When it compresses	Foreground slowdown	Memory reduction
Linux configuration	pressure-driven, not execution-aware	not reported	2.1× (paper)
Aggressive compression, unscheduled	unscheduled, not execution-aware	as high as 3.1× (paper)	not reported separately
AgentZip	in the model's generation window, with restore prefetch	1.40× (paper)	up to 8.7× (paper)

Hold three numbers fixed and the trade becomes arithmetic. Say one tool call needs 200 ms of sandbox CPU, a task makes 40 tool calls, and the model takes 3 s to write each one (all three illustrative; the ratios below are the paper's). The model time alone is 40 × 3 s = 120 s — time the sandbox spends idle — so an uncompressed task runs 120 s + 40 × 200 ms = 128 s.

Unscheduled aggressive compression at 3.1× turns each 200 ms call into 620 ms. The extra 420 ms lands 40 times, so the task grows by 16.8 s — 13% longer, every second of it in front of the user. AgentZip at 1.40× turns the same call into 280 ms, adding 80 ms × 40 = 3.2 s, or 2.5%. The compression did not get cheaper; 13.6 seconds of it moved into the 120 seconds the sandbox was already wasting.

The window is the whole premise, which tells you where the idea stops paying. A small local model that answers in 150 ms leaves almost no interval to hide work in, and an agent that streams tool calls back-to-back without a model turn between them has no curtain at all. The technique is strongest exactly where agents are heading — slow, expensive, reasoning-heavy model turns wrapped around short tool calls — and it is worth asking of any agent cost you are looking at: is there a window here, and is anything using it? Answering that starts with knowing where an agent's time and money actually go, the same question behind parallelizing tool calls and behind what it costs to coordinate a team of agents.

Goes deeper in: Agent Engineering → Cost \& Latency → The Cost Profile of an Agent

Related explainers

AgentZip compresses agent sandboxes 8.7× — Cross-sandbox memory redundancy — the other half of the same paper: what each page is compared against, and why 200 clones of one template compress far better than 200 strangers.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based