Schedule agent tools by task-specific CPU bottlenecks — Task-aware CPU admission vs uniform resource allocation — What does it mean?
The news. On 17 September 2026, researchers from Korea University and Microsoft Research Asia posted Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics — a measurement study of retrieval-augmented QA, web search and coding agents running on one 24-vCPU Azure VM. They replay recorded model responses so the numbers reflect local execution rather than API jitter, push the request rate from 0.5 to 6 per second, and find that each task type saturates a different local resource. Two policies follow: capping concurrent tool executions, and sizing each task's core count by its measured CPU sensitivity. Read the paper →
Picture three orders on the pass. The first is a salad: almost all of its time is real work at the station, chopping. The second is a delivery order — the kitchen barely touches it, and nearly all of its elapsed time is the van in traffic. The third is a braise: it sits for a long time, but most of that is the head chef deciding what to do next, with genuine station work in between. All three take wildly different amounts of wall-clock time, and the order that takes longest is not the order that jams the stove.
That is the paper's central measurement, and it is unintuitive enough to be worth stating plainly. Splitting each task's end-to-end latency into local tool execution versus waiting on a remote service, retrieval QA spends 62% of its time in its own container, web search spends only 10.5% there (49% of its time goes to web APIs), and coding spends 39%, with the model call dominating the rest. The average CPU use follows that split almost exactly — 86% for retrieval QA, 14% for web search, 22% for coding — because a request burns CPU while its tools run and sits idle while it waits. Coding produces the longest requests in the study, from 21.5 to 95.4 seconds, and it is still not the heaviest CPU consumer.
The consequence is a provisioning rule. If you size an agent's container by how long its requests take, you will give the most CPU to the task that needs it least. End-to-end latency is the wrong meter; the local execution share is the right one. This is the same distinction the cost profile of an agent step draws between time an agent spends and money an agent costs — here it separates time an agent spends from capacity an agent occupies.
Now crowd the kitchen. As the arrival rate climbs from one order every two seconds to six a second — a 12× increase — the three orders respond nothing alike. The two retrieval-QA benchmarks slow by 10.2× and 13.1×; the two web-search benchmarks slow by 1.04× and 1.5×. Retrieval QA hits its CPU ceiling first, at 1 request per second; web search does not saturate until 2 to 4. The ordering is exactly the ordering of the stove shares above, and for the obvious reason: waiting on a van does not occupy a burner, so two orders can wait side by side for free, while two orders that both want the stove have to take turns.
The stove is not the only thing that can jam, which is the part most capacity plans miss. One coding benchmark in the study slows 6.4× under load yet does not become CPU-bound until 4 requests per second — its disk is already pinned at roughly 200 MB/s against a measured 207 MB/s ceiling from the very lowest request rate, and the cores are simply waiting on writes. Memory moves independently again: it grows 5.8× for web search and 1.6× for coding over the same range. Three tasks, three different resources, one shared host.
Little's Law: L = λ × W is exact for any stable queue. Service time fixed at 50 ms. As λ approaches capacity (~100 req/s), W diverges — that's the knee in the saturation chart on the right.
The dials above are why the slowdowns are so lopsided. They plot the queueing relationship between arrival rate, waiting time and work in flight — drag the arrival rate and watch the wait. Queueing does not degrade gently: while a resource has headroom, adding load costs almost nothing, and past the saturation point waiting time climbs a near-vertical wall. A task that reaches its CPU ceiling at 1 request per second is already on that wall at 2, which is how a 12× rate increase turns into a 13.1× latency increase for HotpotQA. Which resource saturates first is what governs the curve, and the paper measures a different answer for each task type.
The sharpest result falls straight out of this. The authors sweep one representative benchmark per task type at a fixed 0.5 requests per second. For HotpotQA, widening its container from 2 to 8 cores at the baseline recorded model speed cuts end-to-end latency 13.0×. Tripling the model's response time over the same grid raises latency by just 1.01× at 2 cores — the effect only grows to 1.48× at 8 cores, once throttling has already been relieved. For this benchmark, at this rate, the model is nearly free and the burner is everything. That is the measured form of the authors' summary that "faster LLM responses or more CPU cores do not always accelerate agents": a faster van does not clear a jammed stove, and past the point where throttling is relieved, more burners stop helping either.
So an expediter helps — sometimes. CPU-aware tool admission caps the number of tool executions running at once to the number of cores the container owns, and queues the rest. At 0.5 requests per second, that cap makes HotpotQA 3.23× faster on average, because the contention it removes costs more than the queueing it adds. The identical policy makes SWE-bench 2.76× slower on average — coding shows only mixed CPU sensitivity, so its smaller CPU-side gain is outweighed by the added queueing delay. This is a lever that has to be switched on per task type, and a reminder that parallelizing tool calls is not always a win either; both depend on which resource is actually scarce.
| Policy (total budget) | web search | retrieval QA | coding | total avg latency |
|---|---|---|---|---|
| uniform ReAct (6 cores) | 2 cores | 2 cores | 2 cores | baseline |
| sensitivity-guided (6 cores) | 1 core | 3 cores | 2 cores | −32% |
| uniform ReAct (12 cores) | 4 cores | 4 cores | 4 cores | baseline |
| sensitivity-guided (12 cores) | 2 cores | 6 cores | 4 cores | −23% |
Walk the six-core row, because the whole argument fits inside it. Hold three things fixed: a six-core budget for the tool containers, 0.5 requests per second for each of the three benchmarks, and the same recorded model responses. Split the budget evenly and every container gets 2 cores — that is the native ReAct baseline. Now spend the same six cores by measured sensitivity: BFCLv4 (web search) drops to 1 core, HotpotQA (retrieval QA) rises to 3, SWE-bench (coding) stays at 2, and HotpotQA also gets the admission cap from above. One core moved. HotpotQA, the benchmark that was pinned against its limit, comes out 5.4× faster; BFCLv4, which spends most of its time waiting on remote web APIs, pays 28% slower; SWE-bench's allocation is unchanged. Averaged across all three, end-to-end latency falls 32% on identical hardware. Double the budget to twelve cores and the same reshuffle still returns 23% — smaller, because at 4 cores each the uniform split was already leaving less on the table.
Goes deeper in: Agent Engineering → Cost & Latency → The Cost Profile of an Agent
Related explainers
- The Vera CPU and agent step latency — the hardware answer to the same observation: if serial host work bounds an agent step, build a CPU for single-thread speed.
- AgentZip and LLM-wait latency hiding — the scheduling answer from the other direction: move host work into the window where the agent is already waiting on the model.