TAO is a tool-aware joint scheduling framework for LLM agent serving. It values each waiting agent session by decayed cache credit, tool-return readiness and token footprint, then picks which sessions to admit and on which GPU node with a **dynamic program**.

Why does tool-return readiness matter for GPU scheduling?

A session whose tool is about to finish will need a GPU within the next scheduling window. Seating a session whose tool is still far from done wastes a slot that a ready session could use, so TAO uses a 1-second readiness horizon to tell them apart.

How is TAO different from cache-aware routing in systems like llm-d or SGLang?

The paper describes llm-d and SGLang as using per-session scoring and routing rules, so each session is assigned on its own. TAO puts cache value, tool timing and token cost on one value scale and optimizes all admissions and placements jointly, so one session's placement cannot block a better combination.

Does TAO win in every setting?

No. The paper's headline gains are the best cells of a 12-setting grid, and its Table 1 shows cells where a baseline is better, such as decode throughput on Terminal-Bench at 16 workers (−8.8%).

Agent· 2026-10-31

TAO cuts agent-serving TTFT by up to 36.3% — Joint session admission and placement — What does it mean?

Jargon

TTFT: Time to first token: how long a request waits before the model emits its first output token. In an agent loop, every returning tool result pays this wait again. See the serving metrics step.
Agent session: One long-running agent task that alternates between model requests (on a GPU) and tool calls (on a CPU), carrying its history across turns.
KV cache: The stored attention keys and values of tokens the model already processed. Reusing it for a later request skips recomputing that prefix, which is called prefill.
Admission: Deciding which waiting requests are allowed onto a GPU node right now, given the node's remaining capacity.
Placement: Choosing which node serves an admitted session. It matters because the KV cache lives on one specific node.
Dynamic program: A method that solves a big choice problem by solving smaller versions first and reusing their answers. TAO uses one to pick the best set of sessions and nodes without trying every combination.
Half-life: The time after which a value falls to half. TAO halves a session's cache credit every 10 seconds of age.
CGDC: Clustered GPUs with Distributed CPUs: the paper's name for serving where model inference runs on a shared GPU cluster and tools run on separate CPU machines.

The news. On October 27, 2026, researchers posted TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. Across 12 model-workload-concurrency settings it reports up to 26.8% lower latency, 36.3% lower mean TTFT and 29.5% higher decode throughput than the best evaluated state-of-the-art baseline for each metric. Read the paper →

Picture a host at a restaurant. Diners keep stepping away mid-meal to take a call, and each leaves a plate on a table that cools as time passes. A GPU cluster serving coding agents lives the same way. An agent session alternates between a model request on a GPU and a tool call (a compiler, a test run, a retriever) on a CPU somewhere else. While the tool runs, the session holds no GPU, but its cached prefix can still sit on one node until something evicts it. When the tool returns, the session needs a seat again, ideally at the table where its plate is still warm.

The paper names two failures in existing schedulers. First, they ignore when a tool will return, so a seat can go to a session that will not be ready for many seconds while a ready one waits. Second, they decide one session at a time, so admitting one session can block a better combination of the others. TAO fixes the first by giving every session-and-node pair a value built from three signals, compared in a fixed order of priority:

Decayed cache credit comes first. It counts the session's prefix tokens still cached on that node, discounted by age with a 10-second half-life: credit = credit0 × 2^(−age / 10 s). A node with no matching evidence gets zero.
Tool-return readiness breaks ties. It is a yes/no flag: the tool has finished, or is predicted or announced to finish within a 1-second horizon. The prediction uses return deadlines the tool exposes, parsed sleep commands and historical durations, and falls back to tool-specific history, global history or a 5-second default when direct timing evidence is missing.
Token footprint breaks the remaining ties, favoring larger accumulated contexts.

TAO fixes the second failure with a dynamic program. Each node has residual capacity, each session has an admission cost, and each session goes to at most one node. That is a multiple-knapsack shape: choose the whole set of sessions that fit, and where, to maximize the total value. The program walks the waiting sessions one by one, tracks used capacity per node, and either skips a session or places it on an eligible node. The paper states it is exact for the frozen, discretized snapshot, though rounding capacity to a quantum can exclude plans that were feasible before rounding. It runs in an asynchronous worker and revalidates against live state before applying; the paper's recorded scheduler work (preparation plus completed solves) is below 0.12% of runtime in its four-replica sweep.

Tool call startssession released from GPU

Waiting poolscore cache, readiness, tokens

Dynamic programadmit and place together

Validatecheck against live state

Restartsession resumes on its node

A worked example shows why the order of the three signals matters (illustrative numbers). Two sessions wait for room on node 1. Session A has 8,000 tokens cached there, observed 10 s ago, and its tool is predicted to return in 0.5 s. Session B has 8,000 tokens cached there, observed 30 s ago, and its tool returns in about 30 s. With a 10 s half-life, A's credit is 8,000 × 2^(−10/10) = 4,000 tokens and B's is 8,000 × 2^(−30/10) = 1,000 tokens, so A wins on cache credit alone. If both had been observed 10 s ago, the credits would tie at 4,000 and readiness (1 for A, 0 for B) would decide. (illustrative)

The paper's own numbers come from DeepSeek-V4 Flash on SWE-bench Lite with 64 workers (2 nodes, 4× H100 each). There TAO's latency was 185.4 against 253.2 for the vLLM cache-aware baseline, its mean TTFT was 0.224 against 0.351 for Dynamo, and its decode throughput was 2021.5 against 1561.3 for the cache-aware baseline, which the paper reports as +26.8%, +36.3% and +29.5%. The table below adds two cells where TAO did not win.

Setting (paper Table 1)	Metric	TAO	Best baseline	Reported change
DeepSeek-V4 Flash, SWE-Lite, 64 workers	Latency	185.4	253.2 (vLLM cache-aware)	+26.8% better
DeepSeek-V4 Flash, SWE-Lite, 64 workers	Mean TTFT	0.224	0.351 (Dynamo)	+36.3% better
DeepSeek-V4 Flash, SWE-Lite, 64 workers	Decode throughput	2021.5	1561.3 (vLLM cache-aware)	+29.5% better
DeepSeek-V4 Flash, Terminal-Bench, 16 workers	Decode throughput	798.6	876.1 (ThunderAgent)	−8.8% worse
Qwen3-32B, SWE-verified, 128 workers	Latency	933.7	911.1 (ThunderAgent)	−2.5% worse

Read the headline figures as the best cells of a grid, not as an average. The 'up to' numbers are the maximum over 12 settings, and each metric is compared with a different best baseline. The two rows above show TAO losing a few cells. For an engineer building agents, the lesson is about structure rather than the percentages: tool latency and GPU caching are one scheduling problem, which is the serving-side counterpart of prompt caching at the agent layer. The paper's appendix G also reports that the gains persist as tool delays grow.

Goes deeper in: LLM Serving → The Inference Engine → The Scheduler

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based