What is the HEAR protocol?

HEAR is a bidirectional protocol, proposed in an October 2026 arXiv paper, that connects an agent harness with an inference engine. The harness sends workflow descriptions and requests for action; the engine returns its current state and capabilities and the outcome of each request. It defines what each message means but leaves the optimization policy to the user.

Why does an inference engine need to know the agent's workflow?

An engine that sees only individual requests evicts KV cache by recency, so it can remove an agent's context seconds before the agent sends its next call on that context. The next call then has to re-run prefill over the whole prompt. Knowing which contexts will be reused lets the engine keep the right ones.

Why does the harness need feedback from the engine?

The harness cannot see which contexts are still cached on the GPU. Without that, it may send a request whose cache was evicted ahead of one whose cache is still warm. On SCBench, using the engine's residency reports raised KV reuse from 19.2% to 87.6%.

Does cache-aware scheduling starve some requests?

It can. On SCBench, running warm requests first pushed the worst time to first token from 79.0 s to 93.9 s. A 40-second waiting guard, treated as a hard requirement so that no request is bypassed after waiting 40 seconds, brought it down to 65.0 s while keeping a 1.61× batch speedup.

How does HEAR relate to prefix caching?

Prefix caching lets the engine reuse KV cache for a repeated prompt prefix, but only while that prefix is still in memory. HEAR adds the missing information on both sides: which prefixes the agent will need next, and which ones the engine still holds.

LLM· 2026-10-07

HEAR protocol — Harness-engine coordination for workflow-aware KV caching — What does it mean?

TL;DR

What is it: HEAR, the protocol in the October 2026 arXiv paper Can Agent Harnesses and Inference Engines Hear Each Other?, is a two-way message contract that lets an agent harness and an inference engine share what each one knows, so the engine can make workflow-aware KV-cache and scheduling decisions.
Why it’s needed: An agent run is many model calls on one growing context, so whether the KV cache for that context is still on the GPU when the next call arrives decides whether the engine reuses it or pays for a full re-prefill, and neither layer alone has the information to get that right.
vs previous: A standard engine such as vLLM serves requests first come, first served and evicts KV cache by recency alone, so it can evict a context seconds before the agent reuses it, and the harness, which cannot see what is cached, can send a cold request ahead of a warm one; HEAR closes both gaps without changing the workflow or the model.

Jargon

Agent harness: The program around the model that runs the agent loop: it builds each prompt, calls tools, and decides which model call comes next. LangGraph and AutoGen are examples. See The Anatomy of a Harness.
Inference engine: The server that runs the model, such as vLLM or SGLang. It queues requests, batches them on the GPU, and manages KV-cache memory. See The Inference Engine.
KV cache: The keys and values the model already computed for earlier tokens, kept in GPU memory so the next call on the same context does not recompute them. See The KV Cache Solution.
Prefill: The first phase of a model call, where the whole prompt is processed at once to build its KV cache. A long prompt makes prefill expensive, which is why reusing a cached prefix saves time.
Eviction: Removing a cached context from GPU memory to make room for another one. Most engines evict the least recently used entry, which knows nothing about what the agent will do next. See Eviction and Copy-on-Write.
TTFT: Time to first token: how long a request waits from arrival until the first output token appears. It includes queueing time and prefill time.
FCFS: First come, first served: the engine runs ready requests in arrival order, whether or not their KV cache is already on the GPU.
Waiting protection (guard): A hard rule that a request which has waited longer than a threshold, for example 40 seconds, can no longer be skipped by warmer requests. It stops cache-aware ordering from starving cold requests.

The news. On October 5, 2026, a team from Harbin Institute of Technology (Shenzhen) posted Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving to arXiv. The paper defines HEAR (Harness–Engine pAiRing), a bidirectional protocol with four message categories, and tests it with two policies: cache-aware request ordering with KV retention, and per-role selection of inference modes. It reports a 1.61× batch speedup and 2.23× lower median time to first token on SCBench, and 1.23× and 2.45× end-to-end speedups on BrowseComp-Plus and DeepResearchBench, without observed loss in task quality. Read the paper →

Picture a busy restaurant where the waiter and the kitchen never talk. The kitchen sees only order slips: when the counter fills up, it clears whichever station has been idle longest. The waiter sees only the tables: table 3 is finishing its main course and will order dessert in a moment, but the kitchen does not know that, so it clears table 3's station and has to prep it again from scratch. The kitchen is missing the waiter's knowledge of who comes back next, and the waiter is missing the kitchen's knowledge of what is still prepped. Each side makes a choice that looks reasonable with the information it has, and the meal gets slower.

An agent run has the same split. The agent harness knows the workflow: which subagent is waiting on a tool call, which context will be resumed, which branch is finished. The inference engine knows the runtime: which contexts still have KV cache resident on the GPU, how long the queue is, how much memory is free. The paper names both failures. Without workflow intent, the engine may evict a context's KV state shortly before it is reused, which forces a costly re-prefill. Without residency feedback, the harness may send a cold request ahead of an equally ready one whose cache is still warm, so the scheduler wastes the warm cache too.

HEAR is the set of notes through the hatch. The harness sends two kinds of message (a description of the workflow, and requests for action), and the engine sends two kinds back (its current state and capabilities, and the outcome of each request). The paper is strict about what each note means, with four rules: saying a context will be reused is not the same as asking the engine to keep it; a preference may be ignored but a requirement must be met or explicitly rejected; a report that a prefix is on the GPU is an observation, not a reservation; and an engine that accepted a request to prepare a cache has not yet completed it. These rules matter because both layers change on their own, so a policy that treats a hint as a command or an old report as a promise will act on a state that no longer exists.

Harness describesready requests, future reuse

Engine reportsKV residency, queue state

Policy orderswarm requests first

Guard appliesno skip after 40 s wait

Engine returnsoutcome, cache reuse

Walk the paper's SCBench run, holding three things fixed: the model (Qwen3-8B), the hardware (one RTX PRO 6000 GPU), and the workload (multi-turn conversations with random arrivals and think time between turns). Under plain FCFS, only 19.2% of follow-up prompt tokens reused cached KV and the rest were recomputed, and finishing every conversation took 481 seconds. When the harness uses the engine's residency reports to run warm requests first, reuse rises to 87.6%, median TTFT falls from 63.1 s to 4.6 s, and the batch finishes in 224 s (481 ÷ 224 ≈ 2.15×). The cost is the cold requests that keep getting skipped: the worst time to first token rises from 79.0 s to 93.9 s. Adding a 40-second guard, after which a waiting request can no longer be bypassed, gives a worst time to first token of 65.0 s and a batch time of 299 s (481 ÷ 299 ≈ 1.61×). The guard spends 75 seconds of batch time (299 − 224) to remove about 29 seconds from the worst time to first token (93.9 − 65.0), which is the trade a tail-latency target asks for. That is also why the guard is a requirement in HEAR's terms, not a preference: the scheduler may reorder for cache reuse, but never past a request that has already waited 40 seconds. The guard limits skipping; it does not cap the wait itself, since queueing and prefill still add time after the guard applies.

Policy (SCBench)	Median TTFT	Max TTFT	Batch completion	KV reuse	Source
FCFS (default engine order)	63.1 s	79.0 s	481 s	19.2%	Table 3
Cache-aware ordering	4.6 s	93.9 s	224 s	87.6%	Table 3
Cache-aware + 40 s guard	28.3 s	65.0 s	299 s	66.1%	Table 3
Cache-aware + 60 s guard	4.5 s	80.4 s	230 s	84.7%	Table 3

The second test, on a production-derived Mooncake trace served by Qwen3-8B on one H100, shows that the harness's hint alone is not enough. Here the harness estimates how likely each session is to continue and sends that as a description, not a command; the engine combines it with recency to decide what to keep. At 50% load, where requests rarely queue and idle-time eviction is the main loss, this raised KV reuse from 19.5% to 31.9%. At full load, stacking session-aware retention, cache-aware ordering and the guard together gave almost no latency gain and 0.89× the FCFS throughput, while cache-aware ordering alone or session-aware retention alone each beat FCFS. No single policy won at every load, so the hint about future reuse is only useful when the engine can weigh it against its live queue and memory state. This is the paper's main argument for a two-way protocol instead of a one-way hint field.

The same messages also choose how each agent role runs, on a slower timescale. In a LangGraph harness with a Main agent and Reader subagents, the harness describes each role's workload and the engine reports which KV-compression modes it supports and how they perform. Running the Readers with H₂O (a mode that keeps KV only for the most-attended tokens plus the most recent ones) cut their mean latency by 5.60× on BrowseComp-Plus and 7.22× on DeepResearchBench. The best mode for the Main agent depended on the workload, and the selected setups gave 1.23× and 2.45× end-to-end speedups with accuracy of 46.63% vs 46.15% and a RACE report-quality score of 41.1% vs 40.6% against the plain vLLM setup. Using one fixed mapping for both workloads would have cost 7.8% and 14.0% more time, which supports the paper's point that the choice should come from the workload description, not from a fixed setting.

Two limits are worth stating. HEAR is a contract, not an optimizer: the speedups come from the policies built on top of it, and the paper says its benefits depend on which information and controls a given serving stack exposes. The tests also ran a single engine on a single GPU per role, so behaviour in a large multi-node deployment is not measured. For production agent work, the durable idea is the boundary itself: the harness should tell the engine which contexts it will reuse and when (in prompt-cache terms, which prefix is worth keeping), and the engine should tell the harness what is actually cached, instead of each side guessing.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → Eviction and Copy-on-Write

Related explainers

Cache-preserving mid-task system messages — the client-side version of the same goal: change a running agent's prompt without throwing away its cached prefix
Cache-state divergence in agent runs — what prefix-cache state can do to an agent's behaviour, not only to its speed

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based