What is behavioral telemetry for an AI agent?

Behavioral telemetry measures whether an agent **did the right thing** — achieved the user's goal, chose the correct tool, returned a correct answer — as opposed to whether the system underneath stayed healthy. Because those are judgements about content rather than counts, it is produced by graders: LLM-as-a-judge evaluators for open-ended questions, and deterministic matchers for questions with a fixed expected shape.

Why isn't infrastructure monitoring enough for AI agents?

Infrastructure monitoring assumes a broken request is a loud request — a 500, a timeout, an exception. An agent's characteristic failure returns **HTTP 200 inside its latency budget** with a fluent, wrong answer, which moves no counter at all. The dashboards stay green for the entire duration of the incident.

What is the difference between an LLM-as-a-judge evaluator and a deterministic trajectory matcher?

A judge is a model that reads the trace and scores something subjective, like goal success — flexible, but priced per call and subject to its own biases. A trajectory matcher compares the recorded steps to an expected pattern with no model involved: it cannot assess quality, but it is free, repeatable, and can therefore run on **100% of traffic** rather than a sample.

How much production traffic should behavioral evaluation sample?

AWS exposes a range from **0.01% to 100%**, and the right answer usually differs per grader rather than per system. A sensible default runs the **inference-free** deterministic checks at 100% so every trace is checked, and samples the paid judges at whatever rate still makes your smallest interesting regression visible — a 1% regression, for instance, shows up about 1,000 times in a 5% sample of two million turns.

Agent· 2026-09-15

AWS open-sources an agent stack that watches behavior and infrastructure separately — Behavioral vs infrastructural telemetry — What does it mean?

TL;DR

What is it: AWS open-sourced a four-agent airline reservation system built around two separate telemetry layers — behavioral telemetry, which scores whether the agent reached the user's goal, and infrastructural telemetry, which reports whether the runtime underneath stayed healthy.
Why it’s needed: An agent's characteristic failure is behavioral — the model picks the wrong tool, drops a constraint, or answers fluently and wrongly — and none of those has to move latency, error rate or CPU, so a single infrastructure dashboard can report a healthy system right through the outage.
vs previous: Classic application monitoring assumed that a failed request throws, so one layer was enough; an agent that returns HTTP 200 with a confidently wrong answer breaks that assumption, and the behavioral layer has to be measured separately — by judges and trajectory matchers rather than by counters.

Jargon

OpenTelemetry (OTel): A vendor-neutral standard for emitting traces, metrics and logs from a running system. It is the pipe both layers share: the runtime emits once, and different consumers read the same stream for different questions.
Trace and span: A trace is the full record of one agent turn from request to answer; a span is one timed step inside it, such as a single tool call or model invocation. The agent track builds this up as the span-per-tick model.
LLM-as-a-judge: Using a second model to grade an output that no counter can score — was the answer correct, was the right tool chosen, was the goal met. Flexible, but a judge carries a model's costs and a model's biases: see judge bias as activation geometry.
Deterministic trajectory matcher: A check that compares the recorded sequence of agent steps against an expected pattern, with no model in the loop. It cannot tell you whether an answer was good, only whether the agent did what it was supposed to do — and it returns the same verdict every time.
Online vs offline evaluation: Offline evaluation scores a fixed test set before release; online evaluation scores live production traffic as it happens. The curriculum draws the line at Online vs Offline Evals.
SLO (service level objective): The target a request is held to, usually a latency bound such as "95% of responses under 2 seconds". It is the disqualifier that turns throughput into goodput; the serving track covers the percentile mechanics in Percentiles and Tail Latency.
IAM: AWS Identity and Access Management — the service that decides which component is allowed to call which other component. It appears here because a missing permission is an infrastructural failure that surfaces only as a degraded answer.

The news. On September 11, 2026, AWS open-sourced a reference agent stack — a four-agent airline reservation system — with its monitoring deliberately split in two. AgentCore Observability captures the runtime's OpenTelemetry traces and forwards them to CloudWatch, where AgentCore Evaluations then scores those live traces with 16 built-in evaluators — 13 LLM-as-a-judge evaluators and 3 deterministic trajectory matchers — sampling anywhere from 0.01% to 100% of traffic asynchronously, off the user-facing path. A separate AWS DevOps Agent takes an incident by signed webhook, correlates IAM policies, runtime logs and orchestration traces across the invocation chain, and returns a root cause. Read the release →

Picture the kitchen. The equipment inspector walks the line with a clipboard: the oven holds temperature, the fridge is cold, the gas line is sound, nothing is on fire. Every check passes, and the clipboard is honest. Then the dish goes out to a table that ordered something else entirely.

The equipment inspector was never wrong. They were simply not looking at the order slip — and that gap is the entire reason an agent needs a second telemetry layer. Classic application monitoring was built for a world where a broken request throws: a 500, a timeout, a saturated queue, a stack trace. Those are equipment failures, and they are loud. Spans, error rates and latency percentiles catch exactly those, which is why one layer was enough for as long as that assumption held.

An agent breaks differently. It returns HTTP 200 with a fluent paragraph, having called the wrong tool, dropped a constraint the user stated three turns ago, or confidently invented a booking reference. No counter moves. The request was served well inside its latency budget, the runtime never threw, every span is green — and the user is being told something false.

So the second inspector tastes. Behavioral telemetry scores the trajectory against the goal, rather than the runtime against its budget — did the agent achieve what the user asked, did it select the right tool, is the answer correct. Those are judgements about content, so they need graders, not counters.

AWS ships two kinds of grader, and the split between them matters more than the total. Thirteen of the sixteen evaluators are LLM-as-a-judge: another model reads the trace and scores properties like goal success or tool selection. A judge is the most general automated grader, the one that can assess an open-ended answer — and it is also a model, with a model's price per call and a model's biases. The other three are deterministic trajectory matchers, which compare the recorded steps against an expected pattern with no model in the loop. They cannot tell you whether an answer was good. They can tell you, for free and identically every time, that the agent stopped after two steps when the flow requires five.

That is the practical shape the split invites: cheap deterministic checks as a tripwire on everything, expensive judges on a sample.

Hold the traffic fixed and the tradeoff turns into arithmetic. Take an agent handling 2,000,000 turns a month (illustrative — AWS publishes the sampling range, not traffic figures), and work the layer across the 0.01% to 100% range AWS exposes.

Judge every trace at 100% and each of the 13 LLM judges fires on all two million turns: 26,000,000 judge calls a month, every one a model invocation you pay for. Sample at 5% instead and 100,000 traces reach the judges — 1,300,000 judge calls, a 20× cut — while a regression affecting 1% of turns still lands in roughly 1,000 of those sampled traces — enough signal to catch it, provided the alert is watching that rate.

Now run the 3 deterministic matchers at the full 100%. They invoke no model, so two million turns cost 6,000,000 pattern checks and zero inference — and unlike the sample, they see every truncated trajectory and every skipped required step, not a twentieth of them. The judges buy depth on a slice; the matchers buy coverage on everything. That asymmetry is why a 13-and-3 split does more work than 16-and-0 would.

Layer	The question it answers	Instrument	The failure only it can see
Infrastructural	Did the machinery stay healthy?	OpenTelemetry spans, latency, error rates, IAM policies, runtime logs	A timeout, a throttled dependency, a missing permission
Behavioral	Did the agent achieve the user's goal?	13 LLM-as-a-judge evaluators and 3 deterministic trajectory matchers, on 0.01% to 100% of traces	A fluent wrong answer, a wrong tool, a dropped constraint

Serving engineers made a structurally identical split one layer down, years ago, under a different name. Throughput counts every request the system served; goodput counts only the ones that met their SLO. A box can run at full throughput and 60% goodput and still look perfect on every infrastructure counter it owns — which is the kitchen's sentence again, with plates swapped for requests.

Throughput: 10 req/s

Goodput: 3 req/s7 violated SLO

4×

5×

6×

7×

8×

9×

10×

Same system, same second — throughput looks healthy, goodput tells the truth

The serving track draws that distinction out in full. The behavioral layer is goodput for agents: the same refusal to let served stand in for useful. Only the disqualifier changes — a missed latency target there, a missed goal here — along with the awkward fact that the agent version needs a judge to decide, because no counter can see it.

The second half of the AWS release only makes sense once both layers exist. When an incident arrives, the DevOps Agent builds a resource topology and correlates logs, IAM policies and orchestration traces across the whole invocation chain before it proposes a root cause. Keeping the layers apart is what preserves the question: the behavioral layer says what went wrong for the user, and the infrastructural layer says which component made it happen. Root-cause discipline is precisely this refusal to accept the first correlated signal as the cause.

Merge the two into one number and you lose the question. A single blended "agent health" score mixing p99 latency with judged goal success cannot tell you whether to page the on-call engineer or rewrite the prompt — and those are the two actions such a number is supposed to trigger.

Goes deeper in: Agent Engineering → Observability for Agents → Alerting on Agent Behavior

Related explainers

Agentic CLEAR grades agents at three zoom levels — once a behavioral layer exists, the next question is what granularity to score it at.
Demote LLM judges behind five deterministic guardrails — the 13-versus-3 split argued all the way to its conclusion.
LLM-as-judge bias appears as activation geometry — why the judged half of the layer needs calibration of its own.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based