Agent Behavior Maps — Centered action log-likelihoods — What does it mean?
The news. As of October 21, 2026, a new arXiv paper, Agent Behavior Maps: Locating Agents by What They Would Do, Without Running Them, proposes a way to compare LLM agents without collecting new trajectories from each one. It maps 100 open-weight LLMs across four environments and tests whether map distance predicts success and similar behavior. Read the paper →
Picture a chess club that wants to know which of its 100 members play alike, without scheduling a tournament. It keeps one shelf of recorded games instead. Every member reads the same recorded positions and, for each move that was actually played, says how natural that move feels to them. That row of ratings, after the member's general self-assurance is subtracted, is the member's place on the map.
An Agent Behavior Map does this for LLM agents. The harness rebuilds the model's input from a recorded trajectory's external history: the task, the observations and the past actions. The recorded action is then teacher-forced: the agent reports the log-probability it gave that action. One number per step gives a vector; subtracting the agent's own mean centers it. Because the reference trajectories are reused, a new agent costs scoring passes only, with no new environment runs. It is the same habit as reading traces before scoring them, applied across agents instead of across failures.
The catch is that the recorded games belong to someone else. An agent's own run visits positions its own choices create, and the shelf may never contain them. The paper answers with a bound and a test. If two agents' action probabilities stay close on every history (locality) and each agent's run distribution stays close to the reference (reference mismatch), then squared map distance approximates the trajectory KL, with an error that grows with trajectory length: the step version scales with the length T, the trajectory version with about T².
The test used 8 models on ALFWorld, 28 model pairs, with trajectory KL measured by actually running and cross-scoring them. The step-level estimate correlated with it at Pearson 0.90 and Spearman 0.95; the trajectory-level estimate at 0.86 and 0.94. The map preserves the order of the divergences, not their size: the paper reports that both versions overestimate the measured KL in level.
A worked example (illustrative). Take four recorded moves. Player A rates them −0.2, −1.0, −0.4, −0.2 (log-likelihood), mean −0.45. Player B hesitates and rates them −1.7, −2.5, −1.9, −1.7, mean −1.95. Raw, they differ by 1.5 on every move. Subtract each mean and both sheets become +0.25, −0.55, +0.05, +0.25. After centering the sheets are identical and the distance is 0: B is a less confident player, not a different one.
Now the cost, with the paper's setup of 500 reference trajectories and 10,388 steps in total. Mapping 100 agents is 100 × 10,388 = 1,038,800 scored steps. Measuring pairwise trajectory KL directly needs rollouts of all 100 agents and 100 × 99 / 2 = 4,950 pairs to cross-score. (The four ratings are made up for the sketch; the 500, the 10,388 and the 100 agents are the paper's.)
| Question | Result | Scope | Source |
|---|---|---|---|
| Does map distance track the trajectory KL? | Pearson 0.90, Spearman 0.95 (step version) | 8 models, 28 pairs, ALFWorld | paper §3.1 |
| Predict success without running the agent | kNN r = 0.88 vs 0.63 for a model-size baseline | ALFWorld, leave-one-model-out over 100 models | paper Table 2 |
| Same prediction on a different environment | MiniWoB++: kNN 0.28 vs 0.64 for size; Ridge with size 0.63 | MiniWoB++, same 100 models | paper Table 2 |
| Four-benchmark average | Ridge with size 0.83 vs 0.74 for size alone | ALFWorld, TextWorld, BFCL, MiniWoB++ | paper Table 2 |
| Do map neighbors act alike? | Effect size 0.88 vs 0.40 for size neighbors (edit distance) | ALFWorld, five nearest models | paper §3.3 |
| Do harness effects add up? | Median cosine 0.90, norm ratio 0.98; weakest replan + think at 0.81 | 12 LLMs, τ-bench traces, 60 two-component cases | paper §4.2 |
Now hand one player a coach's notebook. In the paper that is a harness component such as retrieval, thinking or replanning. The shift in coordinates from the BASE harness to the harness with one component added is a harness vector, and summing two harness vectors predicts the coordinates of the combined harness. The paper scores 5 configurations for each of 12 LLMs (60 agents) and builds composed coordinates for 7 more per LLM (84 in all). Harness effects roughly add, with accuracy that depends on the component and the environment: retrieve with planning or reasoning agreed best (cosine 0.97 and 0.94), while replan + think was weakest, with a measured shift shorter than the sum predicted (norm ratio 0.74), all on τ-bench traces.
The map has limits the paper states itself. It needs access to log-likelihoods, it depends on which trajectories sit on the shelf, and its guarantees are local. In the table the size baseline beats the coordinates on MiniWoB++, and the neighbor effect survives controls for success rate and size mainly on ALFWorld. Treat the map as a cheap screen for who to run next, in the spirit of shadow mode, which also tests a candidate before it acts, though the map goes further and never runs the candidate at all.
Goes deeper in: AI Agents → Evals & Diagnostics → Error Analysis First
Related explainers
- A 176-setting study ablates the coding-agent harness — the measured side of harness components, where the map predicts a shift instead
- LongHarness — evidence-conclusion coevolution — another way a harness changes what an agent does