TAM benchmark — Exact-match scoring over long procedural rule chains — What does it mean?
The news. On September 11, 2026, researchers published Tasks over Application Manuals (TAM), a benchmark for long-horizon procedural reasoning. It curates human-validated tasks from two domains where an authoritative manual defines the right answer: ICD-10-CM clinical coding and U.S. federal sentencing. Each task requires following a manual of hundreds of pages, executing a sequence of interdependent steps drawn from tens of thousands of rules, and producing one exact answer. Evaluating retrieval-augmented prompting, ReAct-style prompting and an agent-harness baseline on GPT-5, the authors report best exact-match scores of 1% on ICD-10-CM coding and 15.5% on sentencing, and release the data and code. Read the paper →
Picture the customs desk in the illustration. A crate arrives with a written description, and the officer has to write exactly one code on the form. Reading the description is not the difficult part. In the picture, the difficulty is that the book does not answer the question, it answers with a redirection: the index sends you to a heading, the heading carries a note excluding the category you were about to pick, that note sends you to a different section, and a further rule there decides the last character of the code.
TAM's tasks are that walk, not a lookup. Its two domains are ICD-10-CM clinical coding, which maps a written medical condition to one diagnostic code, and U.S. federal sentencing, which computes an offense level from the facts of a case. Both are governed by a manual running to hundreds of pages and holding tens of thousands of rules, and both require a sequence of interdependent steps across different sections of that manual before a single exact answer falls out. The authors call this long-horizon procedural reasoning, and the horizon is the whole point: the steps depend on each other, so an early wrong turn does not cost you one step, it invalidates every step after it.
The grading matches the work. TAM scores by exact match — the answer equals the human-validated label or the task is failed, with no credit for a chain that reasoned well right up to its last step. That is not the benchmark being harsh. Exact match tests whether the complete procedure produced the human-validated answer, not whether the individual intermediate steps looked plausible. Pass/Fail vs Score covers when a graded rubric is the honest measure of an agent and when it is a way of not looking at a failure.
Hold the chain length fixed and the arithmetic does the rest. Take an eight-step chain (illustrative — TAM does not fix one step count) and suppose the agent is independently right at each step with probability p. Exact match needs all eight, so the end-to-end rate is p8. At p = 95% that is 66%: a third of the tasks lost to steps you would describe as reliable. At p = 90% it is 43%. At p = 80% it is 17%. Now run it backwards from the reported results. The best GPT-5 baseline scored 1% exact match on ICD-10-CM coding, which over an eight-step chain implies roughly 56% accuracy per step, and the 15.5% on sentencing implies roughly 79% per step. Neither of those per-step figures looks like a broken system. That is the finding: a system can be right at most individual lookups and still fail nearly every task, and a benchmark that asks for only a small number of steps cannot tell those two states apart.
| Domain | Best reported exact match | Implied per-step accuracy |
|---|---|---|
| ICD-10-CM clinical coding | 1% | ~56% (illustrative 8-step chain) |
| U.S. federal sentencing | 15.5% | ~79% (illustrative 8-step chain) |
Two things follow for anyone building on this. First, measure the chain, not the step. An eval that reports per-tool or per-hop accuracy stays healthy right up to the moment a customer reports the end-to-end number; Compounding Errors runs the same arithmetic in the useful direction, from a target end-to-end rate back to the per-step accuracy you have to pay for.
Second, find out which step is failing before you buy anything. TAM's baselines span retrieval-augmented prompting, ReAct-style prompting and an agent harness; across all three the best reported exact-match performance stays extremely low — 1% on ICD-10-CM coding and 15.5% on sentencing — but the paper does not say which part of the chain caused the failures. Retrieval quality, rule application, state tracking across sections and recovery from a wrong turn are all live hypotheses, and they send the budget to different places: into retrieval, or into making each attempt recoverable so a wrong turn at step 2 does not propagate silently to step 8. For an agent already in production, the same question is asked of the gate rather than the model — see Online vs Offline Evals.
Goes deeper in: AI Agents → Evals & Diagnostics → Compounding Errors
Related explainers
- Workflow-GYM — End-to-end GUI workflow completion — the same all-or-nothing grading applied to computer-use agents.
- ToolFailBench — Tool-use failure taxonomy — once the chain fails, this is how to label which step broke it.
- SWE-Bench Pro — Benchmark task validity — the other way a benchmark misleads: when the answer key itself is wrong.