The news. On July 6, 2026, researchers posted ToolFailBench to arXiv: 1,000 tool-use tasks spanning finance, medicine, law, cybersecurity, and real estate, built to grade how an agent uses its tools rather than only whether its final answer is right. It labels every trace by failure mode — Tool-Skip, Result-Ignore, Output-Fabrication, or Unnecessary-Tool-Use — and reports that the best of 19 headline models keeps its tool use fully clean on just 86.33% of runs. Read the paper →
Picture an open-book exam. The calculator is right there on the desk, and the honest move is to reach for it exactly when the arithmetic is hard and to trust what it says. But there are four ways to get the answer "right" while using that calculator badly. You can skip it and guess on a hard multiplication. You can punch in the sum, see 3901, and write down 3900 anyway. You can write "calculator: 3901" without ever pressing a key. Or you can reach for it to compute 2 + 2 — a question you were meant to do in your head. A grader who checks only the final number on the page cannot tell any of these apart from an honest solve — and that missing distinction is the entire point of ToolFailBench. The calculator is the agent's tool, and those four moves are its tool-use anti-patterns.
An agent's tools are exactly this kind of desk calculator: a search, a query, a code call the agent can invoke instead of answering from its weights. The obvious way to grade the run is to check the artifact it ends with — did the final answer match the key? That cheapest grade is precisely the one that never asks whether the tool was called, trusted, or even touched, which is the pass/fail collapse the Evals & Diagnostics module warns about: one bit of credit for a whole multi-step run. ToolFailBench refuses that single bit and instead sorts the failure into one of four named boxes — the same move, at the level of tool use, that the module makes with its four eval failure modes.
Three of those boxes — skip, ignore, fabricate — can be spotted by reading a single tool-required trace. The fourth, over-use, is the one a correct answer hides, and it needs a cleverer test. So ToolFailBench pairs every tool-required task with a control task: the same tools are attached, but this time they should not be used. Reach for the calculator on a mental-math question and the reach itself is the failure, even though the number you wrote is correct. To keep the labels honest, each trace is scored not by one model but by a rule classifier plus two LLM judges, aggregated by majority vote — three graders so a single misread cannot mint a wrong label.
| Failure mode | What the agent does | Why a pass/fail grade misses it |
|---|---|---|
| Tool-Skip | never calls a tool it needed; answers from guesswork | the guess can still be the right number |
| Result-Ignore | calls the tool, then overrides its answer with its own | the override can coincidentally match the key |
| Output-Fabrication | reports a tool result it never actually received | the fabricated value can match the key |
| Unnecessary-Tool-Use | calls a tool on a control task that forbids it | the answer is right; the tool call itself is the error |
Where the number bites
Here is why the Clean Tool-Use Rate is more than a leaderboard vanity metric. Take the best of the 19 models and its 86.33% clean rate over the 1,000 tasks. That leaves 13.67% of runs — about 137 tasks — where the strongest agent still skipped, ignored, fabricated, or over-used a tool, and each of those failure modes is defined so that the final answer can still look correct. (The 86.33% clean rate, the 1,000-task set, and the 19-model field come from the paper; the 13.67% and ~137-task figures are just its arithmetic complement.) On agents you are about to trust with a database write or a payment API, the gap between "answered correctly" and "used its tools correctly" is the difference between a demo that dazzles and a system you would actually put in front of a customer — which is exactly why this belongs in your online-versus-offline eval plan, not just a one-off benchmark run.
Goes deeper in: AI Agents → Evals & Diagnostics → The 4 Eval Failure Modes
Related explainers
- WeaveBench — trajectory-aware vs outcome-only grading — the sibling move for computer-use agents: grade the whole run, not just the final artifact, to catch shortcut behavior
- TELBench — span-level error localization — once a run fails, TELBench finds where it broke; ToolFailBench names how the tool use broke
- Agentic CLEAR — system/trace/node eval granularity — the broader shift to grading an agent's process at multiple zoom levels, not just its final answer