RATED reranks agent tools by live reliability — Reliability-aware tool reranking — What does it mean?
The news. On October 9, 2026, Weiliang Chen and Song Liao of Texas Tech University posted RATED (Reliability-Aware Tool retrieval via Execution Dynamics) to arXiv, a paper for the CIKM 2026 conference. In a simulated outage that disables 30% of the tools, RATED picked a broken tool on 2.9% of steps, against 86.2% for the ToolLLM API retriever, and it added 0.172 ms per step over encoding the query alone. The paper also releases EcoAgent, a benchmark of 5,172 real tools from LangChain, CrewAI, LlamaIndex and n8n with more than 210,000 query–tool pairs. Read the paper →
Picture a hotel concierge on a busy night. A guest asks for an Italian dinner nearby. The concierge opens the guidebook and finds four places that match: Aldo's, Bruno's, Carla's and Dino's. The guidebook is good at matching what the guest asked for, but it was printed months ago, and it cannot know that Aldo's kitchen closed an hour ago. A good concierge does not trust the guidebook alone: they keep a notebook of what guests reported tonight, and two guests coming back from a closed door is enough to send the next one to Bruno's. They also let old notes fade, so a good report from last week counts for less than a complaint from twenty minutes ago.
An agent with thousands of tools has the same problem. Deferred tool loading already moved tool choice out of the prompt and into a retrieval step that returns the top few tool schemas per turn. That step matches the request against tool descriptions, and a description stays the same when the API behind it breaks. The paper also found that many crawled tools do the same job, so meaning alone cannot tell the agent which of several similar tools will actually run.
RATED splits the decision into two stages. First, a frozen dual encoder makes the guidebook shortlist. The encoder is DRAGON, a retriever with one encoder for queries and one for tool documentation, fine-tuned once with a contrastive loss on 69,501 synthetic queries and then frozen. It scores every tool by the dot product of the two embeddings and keeps the top 10. Adding a new tool only means encoding its documentation; nothing is retrained.
Second, a contextual bandit reorders those 10 from execution results. Each tool keeps its own small linear model (LinUCB) whose input is the query embedding. When the agent runs the chosen tool, it gets one bit back: success, or an error or timeout. Only the chosen tool learns from that bit, which is exactly the setting a bandit is built for, because you only see the outcome of the option you picked. The bandit score has two parts: an estimate of how likely this tool is to succeed for this kind of query, plus a small bonus for tools with little evidence, so a tool is not written off before it has been tried.
The final ranking adds three signals: 0.75 × semantic match + 0.15 × bandit score + 0.10 × metadata prior. The metadata prior is the guidebook's star rating, a 0-to-1 score built from catalog fields only: whether the tool's name agrees with its description, how credible the source platform is, and how complete the documentation is. It also seeds each tool's bandit, so a tool with no history starts from its catalog quality instead of from zero.
The detail that makes this work during an outage is forgetting. Plain LinUCB weighs a success from an hour ago the same as a failure a moment ago, so a tool with a long good record keeps winning long after it breaks. RATED uses discounted LinUCB: each time a tool is used, its old evidence is multiplied by γ = 0.95 before the new result is added.
Forgetting is also what separates it from the two fixes a harness usually has. A health-check blacklist removes a tool after one failure and never brings it back. A retry policy tries up to three candidates inside one step, then remembers nothing, so the next request walks into the same broken tool. A discounted bandit sits between them: it remembers across requests, and the memory decays. In principle, a tool that recovers can earn its place back; the paper's outages are permanent, so it does not test that case.
| Method (30% of tools down) | Memory across requests | Picked a broken tool (FSR) | Post-Fail SR (correct tool, and it was up) |
|---|---|---|---|
| ToolLLM API retriever (Table 2) | none, static ranking | 86.2% | 10.5% |
| DRAGON, not fine-tuned, no feedback (Table 2) | none, static ranking | 21.2% | 75.7% |
| Health-check blacklist (Table 2) | permanent ban after 1 failure | 1.9% | 80.2% |
| Retry, up to 3 tries (Table 2) | none; retries inside one step | 19.1% | 82.9% |
| RATED with one global bandit (Table 2) | shared, ignores the query | 13.2% | 74.3% |
| RATED, full (Table 2) | per tool, per query, discounted | 2.9% | 82.1% |
The table has two surprises. The results are a tradeoff, not a clean win: the blacklist has the lowest broken-tool rate and Retry has the highest success rate, while RATED is close to the best on both at once. The blacklist pays in recovery time: with 20% of tools down, it needed 2,620 steps to climb back to 90% of its pre-outage success rate, against 1,603 for RATED, and with 30% down it never reached its recovery target. Retry still picked a broken tool on 19.1% of steps, and it has to spend extra calls inside a step to get to its answer. The global-bandit row shows why the query matters as context: one reliability score per tool, ignoring what was asked, is worse than full RATED on both columns.
Here is what those rates mean in calls. The paper scores the 3,000 steps after the outage starts. With 30% of tools down, the ToolLLM retriever sends about 3,000 × 0.862 ≈ 2,586 requests to a broken tool, and RATED sends about 3,000 × 0.029 ≈ 87, roughly 30× fewer. The forgetting rate is just as concrete: with γ = 0.95, a success recorded 14 uses ago keeps 0.9514 ≈ 0.49 of its weight, and one recorded 45 uses ago keeps 0.9545 ≈ 0.10. So a tool's good record loses about half its weight within 14 uses, which is why a run of fresh failures can overturn it quickly. The price is small: encoding the query alone took 3.223 ms per step on one RTX 5070 GPU, and the full RATED pipeline took 3.395 ms (+0.172 ms), against 17.839 ms for a ToolRL-like baseline that retrains the model on batches of past runs instead of updating per call.
Two limits matter before you copy the design. On real-world queries, RATED's offline ranking is not the best: its MRR of 0.618 is 52.2% above the ToolLLM retriever's 0.406 but below OpenAI Ada-002 (0.670) and SPLADE-v3 (0.626), and the authors say themselves that good offline ranking does not guarantee robust selection when tools fail. And the outages are simulated: a fixed set of tools fails at step 2,000 and stays down, so the paper does not test tools that flap, tools that recover, or new tools with no history, and it lists live agent runtimes as future work. The part that transfers is the split: keep the expensive semantic retriever frozen, and put a cheap, fast-forgetting reliability layer on top that learns from what your harness already logs, the success or failure of each call.
Goes deeper in: AI Agents → Context Engineering → Tool-Loading Strategy
Continue in trackContext Engineering: load only the tools a turn needsRelated explainers
- Contextual-bandit tool routing — the same bandit idea, used to choose between equivalent providers by answer quality rather than to dodge outages
- MCPEvol-Bench: tool-interface drift — another way the tools under an agent change after it was built
- ToolFailBench: a tool-use failure taxonomy — the failures that happen even when the tool itself works