Transferable tool-selection paper — Plan, Retrieve, Select with a trained 8B model — What does it mean?
The news. On October 22, 2026, researchers at Meta and Carnegie Mellon published Training Small Language Models for Transferable Tool Selection. They post-train Qwen3 models (1.7B, 4B and 8B) to run a Plan, Retrieve, Select workflow over a BM25 retriever, and report that the 8B model beats a prompted Llama-4-Maverick (402B) on ToolBench and carries over to other tool corpora and to a different retriever without retraining. Read the paper →
Picture the warehouse. A customer hands over a job note: book a flight and find a hotel. The clerk at the counter cannot read the whole note and understand it; the clerk only matches words on labels. Hand over the whole note and the flight words and the hotel words get mixed into one pile of shelves. So the foreman first splits the note into one line per need, sends the clerk off with one line at a time, gets back a single shelf of at most 15 items, and picks the one item that fulfills that line. The union of the picks is the answer.
That is the paper's loop. Retrieval narrows; selection decides. A retriever returns a ranked list and cannot say how many of those tools the task really needs. The paper puts the model in charge of the loop and treats the retriever as a tool the model calls on demand. This is the same pressure described in the tool-loading step of Context Engineering: loading every schema into context is not an option once the tool count grows, so something has to decide which few schemas to load. The retrieve-then-act shape also mirrors retrieve-then-generate in RAG, with the difference that the output is a tool set instead of an answer.
Each model call is stateless and bounded. The Plan call sees only the instruction. A Select call sees only one substep and at most K candidates (the paper uses a score threshold of 3 and K = 15). So the context for one call does not grow with the size of the corpus or the length of the plan. The same model plays both roles under two different prompts. The tool schema of each candidate is just its documentation text, which is why the selector needs no knowledge of a specific domain: it only has to match a short request against short descriptions.
Training happens in two stages. First, supervised fine-tuning on synthetic trajectories written by Llama-4-Maverick as a teacher. The authors filter the gold tools so that a (query, tool) pair is kept only when the tool can be identified from the query alone, because ToolBench's gold tools come from a reference solution and some are not implied by anything the user wrote. Second, reinforcement learning with GRPO, with Plan and Select trained as separate stages. The Plan reward is the F1 of precision and rank-graded recall of the gold tools retrieved by the substeps, so redundant substeps and buried gold tools both cost reward. The Select reward is 1 only when the model picks exactly the gold tool and nothing else. Neither reward needs human labels beyond the gold set.
| Setup (ToolBench G3 test, 2,124 instructions) | Success rate, exact match | Source |
|---|---|---|
| Llama-4-Maverick (prompted) + hierarchical search | 8.6% | Table 1 |
| Llama-4-Maverick (prompted) + this paper's workflow | 10.1% | Table 1 |
| Llama-4-Maverick (prompted) + ReAct | 28.4% | Table 1 |
| Llama-4-Maverick (prompted) + Search-R1 style | 30.6% | Table 1 |
| Qwen3-1.7B, SFT + RL, this workflow | 37.6% | Table 1 |
| Qwen3-4B, SFT + RL, this workflow | 40.3% | Table 1 |
| Qwen3-8B, SFT + RL, this workflow | 41.5% | Table 1 |
Read the first rows twice. The same workflow gives the untrained 402B model only 10.1%, but a trained 8B model 41.5%. The workflow alone is not the win; training on the workflow is. The paper's ablations point the same way: with the Select stage fixed, a trained 8B planner reaches 41.5%, untrained Qwen3-8B 27.6% and Maverick 17.2%, so the larger model plans worse. Replacing the learned Select stage with a rule that always commits the top-1 BM25 candidate gives 28.6%. Warm-up matters too: starting from the untrained model 20.5%, SFT alone 36.2%, RL alone 35.7%, and SFT then RL 41.5%.
Why is exact match such a hard bar? Take a three-substep instruction and suppose, as an illustration, that the selector is right on any one substep 90% of the time and the substeps are independent (illustrative). The whole set is then right only 0.9 × 0.9 × 0.9 = 72.9% of the time, and one wrong or extra tool fails the instruction. Now the context side, also illustrative: if each tool description costs about 100 tokens, a 10,000-tool corpus is about 1,000,000 tokens, while one Select call with K = 15 sees about 15 × 100 = 1,500 tokens (illustrative). The loop trades a few small calls for never loading the corpus at all.
Transfer is the second claim, and it is partial. Trained only on ToolBench G3 (21,361 training instructions over 1,419 APIs), the 8B model scores 25.8% on ToolBench G2, 29.9% on Gorilla, 69.0% on CRAFT and 72.8% on TaskBench, with no further training. Swapping BM25 for the dense retriever COLT at inference improved the success rate at every model size. But the picture has limits: on G2 the 8B model drops to 25.8%, well below its in-domain 41.5%, and on TaskBench a prompted Llama-4-Maverick with a Search-R1-style flow scored 80.6%, above the trained model's 72.8%. The authors read this as the Plan stage needing some domain knowledge. The method also assumes each substep needs exactly one tool and that substeps do not depend on each other's output, so it does not cover tool chains where step two needs the result of step one.
Goes deeper in: AI Agents → Context Engineering → Tool-Loading Strategy