SynthAPI is a pipeline that uses an LLM to write small Python apps (a database schema plus tool functions) and then tasks with verifiers on top of them. It exposes controls for the number of apps, tasks, database states and distractor data, so researchers can measure which kind of variety helps tool-use agents most.

Does more diversity always beat more tasks?

No. In the paper, fewer apps worked better at small budgets (64–512 tasks), and 64 apps became clearly best from 4,096 tasks. The advice applies as budgets grow, not at every size.

What do distractor data and database states add?

Database states vary the starting data for the same instruction, and distractor data adds irrelevant rows. Both force the agent to search and handle data instead of pattern-matching one setup, and the paper found higher values helped at high budgets.

How does this relate to tool design for agents?

Tool design decides what an agent can call; this work shows how varied the tools in training should be. Both depend on tools that are clear and consistent, which the Tool Design step of the AI Agents track covers.

Agent· 2026-11-01

Apple SynthAPI — Tool diversity before task count — What does it mean?

Jargon

SynthAPI: Apple's pipeline that lets an LLM write small Python "apps" (tools plus a database), then writes tasks on top of them, so researchers can control how varied the training environments are.
Tool-use agent: A language model that finishes a goal by calling functions (tools) over several steps and reading their results. See Why Tools?.
Scaling law: A fitted formula that predicts model performance from the size and make-up of the training data, so you can forecast a bigger run from smaller ones.
Verifier: A short program that reads the database before and after an episode and answers yes or no: was the task really done? It replaces a human or LLM judge.
Oracle solver: A program that solves the task using privileged access to the whole database. If it cannot pass the verifier, the task is thrown away as broken.
Distractor data: Extra rows added to the database that the task does not need, so the agent must search and filter instead of finding exactly one matching row.
SFT (supervised fine-tuning): Training a student model to copy the next token of expert trajectories. The paper uses SFT, not reinforcement learning, for its many experiments because RL for every setting would be too expensive.
Extrapolation: Fitting the formula on cheaper runs and then checking it on runs it never saw. Here the held-out runs used up to 16 times the largest compute budget used for fitting.

The news. On October 27, 2026, an Apple research team published "Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis". They generated 458,752 tasks across 64 synthetic apps, trained 849 models of four sizes, and fitted a law that predicts tool-use performance from how the data is composed, not only how much of it there is. Read the paper →

Picture learning to drive. Ten thousand trips through the same small town teach you that town very well and little else. Three trips each in ten different cities teach you what a one-way street, a roundabout and a detour look like in general. SynthAPI turns that intuition into dials, so it can be measured instead of guessed.

To build a city, a "builder" LLM (Gemini-3.1 Pro in the paper) writes an app: a short description, a database schema in db.py, and a set of Python functions in apis.py that read and change that database. Those functions become the tools the agent can call, like search products or create an order. Then it builds trips: each task is a tuple of an instruction, a verifier, a starting database and an oracle solver. A task is kept only if the verifier fails on the starting database and passes after the oracle solver runs. The agent itself sees only a few meta-tools (list_apps, list_tools, get_tool_documentation, complete_task), so it learns to discover tools instead of memorizing them. This is the same discipline as tool design for agents, seen from the training side.

The paper then turns four dials, one at a time. The number of tasks is the product N = A · T · S, while D changes how hard each task is without changing N.

Dial	What it changes	What the paper found (9B student)
A apps	How many different tool sets (cities)	Fewer apps win at 64–512 tasks; 64 apps is clearly best from 4,096 tasks (paper §4.2)
T tasks per app	How many trips inside one city	Sets how fast performance climbs toward the ceiling, not the ceiling itself (paper §4.3)
S database states per task	Same instruction, different starting data	1 state is best at 32–512 instances, 2 at 512–4,096, 4 beyond that (paper §4.2)
D distractor batches	How much irrelevant data surrounds the core data	D=4 clearly beats D=1 and D=2 at high budgets (paper §4.2)

The result is a crossover, not a slogan. With a small budget, concentrating on a few apps learns faster, like getting to know one town. As the budget grows, the benefit flips and wider tools and database variety raise how high performance can go. The paper's fitted law puts the ceiling in terms of A·S·D and leaves T to control only the speed of approach. The authors also note that the distractor result may partly reflect today's benchmarks being not very sensitive to hard database states.

A worked example from the paper's comparison shows the composition effect at a fixed size. Both runs use the same 32,768 tasks. The sub-optimal recipe sets A=32, T=1024, S=1, D=2, so 32 × 1024 × 1 = 32,768. The optimal recipe sets A=64, T=128, S=4, D=4, so 64 × 128 × 4 = 32,768. Using the paper's median of 144 APIs per app (illustrative), that is roughly 4,600 distinct tools against roughly 9,200, with each instruction seen on 4 database states and more distractors. Same task count, double the apps, and the diverse recipe consistently outperformed the task-heavy recipe in the paper's benchmark comparison.

The fitted law is useful because it predicts. Fitted on the cheaper runs and tested on the top 25% of training compute (an extrapolation to up to 16 times the largest compute budget used for fitting), the composition-aware formula had a mean absolute error of 0.0285, against 0.1706 for a formula that only counts tasks. Composition, not only volume, is what the forecast needs. Three limits to keep in mind: the experiments use SFT on expert trajectories rather than RL, all tasks use a single app, and the comparisons with other published methods mix different training regimes, so they do not isolate the synthesis method itself.

Goes deeper in: AI Agents → Tool Use & Function Calling → Tool Design — ACI + Anti-Patterns

Related explainers

Qwen-AgentWorld trains a language model as a world model for RL agents — another way to build the environments agents train in

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based