Meta's Cap-and-Track paper — Refusal-driven capacity tracking — What does it mean?
The news. On November 18, 2026, Meta engineers posted "Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity" (arXiv 2611.24891). It describes Meta's traffic shaper, the internal dispatch service for asynchronous LLM inference behind post and live-video understanding, which carries more than 300 million online requests a day. In a same-traffic production A/B test, the shaper served a classifier on 39 H100s against its precursor's 68 — 43% fewer — with coverage and scores at parity, though lower-priority work waited longer at the peaks of backfills, bulk jobs that reprocess existing content. Read the paper →
Picture a downtown garage on event night, with an overflow lot down the street. The old rule is an attendant who releases one car every ten seconds, a pace someone chose on a quiet evening. If a level closes, cars still arrive on schedule, drive in, and circle the ramps looking for spots that are not there. If an extra level opens, spots sit empty while cars wait in the lot. A fixed release rate is correct only for the garage it was tuned on, and the garage changes every night. LLM serving has the same problem on both sides. Supply moves: fleets resize, replicas are preempted, and shared provider pools congest. Demand moves too, because the work behind one request depends on an output length nobody knows at admission — the scheduler only learns it as tokens come out. Each of the paper's precursor systems had one hand-set global rate per job type, and engineers retuned those rates by hand, again and again.
The first fix is the FULL sign at the door. By default, vLLM accepts requests beyond its running batch and holds them in a waiting queue, and a model server that blocks callers adds a second queue in memory. Those are the circling cars: each one holds memory and an open connection while its timeout runs, with no priority order across the fleet and no redelivery if it is dropped. Cap replaces that inside wait with an immediate refusal: at its limit, the replica answers HTTP 429, and the request goes back to wait outside the engine, where no timeout clock is running. Meta set vLLM's batch limit (max_num_seqs) to 128 and the model server's cap to 140 per replica (one engine on one H100 or two MI300X), so a few requests wait just behind a full batch and a freed slot refills without a round trip to the dispatcher. Because each replica carries its own cap, the fleet's total admission is the sum over live replicas, and it rises and falls with the fleet with no reconfiguration.
Why does the place of waiting matter so much? Little's law says the work in flight equals the arrival rate times the time each item spends in the system. The paper takes 5,000 requests per second, about its busiest hourly rate on a typical day: if requests may wait out a 5-minute timeout in the serving path, up to about 1.5 million are in flight, against about 65,000 for a 13-second inference. In a replay at about twice a replica's capacity with no cap, the engine stayed busy, but goodput fell to zero and never recovered: once queued requests outlived their timeout, the engine spent its time on requests whose callers had already left. With the cap, goodput held steady. In production, the uncapped classifier's per-minute p99 latency, averaged hourly, climbed from 5 s to 511 s; after the caps went on, the next six days had no timeouts. A busy engine and a useful one are different measurements — throughput and goodput:
Now the attendant. Track sets each queue's dispatch rate from one number — the refusal share — and never predicts output length or reads an engine metric. Every 15 seconds a leader worker computes that share across the fleet. At or below the 2% target, it raises the rate by 3%. Above the target it cuts in three tiers: ×0.99 for shares up to 50%, ×0.85 up to 65%, and ×0.5 above that. Both directions are multiplicative (MIMD). The authors began with AIMD, the rule classic TCP congestion control uses, but its fixed step overshot low-rate queues and climbed too slowly for high-rate ones; with one controller per capacity domain, Track also needs none of AIMD's fairness between senders. One 3% raise takes about three 0.99 cuts to undo, so the rate traces a small sawtooth around the ceiling, and the refusal share settles a little above target — near 4–5% behind a provider's admission. That cost is small here in a way packet loss is not: a refused request returns fast, uses no reserved tokens, and stays queued.
| Dispatch setup (same caps, classifier replay) | Goodput per host, stable ceiling | Refusal share, stable ceiling | One of two replicas removed for 10 minutes |
|---|---|---|---|
| Fixed rate at 100% of measured capacity | 107.0 req/s, §6.2 | 1.1%, §6.2 | 49% of attempts refused, §6.2 |
| Fixed rate at 120% of measured capacity | 109.5 req/s, §6.2 | 15.5%, §6.2 | 58% of attempts refused, §6.2 |
| Fixed rate at the cut capacity | half of full capacity, §6.2 | not reported | fits the outage, then stays at half after the restore, §6.2 |
| Track, started at half capacity and told nothing else | 108.5 req/s, §6.2 | 3.8%, §6.2 | refusals fell as it descended; full capacity again within about 5 minutes of the restore, §6.2 |
Read the table as a trade. A well-tuned fixed rate can match Track while the ceiling holds — the 120% rate even served slightly more — but it does not move when the ceiling moves. During the outage the 100% and 120% rates were refused on about half of their attempts or more, and refused attempts are not free: they still load the gateway, the network and the admission tier, and staged media waits until a retry succeeds. Track paid 3.8% refusals at the stable ceiling for the probing that let it find the lower ceiling during the outage and the full one again afterward.
Here is the arithmetic, using the paper's constants and an illustrative ceiling of 1,000 requests per second. Suppose the fleet loses 40% of its capacity, so the ceiling falls to 600. A fixed rate of 1,000 now draws 400 refusals every second — a 40% refusal share — until a person retunes it. Track sees the same 40% share, which is in the mild ×0.99 tier, so it needs 0.99n ≈ 0.6, or about 51 decisions — about 13 minutes at one decision per 15 s — to reach the new ceiling; the paper names this moderate drop as the case where Track responds slowest. Now suppose the ceiling doubles to 2,000. The fixed rate leaves 1,000 requests per second of capacity idle. Track sees almost no refusals and climbs 3% per decision: 1.03n ≈ 2 needs about 24 decisions, about 6 minutes (our arithmetic from the paper's constants). In both directions the fixed rate stays wrong until someone notices; the controller is wrong for minutes and then right again.
The design has stated limits. It assumes work that can wait — no user is waiting on these requests — and backends that refuse promptly, with one controller per capacity domain; the authors have not applied it to interactive or agent-session traffic. The smaller fleet traded freshness at peaks: on the two peak days of a four-day classifier backfill, the lowest priority band waited a median 6.6 hours in the classifier stage against the precursor's half hour. And a quota that meters tokens at response time lags the load it counts, which makes Track's rate swing. The idea outlasts this paper: when you cannot measure a backend's capacity directly, let the backend refuse early and steer by the refusals, the way a network sender steers by packet loss.
Goes deeper in: LLM Serving → Serving Metrics & SLOs → Capacity Planning & Saturation
Related explainers
- Bursts That Outlast Themselves — Prefix-Cache Metastability — another serving system that recovers by admitting less work, not by working harder
- DLFP in vLLM — Decode-latency feedback for chunked prefill — a feedback controller inside the engine, and what breaks when its signal is not the real bottleneck
- Task-aware CPU admission vs uniform resource allocation — admission control for agent tool calls, sized by the resource that actually saturates