Crossflow — Revocable decode-node leases — What does it mean?

The news. On September 22, 2026, a team at Meta Platforms posted Crossflow: Prefill–Decode Elasticity for Agentic LLM Serving on arXiv. They measured a large production fleet and a public agent trace, found that the balance between prompt work and output work moves much faster than a cluster can be re-split, and built Crossflow inside SGLang to let decode machines temporarily take prompt work. They evaluate it on GPT-OSS-120B and GLM-5.2 running on NVIDIA GB300 servers. Read the paper →

Picture the kitchen at the start of a shift. The manager has put two people on prep and four on the line, and moving a cook to prep means changing stations, setting up knives and warming a new board, which takes long enough that the rush is over by the time they are ready. The staffing split is fixed, but the orders are not: some hours bring many new tables (lots of prep), other hours bring tables that are already eating (lots of line work). An LLM cluster has the same shape. Prefill reads each new prompt in one large pass, decode then produces tokens one step at a time, and disaggregation gives each phase its own machines so that long prompt passes do not slow down the token stream.

The paper measures how fast the mix moves. It tracks the ratio of uncached input tokens to output tokens (cached prefix tokens do not count, because prefix caching skips them). In the production fleet this ratio reached a peak-to-mean of 4.7× at minute scale. In a public agent trace over 134 days, the hourly ratio swung by a median 24.5× within a single day, with only a weak time-of-day pattern, so an operator cannot plan a re-split in advance. Agent traffic makes this worse: tool results release several dependent requests at once, and cached prefixes change how much prompt work each turn really needs.

Back in the kitchen, the fix is not to move a cook. When prep falls behind, a line cook whose own pans are under control writes a slip: "I can take a little prep work, and this offer expires soon." The expediter reads the slips, reserves one, and only then sends a ticket to that cook. The cook keeps the ticket only while the pans allow it, and the next slip can offer less, or nothing.

The lease is where the mechanism lives. A decode node cannot judge its spare capacity from GPU utilization, so it publishes a lease computed from its live state, and the lease expires on its own. The paper's testbed shows why utilization fails: making the decode batch 2.5× larger raised compute activity by 1.44× but decode throughput by 1.74×, while stretching context from 1K to 82K tokens at batch 8 nearly doubled memory traffic and cut decode throughput by about 10%. The spare room depends on the shape of the work on both sides, which is the roofline point that prefill and decode stress different resources. So each lease carries separate credits for compute, KV-cache space, KV transfer, projected output and borrowed-prompt slots, with a version number. A newer lease replaces the older one, so unused credit never builds up.

Three conditions must all hold before a prompt leaves the normal path. Demand: the prefill pool's estimated wait (outstanding uncached prompt tokens divided by the recent processing rate) is above a threshold. Benefit: the predicted wait plus service time on the decode node is strictly shorter than the normal path's prefill wait, prefill time and KV transfer time. Safety: the node's lease has room, and a cluster-wide borrowing limit keeps some decode capacity free for requests already coming from the prefill pool. If a node reports no valid state, or its latency crosses the SLO, its next lease is zero, and the system falls back to ordinary static P/D rather than guessing.

A worked example with the paper's sizing formula, with the numbers invented to keep the arithmetic simple (illustrative). The share of capacity that prefill should hold is f = ρR / (1 + ρR), where R is the uncached-input-to-output ratio and ρ is decode's per-token throughput divided by prefill's (decode is slower, so ρ is below 1). Hold ρ = 0.1 and suppose R has a 5th percentile of 5, a median of 10 and a 95th percentile of 20. Then f is 33.3% at the 5th percentile, 50% at the median and 66.7% at the 95th. A static cluster that sizes each pool for its own 95th percentile buys 66.7% for prefill plus 100% − 33.3% = 66.7% for decode, which is 133.3% of what the traffic needs at any moment, so 33.3 points always sit idle. With leases, prefill is sized at its median (50%) and decode keeps its 66.7%. At the prefill peak, prefill is short by 66.7 − 50 = 16.7 points while decode needs only 33.3% and has 33.3 points free, so the lent capacity covers the shortfall. Total purchase falls from 133.3% to 116.7%, and the idle share is cut in half, from 33.3 to 16.7 points. On the real fleet's four use cases, the paper reports this asymmetric sizing lowers stranded capacity from 11–18% to 6.0–7.4%.

In the matched 2P4D sweeps, Crossflow increased token throughput and reduced mean TTFT, but its effect on mean ITL depended on the model and trace. Borrowed prompt work runs next to live decode batches, and on GLM-5.2 with the public trace mean ITL rose by 20.9–34.3% even though TTFT fell. Decode headroom mattered in the GPT-OSS-120B robustness test at 16× load: on the public trace, 4P2D reversed the input-throughput and mean-TTFT gains, while on the internal trace Crossflow improved both token-throughput measures at every tested split. The operational change is the more durable lesson. Because decode slack now covers prefill peaks, an operator can plan capacity for prefill near its median instead of its peak and can keep an existing split when the workload drifts. In the GPT-OSS-120B public-trace test at 16× load, Crossflow at 2P4D beat the best static split by 25.1% in input-token throughput and 30.0% in output-token throughput. That is a different answer to the when to use what question than choosing between chunked prefill on one pool and a fixed two-pool split.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Full Disaggregation

Related explainers

• AMD ATOM: prefill/decode disaggregation — the basic two-pool split that Crossflow keeps and makes elastic.
• Reclaiming the prefill reserve — lending memory inside one engine, where Crossflow lends compute across machines.
• NVIDIA Dynamo: encode-prefill-decode disaggregation — splitting the pipeline further, which makes the fixed-ratio problem sharper.