DolphinRouting — Local-vs-remote prefill routing — What does it mean?

The news. On October 29, 2026, researchers at Nanjing University posted DolphinRouting on arXiv. On a real prefill/decode-disaggregated cluster serving Qwen3.8-72B, it reportedly cuts P90 time-to-first-token by 5.7–7.9% and P90 time-per-output-token by 2.6–10.1% against NVIDIA Dynamo's routing on the mixed and real-trace workloads. Read the paper →

Picture the triage desk of a hospital with a ward and a specialist wing down the hall. Every arriving patient can be treated in the ward, or sent to the specialist wing and then brought back to the ward. The second route is faster for a complicated case and wasteful for a sprained ankle, because the walk and the transfer of the chart cost time too. The answer also depends on the chart: a patient whose records are already on file in the ward needs far less work than the same patient arriving cold.

Prefill/decode disaggregation leaves exactly this question open on every request. The prefill pool and the decode pool are separate GPU groups. A request can take the aggregated path, where the decode worker runs prefill itself and then decodes, or the disaggregated path, where a prefill worker reads the prompt and ships the KV cache to a decode worker. DolphinRouting is the component that picks the path, and the paper's central claim is that the pick should be learned per group of requests from measured results, not computed per request from a formula.

The paper reports why the pick matters. Using numbers from the TaiChi study that it cites (a simulated ArXiv workload at 12 requests per second), with targets of 5 s TTFT and 250 ms TPOT the aggregated layout meets the SLO for 97% of requests and the disaggregated layout for 42%. With targets of 16 s and 60 ms the two flip, to 7% and 98%. Neither layout wins everywhere, so a router has to choose per request.

Each request is described by four numbers: the raw prompt length, the prompt length left after subtracting the cached prefix on the prefill worker, the same on the decode worker, and the expected output length. The two effective lengths are what each side would really have to compute, because prefix caching lets a worker skip tokens whose KV it already holds. They differ because the pools hold different histories: a decode worker that served an earlier turn of a conversation may already hold most of the new prompt. Output length is unknown at admission, so the paper uses a small CPU classifier that sorts the answer into short, medium or long and maps each class to a representative length.

On the desk, the effective lengths are how thick the unwritten part of the chart is in the ward versus in the specialist wing.

Scoring every request against a cost model on arrival would put work on the hot path, so the paper moves the thinking offline. Historical requests are clustered (the paper tests k-means and a Gaussian mixture) after a log-and-standardize step, and each of K clusters gets exactly one bit: 0 for local prefill, 1 for remote prefill. At runtime the router extracts the four features, finds the cluster and reads one table entry. That is the laminated card: patient group on the left, ward or wing on the right.

The bits come from measurement. A cluster's reward is the share of its requests that finish inside their end-to-end SLO, where end-to-end time is TTFT plus (output tokens − 1) × TPOT. Search starts with per-cluster scanning: hold the other clusters fixed, send one cluster down the aggregated path for a period, then down the disaggregated path for the same period, and keep the better one. That takes 2K scans, so 16 for K = 8. It then refines the whole table with epsilon-greedy search, because clusters share workers and caches and one cluster's best action depends on what the others do. All of it is measured under saturated load on the real system, and the table is learned separately for each prefill/decode split.

Here is how the features separate requests, using the paper's workload averages (illustrative; the cache placement is assumed). Three requests arrive in the same second at one fixed prefill/decode split. Request A is an 8,088-token paper to summarize (the ArXiv workload average) with about 0% cached, so both effective lengths stay at 8,088 and a local prefill would hold the decode worker for a full long pass. Request B is a 19,019-token document (the L-Eval average) with over 80% cache hits; if the decode worker holds that prefix, its decode-side effective length is at most about 19,019 × 0.2 ≈ 3,800 tokens. Request C is a 198-token prompt (the HumanEval average), cold on both sides, where the transfer overhead could plausibly outweigh any gain from moving it.

The longest prompt, B at 19,019 tokens, can be the cheapest to prefill locally, while the shorter A at 8,088 tokens is the expensive one. A rule on raw length gets that backwards. Clustering on both effective lengths puts A, B and C in different groups, and the offline search is then free to learn remote for A's group and local for B's and C's, if the measurements say so.

The paper's headline table shows where this helps and where it does not. Values are the paper's normalized P90 numbers, where lower is better, so compare the Dynamo and DolphinRouting columns within a row. Each entry is TTFT / TPOT.

The gains appear on mixed traffic, where request types genuinely differ, and vanish on single-type workloads. On Hybrid the TTFT drop is 7.5–7.9% and the TPOT drop 6.5–10.1% across the two splits; on the real trace it is 5.7–6.5% and 2.6–5.5%. On the three pure benchmarks the router is level with Dynamo or slightly behind, which is what you would expect when one answer fits every request. All results come from the authors' own cluster (4 H20 GPUs per node, one 72B model), and this is a preprint, so treat the percentages as one team's measurement.

The ablations show what carries the result. Going from K = 8 to K = 16 roughly triples the search rounds, from about 26 to about 76, while the metrics barely move; K = 2 or 4 converges faster but is about 9–20% worse. Skipping the scan initialization raises rounds from 25.8 to 39.6 and TPOT from 0.80 to 0.91. Dropping the output-length feature hurts too: under the 3 + 1 split, TTFT rises from 0.37 to 0.42 and TPOT from 0.80 to 0.93, which is why a coarse answer-length guess earns its place even though it is imperfect. The cost is operational: the table must be re-measured under saturated load whenever the prefill/decode split changes.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Full Disaggregation

Related explainers

• NVIDIA Dynamo — Encode-prefill-decode disaggregation — adds a third pool for image encoding; DolphinRouting instead decides which of two pools should prefill.
• AMD ATOM + ATOMesh — Prefill/decode disaggregation on ROCm — the two-pool layout that a router like this sits in front of.