TokenRouter paper — Delayed batching for token-level LLM routing — What does it mean?

The news. On October 8, 2026, researchers from Tsinghua University and Carnegie Mellon University posted TokenRouter (arXiv 2610.12242). It runs existing token-level routing algorithms — R2R, CITER, Co-LLM, R-Stitch and a three-model ensemble — on one serving runtime instead of each algorithm's own research code. Across 15 algorithm and workload combinations on an 8×A100 server, it reports 2.01–64.15× higher decoding throughput than the stronger baseline in each case. Read the paper →

TokenRouter is a serving system that lets several LLMs take turns writing one answer without forcing them to move in lockstep. Picture a kitchen with a fast line cook and a slow head chef. Most steps of every dish are simple, so the line cook does them. A few steps are hard, so the line cook hands that ticket to the chef. This is what token-level routing does: a small model writes the easy tokens and a large model writes the hard ones. The payoff is real — the paper notes that the R2R algorithm can match a 32B model's quality while sending only about 5% of tokens to the 32B model, where the paper's query-level comparison sends around 40% of whole queries to it.

Now run that kitchen the way a normal inference engine runs. An engine like vLLM or SGLang keeps every request in the batch on the same decoding step: the step starts, every request gets one token, the step ends. With two cooks, a shared step must wait for the slower head chef to finish, leaving the line cook idle. The paper calls this step desynchronization. The other obvious fix — let only one cook work at a time — is no better: every dish waiting on the idle cook stalls. This is also why speculative decoding does not transfer: there the small and large models alternate in a fixed rhythm, while routing calls the large model at random steps.

TokenRouter's first fix is to give each cook their own station. Each model runs in its own subserver with its own scheduler and decoding loop, and a third loop passes requests between subservers. When a token is routed to the large model, the request leaves the small model's batch and is marked pending — like a dish parked on the rail with its prep still on the board. The scheduler skips it, but its KV cache and prefix state stay in place. When the large model returns its token, the request switches back to running and rejoins the batch as if it had never left, so a model switch costs little more than appending one token instead of a new prefix match and fresh cache allocation. Developers still write the routing rule as if following one request through the kitchen — route, send, receive — and the runtime handles the stations.

Separate stations create a second problem. Hard tickets reach the head chef one at a time and at irregular moments. If the chef starts a pan the instant any ticket arrives, the next ticket that shows up a moment later must wait for that whole pan to finish. The paper calls this wait batch admission delay, and with frequent switches it happens over and over, leaving the chef cooking many small, fragmented batches. Delayed batching fixes it by letting the chef wait until B tickets are on the rail, then cooking them in one pan. A wider pan costs the chef little extra time, because a decoding step on a large model is mostly the time to stream its weights from memory, and that cost is shared by every request in the step.

The waiting is not free, so the threshold matters. Too small a B brings back the admission delay; too large a B keeps requests queued at the chef while the line cook runs out of work. TokenRouter models the routing process as a discrete-time Markov chain, using the number of concurrent requests, how often each model is chosen, and each model's step time, and picks the B that maximizes predicted throughput. The model assumes the number of tokens between two hand-offs follows a geometric distribution; the authors say this holds for most token-level routing algorithms and leave the exceptions to future work.

Here is the tradeoff with small numbers (illustrative, not from the paper). Say a large-model step takes 4 time units whether it serves one request or two, ticket A arrives at time 0 and ticket B at time 1. Eager: A starts at 0 and finishes at 4; B waits until 4 and finishes at 8 — waits of 0 and 3, average 1.5. Delayed with B = 2: the chef waits for the second ticket, starts at 1, and finishes both at 5 — waits of 1 and 0, average 0.5. A finishes one unit later, B finishes three units sooner. The paper's own breakdown shows the parts adding up (the R2R algorithm, 8 concurrent requests): the official R2R code reaches 134.9 tokens/s; engineering work such as extended CUDA graphs (recording GPU kernel launches once and replaying them) lifts that to 230.79 (1.71×); asynchronous subservers to 296.86 (×1.29 more); delayed batching to 372.48 (×1.25 more). 1.71 × 1.29 × 1.25 ≈ 2.76× in total, and delayed batching alone adds about 25% on top of the asynchronous version. With a single request, the same table shows delayed batching adding nothing (73.02 tokens/s with or without it): with one dish in the kitchen, there is no second ticket to wait for. Figure 1a and Table 6

The idea outlives this paper: when the unit of work is a single token that can move between models, the batch is no longer something the engine gets for free, and the scheduler has to build it on purpose. It is the same throughput-versus-latency trade every serving system makes, moved down to each model's queue. Limits worth knowing: all results are on one 8×A100 server, mostly with Qwen3 model pairs; the large-model wait adds latency to the first hand-off of a quiet period, so a strict per-token deadline may want a smaller B; and quality is set by the routing algorithm, not by TokenRouter, which only serves it.

Goes deeper in: LLM Internals → Batching → Continuous Batching