vLLM vs SGLang benchmark gap — Chunked-prefill confound — What does it mean?

The news. On November 4, 2026, researchers at Thompson Rivers University posted a controlled comparison of vLLM and SGLang on a Kubernetes testbed, using seeded multi-agent workloads. Under vendor defaults, vLLM's median TTFT was up to 64× SGLang's. Matching chunked prefill on both engines removed 78% of the capacity gap where it was largest and 30 to 67% of the latency gap. Read the paper →

Picture two kitchens that get the same night of orders, and one banquet order that is huge. Kitchen A cooks the banquet in one uninterrupted block, so every small ticket behind it waits. Kitchen B cooks the banquet in slices and squeezes small tickets in between. If Kitchen B serves small tickets faster, you could say it has the better recipe cards. But the cards may not be the reason. The slicing alone can explain much of the difference.

That is the situation with vLLM and SGLang. Both reuse cached state for prompts that repeat, which prefix caching makes cheap, but they do it differently: vLLM reuses fixed-size blocks, and SGLang keeps a radix tree. Benchmarks of default deployments show a large gap, usually credited to this cache design. The paper points out a second difference that has nothing to do with caching. Each engine's defaults decide by themselves whether to chunk long prefills, based on the hardware and the release. In the versions tested, vLLM 0.6.3 turned chunking on only above a 32,768-token context, and SGLang 0.4.1 turned it on for any GPU with less than 25 GB of memory. On the authors' L4 GPU, SGLang chunked and vLLM did not. Neither engine was misconfigured. The comparison simply changed two things at once.

The authors then separated the two. They ran each engine under vendor defaults, and again with chunked prefill matched at a 2,048-token budget. Only vLLM's setting changed. SGLang ran the same configuration in both arms, so it served as a control: rebuilt on an independently provisioned cluster days later, its latencies agreed within 0.5 to 11.7%. That agreement is what lets the authors say the change in the gap came from vLLM's setting and not from noisy infrastructure.

There is a second trap, and it explains why a small leftover gap can still look huge. Near saturation, TTFT measures the length of a queue, not the speed of an engine. At the matched load of 1.68 requests per second, SGLang completed 1.66 and kept pace. vLLM completed 1.58, and the paper reports a 4.4% throughput shortfall (the printed rates are rounded), so it fell behind and its backlog grew for as long as requests kept arriving. A fluid queue model with no free parameter, driven only by that shortfall, reproduces the measured latency gap. The model overestimated vLLM's median TTFT by a factor of 1.6 on average, because agent requests that wait also delay the later requests that depend on them. The direction and the scaling still held. Read more about this regime in Capacity Planning & Saturation.

Here is a small worked example with illustrative numbers. Say 100 requests per second arrive and Engine X serves 95.6 per second, a 4.4% shortfall. The backlog grows by 4.4 requests per second. In a 600-second run, a request arriving at the midpoint (second 300) finds a backlog of about 1,320 requests and waits about 1,320 / 95.6 ≈ 13.8 seconds. In a 300-second run the same engine shows about 6.9 seconds, and in a 1,200-second run about 27.6 seconds. The engine did not change. Only the length of the benchmark did, which matches the paper's statement that the same 4.4% shortfall gives a gap about half as large in a 300 s run and about twice as large in a 1,200 s run. A TTFT ratio taken there depends on the engine and on the run length together, so it cannot be compared across studies unless throughput is reported next to it.

Not everything disappears after matching. At equal throughput, SGLang kept a 2.4 to 6.4× advantage at the 95th percentile of TTFT. The authors read this as consistent with cache-aware ordering: SGLang admits first the waiting request with the longest cached prefix, which is cheapest to prefill, while vLLM serves in arrival order. They do not isolate it as the cause, and they note that the residual is not attributed. The honest reading is that the cache design may matter, but the default-versus-default gap is an upper bound on how much, since it also contains the chunking effect. The tail is also why the paper asks for the median and the 95th percentile, as covered in Percentiles & Tail Latency.

The paper turns this into six habits for comparing engines: record the configuration each engine actually resolved (read the start-up log, not the flags you passed); change one setting at a time and keep a control; report throughput beside latency; state the run length when any engine is saturated; report the median and the 95th percentile; and pair runs by seed. The limits matter too: one GPU, one model, one context length, three repetitions, and a synthetic workload. Newer vLLM versions enable chunked prefill by default wherever possible, so this exact pairing has since changed upstream. The lesson about defaults that move is what lasts.

Goes deeper in: LLM Serving → Serving Metrics & SLOs → Capacity Planning & Saturation