What is Decode-Latency Feedback Prefill (DLFP)?

DLFP is a **model-free controller** in the vLLM V1 scheduler that caps prefill chunks only when they share an iteration with active decodes. After each guarded cycle it scales the cap by target ÷ observed interval (target 80 ms), rounds down to 128 tokens and clamps it between 1,024 and 6,144.

Why does the prefill chunk size matter for latency?

Every request that is already decoding waits for the whole iteration, including any prefill work in it. A big chunk stalls all of their next tokens (higher ITL); a small chunk needs more iterations to finish the prompt (higher TTFT). The chunk size decides who pays the latency.

Why did DLFP fail on larger models?

Its measurement was the interval between CPU scheduler calls, not the time the GPU finished the iteration. Under asynchronous scheduling, larger kernels and tensor-parallel sync, those two times diverge, so the controller could grow the chunk while the GPU was still busy. On Qwen3-8B at 0.10 req/s, P99 ITL rose from 19 ms to 281 ms.

How does DLFP relate to fixed chunked prefill and disaggregation?

Fixed [chunked prefill](/tracks/llm-serving/prefill-decode-disaggregation?step=chunked-prefill) uses one hand-tuned cap; DLFP adjusts the cap from observed time. Disaggregation removes the interference by running prefill and decode on separate GPUs, which is not available on a single device — the setting the authors target next.

LLM· 2026-10-02

DLFP in vLLM — Decode-latency feedback for chunked prefill — What does it mean?

TL;DR

What is it: Decode-Latency Feedback Prefill (DLFP), from a September 29, 2026 arXiv paper, is a feedback controller for chunked prefill in vLLM: it times the last scheduling cycle and resizes the cap on the next prompt chunk to reduce how much a new prompt delays the requests already generating tokens.
Why it’s needed: In one GPU batch, a long new prompt and the users already streaming tokens share every iteration, so the prefill chunk size is the knob that trades time to first token against the smoothness of every other user's stream — and the best setting moves with model, hardware and load.
vs previous: Instead of a fixed chunk size tuned by hand, or an offline latency model that predicts it, DLFP closes the loop on observed time — and the paper shows the catch: the time it observes is the host scheduler's call interval, not GPU completion, so the controller helps a 0.6B model and fails on 8B, 32B and two-GPU runs.

Jargon

Prefill: The first phase of a request: the model processes the prompt tokens and stores their keys and values in the KV cache. It has a lot of parallel work, so a long prompt in one iteration makes that iteration slow; with chunked prefill the work is spread over several iterations.
Decode: The second phase: the model generates one token per request per iteration. Each user sees the next token only after the whole iteration finishes.
Chunked prefill: Splitting a long prompt's prefill across several iterations, with a cap on prompt tokens per iteration, so it can share batches with decode work. The cap is usually a fixed setting; DLFP makes it adaptive. See Chunked Prefill.
TTFT: Time to first token: how long a user waits from sending the prompt until the first output token. Small prefill chunks make it longer, because the prompt needs more iterations.
ITL: Inter-token latency: the gap between two consecutive output tokens of one request. A large prefill chunk in the same iteration makes this gap jump for every decoding user.
P99: The 99th percentile: the value that 99% of observations stay under. It measures the tail, which is what users notice as a stall. See Percentiles & Tail Latency.
Proportional controller: A feedback rule that scales its output by the ratio of target to measurement: twice too slow means half the chunk. It needs no model of the system, only an accurate measurement.
Asynchronous scheduling: The CPU schedules iteration N+1 while the GPU still runs iteration N. It hides CPU overhead, but it also means a CPU-side timestamp no longer marks when GPU work finished.

The news. On September 29, 2026, Gaurav Agarwal, Ashish Garg and Isha Singhal posted Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits. They patched the vLLM V1 scheduler (vLLM 0.17.1) so that prefill chunks that share an iteration with active decodes are capped by a proportional controller, and evaluated it on Qwen3 models on A100 80 GB GPUs with open-loop Poisson arrivals and exact token accounting. The paper reports a clear win on Qwen3-0.6B on one GPU, and it reports, with equal weight, that the same mechanism fails on Qwen3-8B, Qwen3-32B and a two-GPU tensor-parallel setup. Read the paper →

Picture the ferry. Every trip carries the same commuters, and each commuter gets one stop closer to home per trip. When a truck of freight arrives, the clerk can load all of it onto the next trip, and then the trip is slow and every commuter on board waits for the freight. That is the interference problem: in continuous batching, one iteration mixes decode tokens from many users with prefill tokens for a new prompt, and the paper writes the iteration time as roughly decode work + prefill work + overhead. Every decoding request observes that whole time before it gets its next token, so a large prefill creates a stall that all of them feel at once.

The standard answer is to split the freight into crates: fixed-cap chunked prefill admits at most a set number of prompt tokens per iteration. But in that baseline the crate size is a hand-tuned setting. Big crates finish the prompt in fewer trips but make each trip slow; small crates keep trips fast but add trips, so the new user waits longer for the first token. The chunk size does not remove latency; it chooses who pays it — the new user through TTFT, or everyone already streaming through ITL. The paper's own fixed-cap controls show how fragile the hand-tuned setting is: a 2,048-token cap reduced SLO compliance to 287 of 300 requests, and a 4,096-token cap did not reliably clear the paper's 10% ITL-reduction gate.

DLFP replaces the hand-tuned crate with a clerk holding a stopwatch. It acts only when at least one request is already decoding — an isolated prefill, a ferry with no commuters, is never capped. After a guarded cycle it compares the observed interval with a target of 80 ms and sets the next cap to the current cap times target ÷ observed, rounded down to a multiple of 128 tokens and clamped between 1,024 and 6,144. A slow trip shrinks the next crate; a fast trip grows it. The rule needs no model of the GPU, the kernels or the batch — which is exactly why it is only as good as its stopwatch.

Here is the update the paper recorded in a live run, with the target fixed at 80 ms and the cap starting at its initial value of 3,072 tokens. The controller observed a 57.51 ms interval, faster than the target, so it multiplied: 3,072 × 80 ÷ 57.51 ≈ 4,273 tokens. Rounding down to a multiple of 128 gives 33 × 128 = 4,224 tokens — the exact value the paper logged, so the cap on the next prompt chunk grew by 37.5%. Run the same rule the other way (illustrative, same 80 ms target and 3,072-token starting cap): an observation of 120 ms gives 3,072 × 80 ÷ 120 = 2,048 tokens, and the cap shrinks by a third. On Qwen3-0.6B on one A100, this loop cut P99 inter-token latency in all three paired trials — for seed 801 from 122.22 ms to 91.95 ms — for a mean reduction of 27.7% (95% CI 21.0–34.3%), with all 76,800 paired token IDs identical. The price was on the other side of the trade: mean P99 TTFT rose 34.8%, still inside the declared SLO.

Configuration	P99 ITL, baseline → DLFP	Other effect	Source
Qwen3-0.6B, 1× A100, 1.0 req/s	−27.7% mean over 3 trials	P99 TTFT +34.8%; SLO passes 300/300	§5
Qwen3-0.6B, 2 GPUs (tensor parallel)	−12.7% mean, CI crosses zero	SLO passes 110 → 101 of 300	§6.1
Qwen3-8B, 0.10 req/s	19 ms → 281 ms	SLO passes 10 → 7 of 20	§6.2
Qwen3-8B, 0.25 req/s	1.763 s → 707 ms	P99 TTFT 12.33 → 14.81 s; SLO passes 2 → 0 of 20	§6.2
Qwen3-32B, 2 GPUs, 0.05 req/s	314 ms → 670 ms	SLO passes 5/12 both; energy per token +4.7%	§6.3

Now the stopwatch. The prototype does not time the ferry from dock to far shore; it times how often the ticket booth calls the next boarding — the interval between scheduler calls on the CPU. vLLM runs with asynchronous scheduling, so the CPU can call the next boarding while the GPU still has queued work from the last one. On the small model the two clocks happened to agree. With larger kernels, a deeper asynchronous queue and tensor-parallel synchronization, they drift apart, and the controller can see a short host interval, conclude the trip was fast, and grow the crate while the device is still busy. The paper calls the small-model success an accidental correlation. A feedback loop with a proxy sensor can steer confidently in the wrong direction. The authors also tried a completion-aware revision that updated the controller when the scheduler output returned and bounded each step; on Qwen3-32B under synchronous scheduling it still lost (P99 ITL 41.1 ms baseline versus 194.8 ms), which they read as evidence against simple retuning. Their proposed next step is a controller timed on real completion in runtimes that expose it — a design the paper motivates but has not yet tested. This is the same boundary the scheduler's two halves draw in vLLM: the budget is spent in schedule(), but the truth about the iteration only arrives when the output comes back.

There is a second, quieter lesson in the 8B row. At 0.10 req/s there was almost no interference, so the baseline P99 ITL was 19 ms; DLFP cut prompts into more pieces anyway and P99 ITL rose to 281 ms. The authors note that splitting one large stall into several moderate stalls can increase the fraction of token gaps that reach the global P99. One long pause hurts one gap per user; five medium pauses can each land in the tail. That is why the paper judges every configuration on goodput — requests that met TTFT, mean ITL and end-to-end limits together — and not on ITL alone: at 0.25 req/s the 8B run moved latency from ITL into TTFT and SLO passes went from 2 to 0.

The authors see the opportunity in concurrent edge inference on phones and CPU-only laptops, which they have not yet evaluated: a laptop that generates an answer while it indexes a document, or several agent branches on one GPU, where full prefill/decode disaggregation is not available. A synchronous CPU runtime would expose real completion time directly, which removes the faulty stopwatch. The paper is explicit that it has not tested phones or CPU laptops and that adaptive chunk sizing itself is prior art; its claim is this specific model-free controller and where it stops working.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Chunked Prefill

Related explainers

Reclaiming the prefill reserve: chunked prefill vs elastic KV cache — the memory side of the same chunk-size knob

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based