What is deferred multimodal prefill?

It splits the image or video tokens into an **important** group, prefilled before the first answer, and a **deferred** group, prefilled in chunks during decode and in the gap after the answer. Nothing is discarded, so later questions can still use every token.

Why does it lower time to first token?

First-turn TTFT depends on how many tokens are in the initial prefill. With 30% retention the first pass is much shorter. The paper reports up to 24.1% lower mean first-turn TTFT on MultiVerse with Qwen3-VL-8B in SGLang.

How is it different from hard token pruning?

Hard pruning drops unselected tokens permanently, so a later question about a pruned region cannot be answered well. Raincheck uses the same pruning method to choose the first group, then recovers the rest, regaining on average 77.7% and 60.4% of the lost quality on MultiVerse and ConvBench.

The deferred work needs spare compute. On long videos the paper reports mean TPOT up 19.93% and 14.92%, and larger decode batches leave less room, so the benefit is aimed at latency-first, small-batch serving.

LLM· 2026-10-31

Raincheck cuts first-turn TTFT up to 24.1% — Deferred multimodal prefill — What does it mean?

Jargon

MLLM: Multimodal large language model: an LLM that reads images or video as well as text. Each image patch or video frame region becomes many visual tokens.
Prefill: The first phase of serving, where the model reads the prompt in one parallel pass and fills the KV cache. Its length sets TTFT. In Raincheck the first prefill covers the selected visual tokens plus all text and control tokens. See Prefill/Decode Disaggregation.
Decode: The second phase, where the model writes the answer one token per step. At small batch sizes each step is limited by memory reads, so the arithmetic units sit partly idle.
TTFT and TPOT: Time to first token (wait before the answer starts) and time per output token (speed of each later word). Both are defined in Serving Metrics.
Hard pruning: Keeping only the highest-scoring tokens for prefill and permanently dropping the rest. It lowers TTFT but the dropped tokens can never be used by later questions.
Gated DeltaNet (GDN): A recurrent layer type used in hybrid models such as Qwen3.5. It squeezes everything read so far into one fixed-size state matrix instead of a growing KV cache.
Visibility mask: A rule saying which tokens each token may attend to. Raincheck uses separate masks for deferred chunks and for the decode token so that new input does not depend on generated text.

The news. On October 29, 2026, a team from Zhejiang University, Alibaba Group and collaborating universities posted Raincheck, a framework for deferred multimodal prefill. With 30% initial token retention on Qwen3-VL-8B and Qwen3.5-9B, they report recovering on average 77.7% (MultiVerse) and 60.4% (ConvBench) of the quality that hard pruning loses, and up to 24.1% lower mean first-turn TTFT in SGLang with under 0.5% higher mean TPOT. Read the paper →

Think of a kitchen that serves the first course the moment it is ready and cooks the rest while the diners eat. A video of 30 seconds at 720p becomes about 13,500 visual tokens in the Qwen3-VL input setup. Full prefill is a kitchen that refuses to serve anything until every dish on the menu is plated. Hard pruning is a kitchen that serves only the first course and then throws the other ingredients away, so the table cannot order dessert later. Raincheck keeps the ingredients: it prefills the important tokens, answers, and uses the idle burner to cook the deferred tokens during the meal.

Why is there an idle burner at all? In the small-batch, latency-first setting the paper targets, each decode step is limited by how fast weights are read from memory, not by arithmetic, so adding a few extra token rows to the same pass costs little (the paper measures a small TPOT rise, below). Raincheck packs one chunk of deferred tokens together with the one decode token, so the dense layers and feed-forward network read their weights once for both (the same idea as chunked prefill). Fixed-shape CUDA graphs cover 7, 15 or 31 deferred tokens plus one decode token, giving 8, 16 or 32 rows per step, and a per-step budget caps how much TPOT can grow. The order ticket matters too: source order is preserved inside the important group and inside the deferred group, which keeps the causal-attention change small.

The hard part is correctness. A deferred chunk belongs before text and generated tokens that are already processed, yet the input must not depend on the model's own output. For full-attention layers this is a visibility rule: a deferred chunk sees the prefix processed so far and itself, never the processed suffix, while the decode token sees everything. For Gated DeltaNet layers, which hold only a state matrix, the paper writes the effect of the suffix as an affine map of the incoming state, so inserting new tokens only needs the difference in the prefix state to be pushed through recorded suffix transitions instead of rerunning the sequence. The paper holds the recorded suffix factors fixed, adds a repair for the short-convolution boundary in Qwen3.5, and notes the result approximates a full rerun under the expanded input.

Policy	First-turn TTFT (ms)	Mean TPOT (ms)	Unselected tokens
Full prefill	169.94	6.1919	None; everything prefilled first
Hard pruning	133.17 (−21.64%)	6.1354	Dropped permanently
Raincheck	128.96 (−24.11%)	6.1996 (+0.13%)	Prefilled during decode and after the answer

Table: MultiVerse, Qwen3-VL-8B, 30% initial retention (paper Table 4). Values are measured by the authors on one SGLang setup, so treat them as setup-dependent. The paper reports that the paired Raincheck-minus-Hard TTFT interval includes zero, so the two are comparable on TTFT; the gain over Hard is in recovered quality, not speed.

Here is the schedule for a case like the paper's video example, with illustrative numbers (illustrative, not from a measured trace). Take 13,500 visual tokens and 30% retention: 4,050 tokens are prefilled first and 9,450 are deferred. With a chunk of 15 deferred tokens plus one decode token per step (16 rows), the deferred set needs 9,450 / 15 = 630 decode steps. If the answer is 512 tokens long, only 512 x 15 = 7,680 deferred tokens are done by then, so 1,770 tokens finish in the scheduling gap after the answer is returned. In the evaluated policy, a follow-up question that arrives before then waits for that completion (the paper also describes an optional bounded-recovery variant).

Read the cost honestly. On image dialogues the extra work hides inside decode: TPOT rose only 0.13% and 0.47% on the two models. On the video benchmark MT-Video-Bench the paper reports mean TPOT up 19.93% and 14.92%, because video has about ten times as many deferred tokens and answers are short, so 41.67% and 68.75% of requests still had unfinished prefill when the answer ended. First-turn TTFT still fell 13.06% and 11.91%, and mean end-to-end latency fell 4.83% and 6.56% against full prefill. Deferred prefill moves work off the critical path; it does not make the work disappear, and it assumes spare compute, which large decode batches do not leave.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Chunked Prefill

Related explainers

Learned prefill token pruning — another way to choose which prefill tokens to keep, the step Raincheck builds on.
Decode-latency feedback for chunked prefill — how to size prefill chunks that share a pass with decode.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based