Decode-length prediction hits a noise floor — Irreducible sampling error — What does it mean?

The news. On October 29, 2026, a paper from Xi'an Jiaotong University and Shandong University posted on arXiv argued that decode-length prediction from prefill hidden states is limited by the randomness of autoregressive sampling, and that five predictor families converge near the resulting bound when model, data and sampling are fixed. Read the paper →

Picture a host at a busy restaurant who must quote a wait time before a party sits down. The host can read the party: size, order, mood. But even for the very same party, the stay changes from night to night, because a party decides course by course whether to order dessert. A better-trained host does not remove that randomness. That is the situation of a length predictor. It reads the prompt's prefill hidden states, then must guess how long the model will talk, and the model itself is rolling dice at every sentence.

The paper makes this precise by treating decoding as a stopping time: at each step the sampler either emits a stop token or continues. A constant stop probability per token would give a geometric distribution, and the paper reports that the measured behavior rules that out. Instead the stopping probability ramps up over hundreds of steps. The paper's explanation is that a response is a chain of segments (sentences, list items, reasoning steps), and the real decision is whether to start another segment. Adding up many such decisions gives, by a central limit theorem, an approximately normal decode length for each prompt. On 300 ShareGPT prompts decoded 200 times each with Qwen3-32B, the normal model reproduced the per-prompt MAE and the central mass of the distribution to within a few percent. The paper also reports a mild right skew, which matches a runaway-generation tail.

If each prompt's lengths are approximately normal with spread sigma, the best a predictor can do is guess the middle. The expected miss is then set by that curve: MAE can never go below about 0.8 times sigma (the mean absolute deviation of a normal curve is sigma times the square root of 2/pi). The pass rate has a matching ceiling for every tolerance. These limits hold for any predictor that sees only the prompt: an MLP, a graph regressor, entropy pooling or a mixture of experts. The paper also finds that sigma grows with the mean length, following an affine law with coefficients that differ widely across models and post-training recipes. That is why the same predictor looks good on short-answer data and bad on long-answer data, and why absolute MAE changes when the model, dataset or sampling settings change.

Table 2 above uses one fixed setup: Qwen3-32B, ShareGPT prompts, fixed sampling settings, 300 test prompts. Read it through the host metaphor: the simplest host, an MLP on concatenated layers, lands within a few percent of the fancier ones in this setup. Only the entropy-pooling variant is far off, and the paper suggests entropy weighting can inject noise when token entropy does not line up with the length signal.

Here is the worked example, using the paper's numbers. The mean per-prompt sigma on this test set is 162.3 tokens, and 0.8 times that gives the 129.5-token floor in the last row. The best predictor reaches 168.1 tokens. The gap is 168.1 minus 129.5, which is 38.6 tokens, so about 77% of the best predictor's error (129.5 of 168.1, our arithmetic) is the floor, and at most the remaining 38.6 tokens is model error a better predictor could shrink. Pass rate has a ceiling too: the ceiling at Acc@100 is 57.9%, the best measured is 48.9%, and no predictor can reach even 60% on this split, let alone 90%. For a single prompt the effect is large: a prompt with sigma of 50 tokens has a floor near 40 tokens, while one with sigma of 300 tokens has a floor near 240 tokens (illustrative values, same formula).

For serving systems this changes what to build. A scheduler that batches by predicted length, or a capacity plan that sizes decode pools from predicted output, should treat a prediction as a center plus an error bar that cannot be trained away. The paper recommends a benchmark protocol that fixes the target model, dataset and sampling parameters, reports the per-prompt sigma, and reports relative error (MAE divided by the length scale) or pass rate as a fraction of the ceiling, because absolute MAE grows with response length. It also notes the one way to go below the floor: predict again while decoding. Each generated token reveals information about the remaining length, so the uncertainty shrinks, at the price of an extra forward pass per prediction. The authors suggest this is practical mainly at coarse intervals such as every 50 to 100 tokens. One caveat: the evidence comes from one paper and mostly one model and dataset, and the authors list uncontrolled runtime differences between prefill and decode workers as a limitation.

Goes deeper in: LLM Serving → Inference Engine → The Scheduler

Related explainers

• Adversarial length suffixes and shortest-job-first scheduling — what happens to a scheduler that trusts length predictions.