Lost in Rounding — Stochastic rounding for BF16 recurrent state — What does it mean?

The news. On November 9, 2026, Yuchuan Weng, Xin Chen and Chi Liu of PayPal AI posted Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer. They served six open hybrid reasoning models (Mamba-2 and Gated DeltaNet mixers) in the SGLang serving engine on 590 competition math problems, changing only the dtype of the cached recurrent state. With BF16 round-to-nearest, five of the six reasoned longer (geometric-mean length ratio 1.07 to 1.27) and hit the 32K cap more often. A patched decode kernel with stochastic rounding, same storage, brought four of the five back to FP32 behaviour. Read the paper →

A hybrid model's recurrent state is re-rounded on every generated token, so a rounding error that always points the same way can pile up over a long reasoning trace. Picture a tank that leaks a little each minute, and a clerk who re-logs its level every minute on a gauge with coarse marks. If the clerk writes down the nearest mark, and the leak in one minute is smaller than half the gap between marks, the logged level never changes: the clerk writes the same mark again, the next minute starts from that mark, and the leak is lost every single time. The log says the tank is full long after it is mostly empty.

That is what happens at decode time. Each recurrent layer processes one token at a time: the engine reads the state from its cache, computes the update in FP32, and stores the result back in the cache's dtype. BF16 keeps only 8 significant bits (see how precision formats trade bits for range), so its gauge marks near 1.0 are about 0.4% apart. When a value's per-step change is smaller than half that gap, round-to-nearest writes the old value back and the state does not move, an effect earlier work named write erasure. The paper's toy simulation of one quiet, slowly decaying value shows the regimes: round-to-nearest freezes slow decay completely, distorts moderate decay, and is nearly unbiased only for fast decay.

Here is the arithmetic for one value (illustrative numbers, BF16 spacing from the format itself). A state value of 1.0 should decay by a factor of 0.999 per token, so the true next value is 0.999. The nearest BF16 values are 1.0 and 0.99609375 (one step of 2⁻⁸ below), so 0.999 is closer to 1.0 and round-to-nearest stores 1.0. The next token starts from 1.0 again, gets the same answer, and so on: after 1,000 tokens the cache still holds 1.0, while the true value is 0.999¹⁰⁰⁰ ≈ 0.37. Stochastic rounding instead rounds 0.999 down with probability 0.001 / 0.0039 ≈ 0.26 and up with probability ≈ 0.74, so the expected stored value is 0.74 × 1.0 + 0.26 × 0.99609 ≈ 0.999: each log is still coarse, but on average the tank drains at the right rate.

Back at the tank: the clerk's fix is not a finer gauge, it is a weighted coin. When the level sits a quarter of the way from one mark down to the next, log the lower mark a quarter of the time. That is the paper's main argument. If the BF16 failure came from too little resolution, an unbiased rounding at the same width should not help; it does, so the cause is mostly bias. The authors patched SGLang's Mamba-2 and GDN decode kernels to add 16 random bits below the BF16 mantissa of each FP32 state value and then truncate, which rounds up with exactly the right probability. Storage and per-GPU request limits stayed identical to BF16 round-to-nearest.

Why should a frozen number make a model think longer? The paper measures the effect but does not claim a full causal story, and its own toy model explains only part of which layers are affected. What it does show is where the extra length goes. Rerunning Qwen3.5-9B's capped problems with a 128K budget halved the accuracy gap but did not close it (93.6% vs 86.6%): BF16 runs that never finish were twice as common, and 85% of those were exact repetition loops. On Nemotron-Nano-9B-v2, by contrast, 30 of its 31 extra capped runs finished at 128K, so there the cost was mostly extra tokens. On the LiveCodeBench coding benchmark the two models tested lost 12.6 and 13.4 points of pass@1 (the share of problems solved on the first attempt) under BF16 round-to-nearest. The effect is not universal: on Nemotron-3-Nano, stochastic rounding removed the runaway tail (40 of the 45 extra caps) but a small length increase of about 5% remained, which the authors cannot yet explain.

The serving lesson is that the state cache dtype changes what the model says, not only how fast it says it. A BF16 state takes half the memory of FP32, and on the authors' servers that cache budget admitted twice as many concurrent requests per GPU; that is why engines expose it, and why a vLLM bug fix had to switch one Nemotron model's default back to FP32 after BF16 state caused runaway generations (cap hits fell from 23% to 0.25%). Stochastic rounding keeps the BF16 saving: with an integer-hash random generator, Qwen3.5-9B matched FP32 on cap hits and accuracy and decoded at 10,393 tokens/s against 10,163 for a same-day BF16 round-to-nearest run. The authors also recommend that evaluations of hybrid models report the state dtype, the cap rate and output length next to accuracy, because the common practice of scoring only problems both arms finished hides the loss: on Qwen3.5-9B that split reads 99.7% vs 99.7%.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Precision Formats

Related explainers

• Decay-aware bit allocation for low-bit recurrent states: the resolution side of the same state, when it is squeezed to about 4 bits
• Preconditioner-space stochastic rounding: stochastic rounding applied to an optimizer's 4-bit state
• Gated DeltaNet 2: decoupled erase and write gates: how a GDN head decides what to erase and write each step