Greedy decoding is not precision-invariant — Top-two logit margin decides BF16 vs FP16 flips — What does it mean?
The news. On September 22, 2026, a team from the University of Tennessee, the University of Chicago, Amazon and UT Austin posted Greedy Decoding Is Not Precision-Invariant (accepted to TMLR). Across six models from 1.1B to 7B parameters, four model families and three benchmarks, 49–100% of prompts produced different greedy output in BF16 than in FP16 on the same GPU. The paper identifies the final projection as the dominant correctable stage and tests a margin-gated FP32 repair. Read the paper →
Picture a sprint with two official timekeepers. One holds a stopwatch that ticks in coarse steps, the other a stopwatch with finer ticks. When the winner leads by five metres, both declare the same winner. When two runners cross the line almost together, the two stopwatches can name different winners, and neither stopwatch is broken. Each one rounds the finish time to its own tick.
That is what happens at every step of greedy decoding. The model scores every token, and the highest logit wins. BF16 and FP16 are the two stopwatches: BF16 keeps 7 mantissa bits and FP16 keeps 10, so they round the same real number to different values. For a stored value between 16 and 32, BF16 can only represent steps of 0.125, while FP16 can represent steps of about 0.016. That spacing follows directly from the bit layouts shown below. Most of the time the leading token wins by a wide margin and the rounding does not matter.
The paper's key finding is about where the rounding matters. Error from the two formats builds up evenly through all 22 transformer layers of TinyLlama-1.1B, and it is present in prompts that agree as much as in prompts that diverge. So the size of the accumulated error does not separate flipping steps from safe ones; flips concentrate where the top-two margin is small compared with the rounding difference between the two candidates. The lm_head, the final matrix multiply that projects the hidden state onto the whole vocabulary, is the dominant stage that turns small differences into a changed token. Under the authors' rough, order-of-magnitude estimate for TinyLlama-1.1B, the lm_head contributes about 97.6% of the per-logit difference between the two formats, and the transformer body about 2.4%.
The photo finish also explains why one flip matters so much. The winner of one heat decides who runs in the next heat: the chosen token becomes input for every later step. A single flipped token sends the rest of the answer down a different path. On TinyLlama with GSM8K, 24 of 59 diverged prompts ended at a different length, with a mean difference of 34 tokens and a maximum of 236. On Qwen2.5-3B with GSM8K, 19% of prompts changed whether the final answer was correct.
The fix is the photo-finish camera: use it only on close calls. At each step the model computes the logits in its normal 16-bit format and measures the top-two margin. If the margin is below a threshold τ = 10−3, the whole-vocabulary lm_head is recomputed in FP32 for that step only; otherwise the normal argmax is used. That threshold fires on only 0.7–1.4% of steps. The paper also tested a cheaper variant: on triggered steps, recompute only the top two candidate logits in FP32, then round them into integer bins with a fixed tie-break. That variant matched whole-vocabulary recomputation in this experiment.
| Method (TinyLlama-1.1B) | GSM8K EAR | HumanEval EAR | MBPP EAR | Cost | Source |
|---|---|---|---|---|---|
| No intervention | 41% | 36% | 30% | — | Table 3 |
| Integer-rounded logits before argmax | 41% | 36% | 30% | under 1% | Table 3 |
| FP32 recompute on every step | 60% | 56% | 54% | ~2.5× latency | Table 3 |
| Margin-gated FP32 lm_head | 63% | 61% | 55% | under 4% | Table 3 |
| Whole model in FP32 compute | 88% | 88% | 89% | ~2× memory | Table 3 |
Two rows in that table are counter-intuitive. Coarse integer rounding of the logits does nothing, and always-on FP32 recomputation does worse than recomputation gated to close calls. Rounding fails because BF16 and FP16 do not just compare the same numbers differently; they compute different numbers. Always-on recomputation fails because it also changes the logits at safe steps and creates new divergence points later in the answer. Extending FP32 to more of the model has the same effect: adding the normalization layer just before the lm_head (RMSNorm) to the FP32 scope dropped agreement from 63% to 44%.
Here is where the gain comes from, with the model (TinyLlama-1.1B), benchmark (GSM8K, 100 prompts) and threshold (τ = 10−3) held fixed. Without the fix, 41 of 100 prompts agree; the gate fires on 30 prompts and fixes 22, lifting exact agreement from 41% to 63%. So 59 prompts diverge at the start, and the gate fires only on the ones whose margin at the divergence step is below τ. At the measured first-divergence step, the remaining 29 diverged prompts have a margin at or above τ, so the gate does not fire there: their flip is driven by hidden states that have already drifted apart upstream, and recomputing one final matrix multiply cannot bring them back together. The scale of the problem is visible in the margins themselves: on diverged prompts the median margin is about 0.00001 (30 of 59 are exact ties in BF16), on agreed prompts it is about 5.9, and the typical per-logit difference between the two formats is about 0.026. The gated fix adds 1.4% latency in that setup; upcasting the whole model to FP32 compute adds 38%.
The repair has clear limits, which the authors state. It is a partial mitigation, not a determinism guarantee. Its benefit disappears at batch size 8 and above, and under end-to-end FP8, because there the error that causes flips comes mostly from the transformer body rather than the lm_head. Its lift varies by model, from +36 percentage points on Llama-3.2-3B to 0 on Qwen variants that diverge on every prompt. And when you are free to pick one serving format, the paper finds that FP16 alone stays closer to FP32 output than BF16 with the fix. The practical lesson for offline evals is simpler: record the numeric format with every run, because two runs of the same checkpoint in different formats are not the same experiment.
Goes deeper in: LLM Internals → Text Generation → Greedy vs Creative
Related explainers
- Prefix-cache nondeterminism in agent serving — a different reason the same request can produce a different run: the cache state, not the number format
- Phase-asymmetric quantization — NVFP4 prefill + BF16 decode — choosing a different number format per phase on purpose, and what it costs