What is weight sticking in BF16 training?

It is when an optimizer update changes the FP32 shadow weight but not the BF16 copy used for the forward pass, because round-to-nearest maps the slightly changed value back to the same BF16 number. The weight stays stuck until its accumulated update exceeds **half a ULP**, which takes longer in BF16 because its ULP is 8× larger than FP16's.

Why does weight sticking matter more in RL post-training than in pretraining?

RL with verifiable rewards uses learning rates around 10 -6 , far below the half-ULP of a typical BF16 weight, so most updates fall inside one rounding step. The BF16 weights also generate the rollouts, so a stuck weight means the model keeps producing training data from an outdated policy.

How does stochastic rounding fix it without more bits?

It rounds up with a probability equal to how far the value has moved toward the next representable number, so the rounded weight is correct on average. In the NVIDIA paper, rounding the FP32 shadow weights to BF16 this way once per optimizer step matched FP16's convergence speed, and in the paper's drift comparison it stayed close to FP16's drift.

Why not just switch to FP16 or raise the learning rate?

FP16 needs loss scaling and, in the paper's preliminary large agentic run, overflowed often even at a loss scale of 1,024, and each overflow wastes that step's rollouts. A higher BF16 learning rate matched FP16's reward curve but drifted about 1.5× further from the reference model.

GPU· 2026-11-13

BF16 vs FP16 in RL post-training — Weight sticking and stochastic rounding — What does it mean?

TL;DR

What is it: Weight sticking is the failure that the November 2026 NVIDIA paper Closing the BF16–FP16 Convergence Speed Gap in RL Post-Training names: when the precise FP32 copy of a weight is rounded to BF16 for the forward pass, small updates round away, so the weight the model actually uses does not change; the fix is stochastic rounding.
Why it’s needed: Reinforcement learning with verifiable rewards (RLVR) commonly uses very small learning rates, and the BF16 weights also generate the rollouts the model learns from, so in the paper's experiments frozen BF16 weights meant more RL iterations to reach the same reward.
vs previous: The usual mixed-precision recipe assumes keeping FP32 shadow weights is enough because no update is lost there; this paper shows the update still disappears in the round-to-nearest copy to BF16, and that stochastic rounding fixes it without switching to FP16 (which needs loss scaling) or raising the learning rate (which pulls the model further from its reference).

Jargon

BF16: Brain Float 16, a 16-bit number format with FP32's 8 exponent bits but only 7 mantissa bits, so it covers a huge range with coarse steps. How the bits split
FP16: Half precision: 10 mantissa bits (finer steps than BF16) but only 5 exponent bits, so small gradients can underflow to zero unless the loss is scaled.
FP32 shadow weights: The full-precision copy of every weight that the optimizer updates in mixed-precision training; it is rounded to BF16 or FP16 before each forward pass. Also called master weights.
ULP: Unit in the last place: the gap between a number and the next value the format can store. The gap grows with the number's size, and BF16's gap is 8× FP16's for the same value.
Round-to-nearest: The default rounding rule: pick whichever representable neighbour is closer. It is deterministic, so the same input always gives the same output.
Stochastic rounding: Round up or down at random, with the chance of rounding up equal to how far the value sits between the two neighbours. On average the rounded value equals the exact value.
RLVR: Reinforcement learning with verifiable rewards: the model generates answers (rollouts), a checker marks them right or wrong, and the weights are nudged toward the right ones. The paper studies a learning rate of 10^-6, a rate it describes as common in RLVR.
KL divergence (drift): A measure of how far the trained model's output probabilities have moved from the starting (reference) model. RL post-training usually tries to keep it small, because large drift predicts more forgetting.

The news. On November 10, 2026, NVIDIA researchers posted a paper showing that, at the low learning rates used in RLVR, standard BF16 mixed-precision training needs more RL iterations than FP16 to reach the same reward, even when the rollout and training engines produce bit-exact log-probabilities. They trace the gap to weight sticking and close it by changing one rounding step, on models from 1.5B to 30B-A3B parameters. Read the paper →

Picture the fuel gauge. The tank holds the real level, measured to the drop: that is the FP32 shadow weight, the copy the optimizer updates. The driver never looks at the tank, only at the needle, and the needle can only rest on a tick mark. In mixed-precision training the needle is the BF16 copy of the weight, and every forward pass, including the rollouts the model learns from, reads the needle.

Now pour in a few drops. The level rises, but not past the halfway point to the next tick, so a needle that snaps to the nearest tick does not move. With round-to-nearest, a weight stays frozen until the sum of its updates is larger than half the gap between ticks. That gap is one ULP, and BF16 has only 7 mantissa bits against FP16's 10, so its ticks are 8× farther apart for a weight of the same size. The paper writes the stuck case as an exact cancellation: the rounding error is equal and opposite to the update the optimizer just made.

Pretraining rarely shows the problem: the paper cites earlier work finding that BF16 mixed-precision pretraining converges much like FP32. RLVR is different: at a learning rate of 10^-6, the update to a typical weight is far smaller than half a tick in either format, so most weights stay fixed, and BF16 freezes more of them. The paper rules out the usual suspect, a numerical mismatch between the rollout engine and the trainer, by making the two produce bit-exact log-probabilities (the probability each engine assigns to every token); FP16 still reached each reward level in fewer RL iterations.

Stochastic rounding changes how the needle moves, not how many ticks the dial has. Instead of snapping to the closest tick, the needle jumps up to the next one with a probability equal to how far the level has risen toward it. Every drop now changes the odds, so on average the needle tracks the tank exactly; the cost is noise, because any single reading can be one tick off. The paper's bet, which its experiments support, is that zero-mean noise hurts RL less than an error that always cancels the update.

Where the time goes, worked through. Take the median weight magnitude in the paper's 1.5B model, about 0.025. At that size one BF16 tick is about 1.2 × 10^-4, so half a tick is 6.1 × 10^-5; in FP16 half a tick is 7.6 × 10^-6. An Adam step at a learning rate of 10^-6 moves a weight by roughly 10^-6. If every step pushed the same weight in the same direction (an illustrative best case, since real updates change sign), the BF16 copy would need about 61 optimizer steps to cross half a tick, against about 8 for FP16; with 4 optimizer steps per RL iteration in the paper's setup, that is about 15 RL iterations versus 2 of rollouts generated from an unchanged weight. Measured, after the first RL iteration 1.16% of BF16 weights had moved, against 9.21% for FP16 (Table 1). Under stochastic rounding, the same single 10^-6 step gives the weight about a 0.8% chance (10^-6 ÷ 1.2 × 10^-4) of moving up one tick on this step, so its expected value moves by exactly 10^-6.

Remedy	What it changes	Cost reported in the paper
FP16 instead of BF16	Ticks 8× closer, so small updates cross sooner	Needs loss scaling; a preliminary 397B-A17B agentic run overflowed often even at a loss scale of 1,024 (paper)
Higher BF16 learning rate	Bigger steps, so they cross a tick sooner	At 2 × 10^-6 it matched FP16's reward curve but drifted ~1.5× as far from the reference (KL, mean reward above 0.7, 1.5B model) (paper)
BF16 + stochastic rounding	Same ticks, same learning rate, unbiased rounding of the shadow weights	Matched FP16's reward curve on 1.5B, 14B and 30B-A3B models; in the drift comparison (1.5B model) it stayed close to FP16 (paper)

The cheapest version is enough: draw one stochastic rounding of the shadow weights after each optimizer step, and use that BF16 copy for every forward pass until the next step. The paper found that drawing a fresh rounding in every forward pass gave the same reward curve, and that also rounding the backward-pass weights or the gradients stochastically did not speed convergence further. Rounding stochastically only in the rollout engine was not enough: the reward stalled well below FP16's. Every other conversion can stay round-to-nearest.

The authors expect the problem to grow with coarser formats. Fewer mantissa bits mean wider ticks, so 8-bit formats such as MXFP8 (a block-scaled 8-bit float) would freeze even more weights at RL learning rates; early asynchronous RL runs suggest stochastic rounding helps there too, though the paper leaves that study for later. It also notes one result it has not explained: on the 30B-A3B mixture-of-experts model, BF16 with stochastic rounding produced responses about 25% shorter than the other two settings.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Mixed Precision Training & Inference

Related explainers

Lost in Rounding: BF16 recurrent state and stochastic rounding — the same rounding rule fixing a different BF16 failure, in an inference-time recurrent state
ZIP-SR: stochastic rounding for 4-bit optimizer state — stochastic rounding applied to AdamW's second moment instead of the weights

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based