FP4 Pretraining Study — Rare-fact acquisition vs validation loss — What does it mean?

The news. On November 9, 2026, Chang Lu and Mengjiao Zhang posted What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts. They trained small transformers (12M to about 100M parameters) from scratch on synthetic biographies whose subjects appear 1 to 256 times, with six attribute facts per person, alone and mixed into FineWeb-Edu web text, with BF16, emulated FP8, or emulated 4-bit (MXFP4 and NVFP4) matrix multiplications. NVFP4 needed 1.32× (synthetic) and 1.52× (web mix) as many exposures as BF16 to match its recall, while validation loss differed by only 0.5% and 2%. Read the paper →

Rare-fact acquisition is the number of times a model must see a fact before it can recall it, and the paper's finding is that 4-bit training raises that number without moving the loss curve much. Picture two people learning names at the same party. The regulars come past all evening, so both learn their names easily. A newcomer is introduced only once or twice. Now play loud background music for one of the two listeners: they still learn every regular, but they need more introductions before a newcomer's name sticks.

In training, the background music is the error that 4-bit arithmetic adds to every matrix multiplication (see how precision formats trade bits for range). The paper's setup makes the party literal: each synthetic person has six attributes (birth city, employer and so on), each attribute value is one fact, and persons are split into nine equal groups seen 1, 2, 4 and so on up to 256 times. Recall is the accuracy of the first token of the answer when the model is prompted with a training template cut just before the value.

Why does the score for the whole room barely move? Validation loss averages over tokens, so in the synthetic setup each fact counts in proportion to how often it appears in training, and the frequent facts are the ones both models already know. On the synthetic data, NVFP4 raised the loss on value tokens (the tokens of each fact's answer) by 0.67–0.85 nats for facts seen 1–4 times, and by at most 0.07 nats for facts seen 32 times or more. On the web-text mix the problem is worse in a different way: the held-out validation documents contain none of the inserted biographies, which is the realistic case, because a validation set rarely holds the rare facts a model should pick up.

Here is the arithmetic on the paper's nine-group layout (illustrative arithmetic, using the paper's reported gaps). Each person in the group seen 256 times contributes 256 copies of their facts, so across one person per group the token counts are 1 + 2 + 4 + … + 256 = 511. The 256-group alone is 256 / 511 ≈ 50% of all value tokens, and the 1-, 2- and 4-groups together are 7 / 511 ≈ 1.4%. A gap of about 0.8 nats on that 1.4% adds only about 0.011 nats to a frequency-weighted average. The paper's measured version of this: averaged with equal weight per fact the NVFP4 gap is 0.32 nats; weighted by training frequency, as the validation stream is, it shrinks to 0.04 nats, and over all validation tokens, template words included, the loss rises by 0.5%.

Where does the music come from? Snapping values to a coarse grid, as the number line above shows, adds rounding error to every operand (see how numbers shrink). The authors quantized the forward and backward passes separately. Quantizing only the backward pass left validation loss almost unchanged but still cost 1.13× in exposures, and adding zero-mean Gaussian noise of the same size to BF16 gradients reproduced that cost (1.11×). NVFP4 already uses stochastic rounding on the backward pass, so its gradient error is close to zero-mean: the expected gradient is right, and the damage has to come from how the optimizer uses a noisy gradient.

Adam turns gradient noise into two costs, and over the paper's runs only one of them faded. Adam divides each step by a running estimate of the gradient's size, its second moment. Noise inflates that estimate, so every step shrinks, which acts like a lower learning rate; in a controlled test at twice the NVFP4 error, feeding noise only to the second moment matched a reduced-rate BF16 run, and that cost faded over training. Noise in the update itself lowers the gradient signal-to-noise ratio, and that part persisted to the end of training and carried about two thirds of the cost. Raising the learning rate to restore the average step size did not recover it, for the injected noise or for NVFP4. In party terms: turning up your own attention does not help when part of every introduction is replaced by music. Master weights and Adam's state stayed in FP32 in every run, as in standard mixed-precision training; the noise enters through the gradients the 4-bit GEMMs produce, and running the backward GEMMs in FP8 instead lowered NVFP4's cost from 1.32× to 1.23×.

The practical lesson is that a low-precision recipe judged by loss can look fixed while the long tail is still being learned slowly. The authors' adaptation of the published NVFP4 recipe (2D weight scaling, stochastic rounding only on gradients, the last 2 of 8 blocks in BF16) closed 60% of the validation-loss gap but only 42% of the exposure cost, and every remedy they tried recovered loss faster than recall. Their suggestion is cheap: instrument a slice of the training data with facts of known frequency (biographies were 7.2% of tokens in the web-mix runs) and report a recall-versus-exposures curve next to the loss. The results come from models of at most 96.6M parameters trained on at most 1B tokens, with FP4 emulated in FP32 on one RTX 4090 rather than run on Blackwell hardware, so whether the multiplier holds at production scale is still open.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Mixed Precision Training & Inference

Related explainers

• UFP4 and E2M1 shrinkage bias: the biased half of FP4 gradient error, which this paper's zero-mean noise model leaves out
• Joint smoothing and block scales for NVFP4: a recipe that targets NVFP4's loss gap
• Preconditioner-space stochastic rounding: stochastic rounding applied to Adam's own state