What is weight oscillation in low-precision training?

It is a quantized weight flipping between two neighbouring values while its high-precision master copy barely moves. It happens when the master weight sits near a bin edge, or when an outlier moves the shared block scale, and it makes the layer see a noisy weight.

How does ALTO detect and fix oscillating weights?

Every 200 optimizer steps it compares how far each quantized weight jumped with how far its master weight travelled. Elements whose ratio exceeds 4.0 are snapped to their current bin centre. It runs as a post-step hook on AdamW, so the model code is unchanged.

Does ALTO make 4-bit training faster?

The paper does not show that. It reports accuracy with simulated MXFP4 kernels on AMD MI300 hardware and states that no wall-clock speedup over BF16 is reported. It shows the numerics reach loss 3.3350 versus 3.3283 for BF16 at 16,128 steps.

GPU· 2026-10-27

ALTO MXFP4 training — Weight de-oscillation — What does it mean?

Jargon

MXFP4: A 4-bit microscaling format. Each block of 32 numbers shares one scale factor, and each number is stored as a tiny 4-bit value.
E2M1: The 4-bit number layout inside MXFP4: 2 exponent bits and 1 mantissa bit. It can only represent a handful of values, roughly between -6 and 6 after scaling.
Master weight: The high-precision copy of a weight that the optimizer updates. The matrix multiply uses a quantized copy made from it, never the master itself.
Bin: One of the few values a quantizer can output. Every real number is rounded to the nearest bin, and the bin edge is the point where the rounding flips.
Randomized Hadamard transform (RHT): A random rotation applied before quantizing so that one large value is spread across many entries, which keeps a block from being dominated by a single outlier.
Stochastic rounding (SR): Rounding up or down at random, with odds set by how close the number is to each neighbour, so the rounding error averages out instead of leaning one way. See UFP4 and rounding bias.
AdamW: The standard optimizer for LLM training. It moves each weight by a small, adaptive step on every iteration, which is why a weight can sit close to a bin edge for a long time.

The news. On October 23, 2026, AMD researchers posted ALTO, an open recipe for training the 20-billion-parameter GPT-OSS mixture-of-experts model in MXFP4 on the MLPerf Small MoE benchmark. With hybrid block quantization, random Hadamard rotations, stochastic rounding and a weight de-oscillation hook, it reached validation loss 3.3350 at 16,128 steps, 0.007 above the BF16 baseline of 3.3283. All runs used simulated MXFP4 kernels on AMD MI300 hardware, so the paper reports no wall-clock speedup. Read the paper →

Picture someone standing right on the lip between two stair steps while a floor scanner reports which step they are on. Their body hardly moves, yet the scanner alternates between step 3 and step 4. Low-precision training has the same problem. The optimizer keeps a smooth, high-precision master weight, but the matrix multiply sees only its quantized copy, and MXFP4 offers very few values per block, so the bins are wide. A weight that sits near a bin edge gets nudged by a tiny AdamW update and its quantized value hops to the neighbouring bin, then hops back on the next update. The layer therefore sees a jittery weight while the master weight looks perfectly calm.

The paper names a second cause. In MXFP4 each block of 32 values shares one scale, set by the largest magnitude in the block. A single outlier can move the block scale, and a moved scale shifts the bin edges for every other element in the block. An element can then change bins without being near an edge at all. This is the same outlier problem that makes block scaling necessary, now showing up as noise inside training rather than as error at inference.

ALTO detects the flicker with a ratio, borrowing the criterion from the TetraJet-v2 paper. Over a window of P = 200 optimizer steps it adds up two distances for each element: d_w, how far the master weight really travelled, and d_Q, how far the quantized value jumped. An element is flagged as oscillating when d_Q / d_w exceeds 4.0. At the end of the window, every flagged element is snapped to the centre of its current bin, which takes it away from the edge. The check runs as a post-step hook on a standard AdamW optimizer after a warmup of 2,000 steps, so the model code does not change. TetraJet-v2 instead builds the logic into the model, starts at step 8,000 and uses a threshold of 16, so ALTO treats oscillation more aggressively and earlier. The cost is extra optimizer-state memory for the two distances and a weight snapshot, with no extra matrix multiplies.

Here is the ratio on one weight (illustrative numbers, not from the paper). Suppose bins are 0.25 apart. Over a 200-step window the master weight wanders back and forth and travels a total of d_w = 0.02, a tiny amount. Its quantized value crosses the edge 6 times, so d_Q = 6 × 0.25 = 1.5. The ratio is 1.5 / 0.02 = 75, far above 4.0, so the weight is flagged and snapped to its bin centre. A healthy weight that drifts 0.30 and crosses one edge has d_Q = 0.25 and a ratio near 0.8, so it is left alone. The measured effect of the whole hook is steps to the quality target: BF16 needed 15,360 steps, plain MXFP4 needed 18,432 (+3,072), and MXFP4 with de-oscillation needed 16,128 (+768). That is 2,304 fewer steps, and the overhead falls from 20.0% to 5.0%.

Recipe (GPT-OSS-20B, batch size 16)	Steps to loss 3.34	Overhead vs BF16	Source
BF16 baseline	15,360	—	arXiv Table 4
MXFP4 + RHT + stochastic rounding	18,432	+20.0%	arXiv Table 4
MXFP4 + RHT + SR + M+Adam	16,128	+5.0%	arXiv Table 4
MXFP4 + RHT + SR + de-oscillation	16,128	+5.0%	arXiv Table 4

The paper is just as useful for what failed. Differential gradient estimation, outlier clipping and macro-block scaling each improved the signal-to-noise ratio of individual operators, yet end-to-end validation loss barely moved or got worse, so operator-level SNR was an imperfect proxy for training quality here. Dynamic outlier clipping caused a large regression. De-oscillation and M+Adam, an optimizer variant, converged at a similar speed, but the MLPerf closed division requires exactly AdamW, so the hook is the closed-division choice. Treat the result with its limits in mind: it is one model family, one C4 subset, batch size 16, and simulated MXFP4 arithmetic with separate unfused kernels, so it shows that the numerics hold up, not that 4-bit hardware is faster in practice.

Goes deeper in: LLM Internals → Quantization → How Numbers Shrink

Related explainers

UFP4 and E2M1 shrinkage bias — a different 4-bit rounding bias, fixed with a rotation
Nemotron-H FP4 block scaling — why the format of the shared block scale decides whether 4-bit training works

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based