Nemotron-H 8B pretrains in FP4 with no Hadamard transform — UE5M3 block scaling — What does it mean?

The news. On September 2, 2026, a paper from NVIDIA described pretraining Nemotron-H 8B on software-emulated 4-bit floating point for nearly 190 billion tokens. The headline choice is not the 4-bit payload — that part is ordinary E2M1 — but the format of the scale factor stored beside every block of 16 values. Moving that scale to UE5M3 reportedly buys enough exponent range to update scales on a schedule rather than from the current tensor. The same recipe also omits the randomized Hadamard transform and keeps eligible internal linear layers in FP4 instead of exempting final blocks to BF16. A separate native-execution ablation reports 21.2% higher model-body token throughput. Read the paper →

Picture the crate. Sixteen parts go in, and each one gets a single digit scrawled on it — that is all a 4-bit float has room for. E2M1, the FP4 payload format, can represent exactly eight magnitudes: 0, 0.5, 1, 1.5, 2, 3, 4 and 6. The largest is twelve times the smallest non-zero one, and that is the entire vocabulary. A crate whose parts genuinely span a thousand-to-one range cannot be described with those digits alone.

So the lid gets a stamp. The stamp is one number shared by all sixteen parts — multiply every digit by it and the real sizes come back. This is what block scaling means: the 4-bit numbers carry shape, and the shared scale carries magnitude. The scale-and-round step you already know does the same thing per tensor or per channel; a block is simply a much smaller neighbourhood, so the scale tracks the local range far more tightly.

Which leaves the question nobody asks until it bites: what format is the stamp itself?

The stamp is eight bits, and how you spend them is the same range-versus-precision trade FP8 makes twice over. E4M3 spends four bits on the exponent and three on the mantissa, topping out around ±448. E5M2 moves a bit from mantissa to exponent, reaching far higher with coarser steps. Neither is obviously right, which is why both exist.

A block scale gets one extra bit to spend, for a reason that sounds like a technicality and is not: a scale is never negative, so dropping the sign bit lets an unsigned 8-bit scale afford five exponent bits and still keep three of mantissa. That is UE5M3 — roughly E5M2's reach with E4M3's resolution, in the same eight bits.

Here is why the extra reach matters more than it sounds. Within any one block of sixteen, a couple of values are often far larger than the rest — the same outlier problem the quantization module walks through, at a much finer grain. A scale pinned to the block's current maximum has to stretch to cover the outlier, which pushes the other fifteen values down into the bottom of E2M1's eight magnitudes, where the spacing is coarse relative to their size.

The established fix was to rotate the problem away before quantizing: a randomized Hadamard transform mixes each block's values together so no single one dominates, at the cost of an extra transform on the forward pass and on both of the gradient matrix multiplies (GEMMs). This recipe omits that transform altogether. Read the connection carefully, though — the paper presents scale format, rounding policy and transform removal as one package, so it does not isolate how much of the credit belongs to the scale format alone.

Hold the block size fixed at sixteen and count the bits. Sixteen E2M1 payloads cost 16 × 4 = 64 bits. Add one 8-bit scale and the block costs 72 bits, which works out to 4.5 bits per value — the half-bit is the stamp, amortized across the crate. Now swap the stamp format and count again. UE8M0 is 8 bits. E4M3 is 8 bits. UE5M3 is 8 bits. The block still costs 72 bits and each value still costs 4.5 bits, whichever stamp you print — so the extra exponent range is free in memory and free on the wire, paid for entirely by giving up a sign bit nobody was using.

Two smaller choices ride along with it. Because the stamp has range to spare, it no longer has to be re-derived from the current tensor on every step — the recipe updates scales periodically instead, which turns a per-step reduction over the whole tensor into an occasional one. And stochastic rounding is applied selectively to the backward gradients rather than everywhere, so the forward pass stays deterministic.

Read the two headline numbers as two different experiments, because they are. The 190-billion-token run used software-emulated FP4 values, so what it establishes is quality — it reportedly finished with lower final-window training loss than NVIDIA Transformer Engine's NVFP4 recipe. The 21.2% higher model-body token throughput comes from a separate native-execution ablation, an isolated comparison of that one change. Speed and quality are each measured once, in different runs; the throughput figure is not the speed of the 190-billion-token run.

Goes deeper in: LLM Internals → Quantization → How Numbers Shrink

Related explainers

• UFP4 fixes FP4 pretraining's shrinkage bias — the counterpoint: why the randomized Hadamard transform was there in the first place.
• ThriftAttention: importance-aware FP4 attention — the same 4-bit payload, spent selectively inside an attention kernel.
• KVarN: Hadamard-rotated 2-bit KV cache — where rotation still earns its keep, at inference time.