AMD serves NVFP4 checkpoints on MXFP4-only GPUs — Load-time NVFP4-to-MXFP4 requantization — What does it mean?

AMD added load-time NVFP4-to-MXFP4 requantization to SGLang. On September 8, 2026, AMD's ROCm blog described an online NVFP4-to-MXFP4 requantization pipeline now in SGLang. Launch the server against an NVFP4 checkpoint with --quantization quark_mxfp4 and the loader converts the weights layer by layer as they arrive, then serves them through AMD's native MXFP4 kernels. Across five models AMD reports steady-state output throughput within −0.9% to +1.0% of checkpoints that were quantized to MXFP4 offline, and GSM8K scores that recover 98.7-100.7% of each model's original higher-precision release. Read the AMD write-up →

Picture the warehouse. Every item on the shelf is a 4-bit number, and both formats store it in the same type — E2M1, one sign bit, two exponent bits, one mantissa bit. What differs is the crating, and because each crate's shared tag changes, an item's rounded value can shift when it is repacked. NVFP4 packs 16 items to a crate and lets the tag on the crate print any FP8 value; MXFP4 packs 32 items to a crate and its tag can only print a power of two. Four bits cannot express much on their own, so that shared tag is doing most of the work of representing a tensor's real range — which is exactly why you cannot hand an NVFP4 tensor to an MXFP4 kernel and hope. The bytes line up; the grouping and the tags do not.

Repacking has to happen somewhere, and the question is where. Doing it in a separate offline pass means the warehouse now holds two copies of everything. Doing it at request time would be absurd — you would re-crate the same goods on every order. AMD's pipeline does it at the loading dock: SGLang reads the NVFP4 metadata from the checkpoint, unpacks one layer's weights, re-crates them as MXFP4, writes the result straight into a preallocated slot, and frees the intermediate before the next layer arrives. Only one aisle is ever open, so the extra memory the conversion needs is bounded by the largest single layer rather than by the whole model — which matters when the model is a 397-billion-parameter mixture of experts and the naive approach would want hundreds of gigabytes of scratch space.

The pipeline also honours what the checkpoint's producer already decided. Layers a mixed-precision export deliberately left in FP8 or higher branch off to their own load-as-is path, so a layer that was meant to stay precise never gets quietly crushed into 4 bits on the way through. Activations are a separate story: since the matmul is W4A4 — weights and activations both 4-bit, they are quantized to MXFP4 dynamically, per batch, with no calibration pass and no stored activation scales.

Here is the part that should bother you. The weights were already rounded once, from BF16 into NVFP4. Now they get rounded a second time, into a format with coarser 32-wide blocks and a scale that can only be a power of two. Two lossy steps in a row is normally how you destroy a model. It does not happen here, and the reason is worth carrying to every other quantization decision you make.

Call e₁ the error NVFP4 already introduced against the BF16 source, and e₂ the extra error the MXFP4 step adds on top. If the second step amplified the first, the total would be e₁ + e₂. AMD measured the two error signals on Qwen3.5-397B-A17B's mixture-of-experts weights and found their mean pairwise correlation to be about −0.04 — nearly uncorrelated in that sample — and e₂ to be symmetric and centred on zero (mean over σ around 10⁻⁴). For zero-mean, uncorrelated errors the variances add rather than the errors themselves, so their RMS magnitudes combine in quadrature as √(e₁² + e₂²). Take the illustrative case where e₂ is about the same size as e₁: linear compounding would give 2·e₁, quadrature gives √2·e₁ ≈ 1.41·e₁ — about 29% less error than the compounding story predicts.

Then the matmul takes another bite out of it. A single output is a dot product across thousands of weights, and summing n independent zero-mean errors grows the signal with n while the error only grows with √n, so the ratio between them improves by roughly √n — for a 4096-wide projection that is a factor of 64. Put the two effects together and AMD's measured output SQNR for the NVFP4-to-MXFP4 path lands at about 17 dB, only ~3.6 dB below NVFP4 alone. Since −3 dB is a doubling of noise power, the second rounding step roughly doubles the output noise — and from a 17 dB baseline the result is still comfortably signal-dominated. The benchmarks agree with the arithmetic: GSM8K recovery of 98.7-100.7% against each model's original release, and GPQA-Diamond and AIME25 scores within ±0.05 of an emulated NVFP4 reference, on benchmarks whose own seed-to-seed variation is roughly 5-10%.

Matching a native MXFP4 checkpoint on throughput is not something the conversion earns by itself — it earns it by disappearing, and then inheriting a serving path that was already tuned for CDNA4. Two pieces of that path are worth naming. First, for small decode-time matmuls, SGLang can fuse the activation quantization into the GEMM itself, loading activations in high precision and converting them to MXFP4 immediately before they reach the matrix-core instruction. That removes one kernel launch and one round trip through HBM — the GPU's main memory, the slow tier every kernel has to reach across — which is the dominant cost when the batch is small. The tradeoff is real: the fused kernel re-quantizes the same input tiles once per output tile, so for large inputs that redundant work outweighs the saving and a standalone quantization kernel — which touches each element exactly once — is the faster path. Second, MXFP4 weights and their block scales are reshuffled once at load into the exact tile layout the matrix cores expect, so every forward pass reads them coalesced with no in-kernel repacking.

The conversion is a loader concern that runs once per layer at startup, so it adds no per-request overhead — which is why AMD measures no significant steady-state throughput difference against a native MXFP4 checkpoint, rather than the penalty you would expect from converting anything on the hot path. What the inference engine does per request is unchanged.

The bill comes due exactly once, at startup. AMD measured server start times on MI350X and found the requantization pass adds roughly 10 to 55 seconds depending on model size and tensor-parallel degree (how many GPUs the model is split across) — DeepSeek-R1 at TP8 went from about 78 s to about 112 s, MiniMax-M2.7 at TP4 from about 66 s to about 78 s. If you restart servers constantly, or you are profiling cold-start latency, that delay is the one number to weigh. Two other limits are worth stating plainly: the conversion is not bit-exact with the source NVFP4 checkpoint and never can be, since MXFP4's coarser blocks genuinely discard information, so accuracy work that must reproduce an NVFP4 result exactly should not run through this path; and it only helps on hardware that has native MXFP4 compute in the first place.

The transferable lesson is not about AMD or about SGLang. It is that a second quantization step is far less damaging when the error it adds is small, unbiased and nearly uncorrelated with the first — under those conditions the cost compounds in quadrature rather than linearly, and a wide matmul averages away much of what remains. None of that is a guarantee: a large uncorrelated error still hurts, and AMD itself frames the analysis as intuition for a result it measured rather than a proof it would hold elsewhere. Treat it as the right question to ask, and the right thing to measure, the next time someone proposes stacking one quantized representation on top of another.

Goes deeper in: LLM Internals → Quantization → The Quantization Process

Related explainers

• Nemotron-H 8B pretrains in FP4 with no Hadamard transform — a different block-scale encoding, chosen for training rather than serving
• UFP4 fixes FP4 pretraining's shrinkage bias — what happens when 4-bit rounding error is systematic instead of zero-mean
• ThriftAttention runs the top 5% of QK blocks in FP16 — the other lever: spend precision only where it changes the answer