CertiSum: bitwise-identical MXFP4 training across GPU vendors — Exact-accumulation certificates — What does it mean?
The news. On November 16, 2026, the CertiSum paper (arXiv:2611.21584) reported bitwise-identical MXFP4 training of Qwen3-8B-Base across an AMD MI350X and an NVIDIA GB200 for 200 steps, and across changes in data, tensor and context parallelism on MI350X. Across three 8B–14B models on MI350X, it adds 11.8–60.5% to training-step latency over its non-exact MXFP4 twin at the larger tested shapes, while staying up to 1.24× faster than BF16. Read the paper →
Picture two shops counting the same bag of coins onto a tally board that has room for only 24 digit slots, the way FP32 keeps 24 significant bits. When the running total grows wider than the board, the lowest digit no longer fits and falls off. Which digit falls off depends on what is already on the board, so the shop that adds the big coins first and the shop that adds the small coins first can end with different totals from identical coins. That is floating-point addition: each step rounds, so (a + b) + c can differ from a + (b + c).
In training, you do not choose the order of additions — the GPU split and the vendor's kernel choose it for you. Tensor parallelism splits each matrix multiply's dot products into per-GPU partial sums, data parallelism splits the gradient sum across replicas, and kernels split work across streaming multiprocessors. Pinning one order is not enough across vendors either: AMD and NVIDIA round low-precision GEMM intermediates differently, so the same order can still give different bits.
The problem is sharper at 4 bits. In the paper's test, moving Qwen3-8B-Base from 2 to 4 GPUs of tensor parallelism changed 18× more greedy output tokens under MXFP4 than under BF16. MXFP4, where each block of 32 values shares one power-of-two scale, is the format that makes training cheap — and in that test it was also the format whose results moved most when the GPU split changed.
CertiSum's insight is that a sum which never rounds is the same in every order — and MXFP4 sums often never round. A 4-bit element has very few significant bits, and its block scale is a power of two. Express each row of a dot product in its own power-of-two unit (half that row's smallest block scale) and every element becomes a small integer; multiply the two units and every product is an integer in one shared product unit. The board-fit test is then one line: if the sum of the absolute values of all products stays below 224, every partial sum, in any grouping, fits in FP32's 24-bit significand and no addition can round (the paper adds range checks on the unit, called exponent guards, so no value leaves FP32's normal exponent range). The bound uses absolute values because it must also cover partial sums formed before positive and negative terms cancel. On FineWeb-Edu, 99.2% of GEMM outputs needed at most 24 bits for exact accumulation in any order.
Checking that bound by computing every product would cost as much as the GEMM itself. Instead, while it quantizes each row, CertiSum records three statistics: the sum of absolute values (L), the largest absolute value (M) and the sum of squares (Q). The smallest of La·Mb, Ma·Lb and √(Qa·Qb) is a guaranteed upper bound on the absolute-product sum (the last term is the Cauchy–Schwarz inequality), so checking an output costs constant work, whatever the reduction length. If this row bound fails, CertiSum tries a tighter block bound (the same three terms per 32-element block), then a tight bound: a native MXFP4 GEMM run on the operands with their signs removed, which sums the absolute products directly. Each bound is evaluated with a rounding allowance, so a rounded check cannot certify a sum that could round. Only outputs that fail all three are recomputed. For attention and nearly all weight-gradient GEMMs, where certification rarely succeeds, CertiSum skips the native GEMM and runs exact kernels directly; those account for 46–68% of training-step GEMM FLOPs in the paper's runs.
Back at the shops, the uncertain tills are recounted on a wider board, and each till hands over its exact count, never a rounded one. CertiSum's repair works three ways: split the dot product into chunks short enough to certify and merge their sums exactly, accumulate in FP64 when a capacity check shows nothing can round, or fall back to a multi-word integer superaccumulator. The merge is what makes the result survive a change of GPU count. Partial sums from tensor-parallel shards, data-parallel replicas and micro-batches are kept as exact integers in the common unit and added without rounding, after a capacity check confirms the int32 or FP64 accumulator is wide enough, and integer addition is associative, so the result is the same for any split, merge order or reduction tree. One more rule keeps the products themselves fixed: MX blocks are assigned by an element's position in the full tensor, not within one GPU's shard, so re-sharding never changes which 32 elements share a scale. Attention gets the same treatment for both its weighted-value sum and its softmax denominator, the two quantities that blockwise attention accumulates across key blocks.
A worked example (illustrative). Hold three things fixed: a dot product of length K = 4,096, every block in both rows sharing one scale, and E2M1's largest magnitude, 6. In each row's unit (half its scale), each element is an integer of at most 12, so each product is at most 144 product units and the absolute-product sum is at most 4,096 × 144 = 589,824. FP32 holds every integer below 224 = 16,777,216 exactly, so even this worst case leaves about 28× headroom — no partial sum, in any order, can round. Now give one aligned block in each row a scale 32× larger than the rest: that block's elements reach 384, each of its 32 products reaches 147,456, and the worst case grows to 4,718,592 + 4,064 × 144 = 5,303,808, still under the limit. At a 64× gap, that one block alone contributes 32 × 768² = 18,874,368, past 224, so this worst-case reasoning can no longer vouch for the output, and CertiSum turns to its tighter bounds, which use the actual values, or to an exact recompute.
| Approach | Example systems | Same bits across | Reported cost | Source |
|---|---|---|---|---|
| Pin one reduction order | Megatron Core deterministic mode, batch-invariant kernels | Repeated runs (some layout changes); not vendors | 2% to 1.6–2.1×, varies by system | CertiSum Table 1 |
| Cross-hardware replay | OPEN-1B | Runs and hardware | 5.2–6.8× BF16 runtime | CertiSum Table 1 |
| Exact on every reduction | ReproBLAS, ExBLAS | Runs, layouts and hardware | Up to 16× slower GEMM (ExBLAS) | CertiSum Table 1 |
| Certify, repair the rest | CertiSum | Runs, DP/TP/CP layouts and vendors, for MXFP4 | +11.8–60.5% vs non-exact MXFP4 (larger shapes) | CertiSum abstract |
The price is a slowdown against unreproducible 4-bit training, not against the BF16 training most teams run today. At the larger evaluated shapes, CertiSum's step latency was 0.81–1.00× that of the faster ordinary BF16 reference and 0.59–0.89× that of deterministic BF16. At smaller shapes its overhead over the MXFP4 twin rose to 15.4–77.3%, and the advantage over BF16 can reverse, likely because certification and routing add a fixed cost per GEMM. The payoff shows when a run changes shape: in the paper's elastic-training test, the non-exact MXFP4 twin's loss diverged at step 41, right after the first layout change, and reached a 4.2% difference on FineWeb-Edu, while CertiSum matched its reference loss bits at all 200 steps. That is the guarantee reinforcement learning needs when an inference engine samples tokens and a training engine re-scores them with different kernels, and what a training audit needs when it re-runs a published job on other accelerators.
Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Mixed Precision Training & Inference
Related explainers
- OPEN-1B bitwise step replay — the other route to identical bits: pin every operation's order instead of proving order does not matter
- MarginGate batch-invariant decoding — the inference-side version of the problem, solved by re-checking only the risky decoding steps