Protect LLM training from silent corruption with 1.65–6.76% overhead — Silent data corruption in transformer training — What does it mean?

The news. On August 31 2026 a paper called TrainSDC appeared on arXiv, presenting what the authors describe as the first systematic study of silent-data-corruption vulnerability across the major Transformer interfaces, in both the forward and the backward pass. Its central finding is that the vulnerability is not spread evenly. Faults on the forward Q/K path create persistent deviations, while backward-pass risk tracks gradient exponent distributions rather than layer location. The authors evaluate on Llama 3.2-1B and Qwen3-0.6B under both sparse and dense fault injection, and report protection that keeps training close to fault-free behaviour at 1.65–6.76% runtime overhead. Read the paper →

Picture the payroll sheet again. Most slipped keystrokes are like that second row: a digit changes somewhere far to the right, the total moves by a cent, and nobody would ever notice or care. The dangerous slip is the one that moves the decimal point, because it does not change the number a little, it multiplies it. A floating-point number is built the same way. It has a sign, an exponent and a mantissa, where the mantissa carries the digits and the exponent sets the scale as a power of two. Flip a low mantissa bit and the value shifts in its last decimal place. Flip an exponent bit and the value jumps by a factor of two, or 256, or more.

That asymmetry is what makes silent data corruption a different problem from ordinary hardware failure. A fault that crashes the job announces itself: you restart from a recent checkpoint and lose only the work done since it. A fault that merely returns 1247.00 where 12.47 belonged is absorbed into the next matrix multiply and carries on. Training does not stop, it can just quietly get worse, and the bill may not arrive until weeks later, as a model that underperforms for no reason anyone can point at.

So the real question is not whether bits flip. At the scale of a modern training cluster they do. The question is which of the thousands of operations in a Transformer step will still be carrying the error many steps later — and TrainSDC's answer is that it depends on which interface the fault lands in.

On the forward pass, the paper reports that faults on the Q/K path create persistent deviations. That is the projection turning each token's embedding into the query and key vectors whose dot products become the attention scores. A corrupted query is not one bad number sitting in one place: it changes which tokens that position attends to, and the resulting wrong-attention output then flows up through the residual stream into every layer above. The tainted payroll row is exactly this, an early cell that later cells are computed from.

The backward pass follows a different and more counter-intuitive rule: risk tracks the gradient exponent distribution rather than the layer's position in the network. It is not that early layers or late layers are inherently fragile. What predicts the danger is how the gradient exponents are distributed at that point in the run, which is why the paper's backward-pass guard keys on the exponent rather than on depth.

What follows from that is a budget decision rather than a correctness one. You could catch every silent fault by computing everything twice and comparing the two answers, which is the classical approach and roughly doubles the arithmetic. TrainSDC instead spends its protection only where the study found that faults amplify: it recomputes the vulnerable Q/K path, monitors residual gains to catch anomalous forward propagation, and scales gradients according to exponent-aware risk. Each guard is aimed at one of the two failure shapes the study identified, which is why the bill comes to a few percent instead of a second training run.

Put numbers on the trade. Hold three things fixed: a 30-day pretraining run (illustrative — the paper's own measurements are on Llama 3.2-1B and Qwen3-0.6B, which are far smaller), the reported 1.65–6.76% runtime overhead, and checkpoints written every six hours.

Switching the protection on costs between 30 × 0.0165 ≈ 0.5 days and 30 × 0.0676 ≈ 2.0 days of extra wall-clock: twelve hours at the low end, a little over two days at the high end. Now price what that buys. A silent fault is never caught by the checkpoint mechanism, because nothing failed, so the six-hour cadence buys you nothing here; it is caught at evaluation, whenever that happens to be. If corruption entered eight days before anyone noticed, every checkpoint written in those eight days is suspect and the honest recovery is to roll back and retrain them. So the trade is half a day to two days of certain overhead against an eight-day loss that arrives at an unknown time. Whether that is worth it in expectation depends on how often faults actually strike your hardware and on how much of the damage the guards catch, neither of which this arithmetic settles. What the paper supplies is the part you cannot guess: a map of which interfaces amplify a fault, guards aimed at them, and a measured bound on what they cost.

The honest caveat is the model scale. The 1.65–6.76% band is measured on two sub-2B models, and the paper does not claim it holds at frontier scale, where the arithmetic, the memory traffic and the failure rate all change together.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Precision Formats