Fisher quantization screen paper — Predicted KL vs quantize-and-measure — What does it mean?

The news. On November 9, 2026, Dezhen Lu (University of Tübingen) and Zixiang Ni (Xi'an Jiaotong University) posted How Far Does Second-Order Linear Response Predict Quantization Mismatch? Pre-Registered Measurements on Five Language Models. They tested the standard Fisher screen on Qwen3-8B, Qwen3-4B, OLMo-2-7B, Qwen3-32B and the mixture-of-experts Qwen3-30B-A3B, with channel-wise int8, per-tensor fp8 and group-wise int4 weights, and froze every prediction before its GPU run. A trimmed average put 10 of 12 cells on Qwen3-4B and OLMo-2 within 15% of measured KL. A screener built from the findings then met all its pre-registered criteria on an unseen model family, Mistral-7B-v0.3. Read the paper →

Picture a moving crew deciding how much padding each box gets. Thin wrap is cheap and takes little room in the truck; a wooden crate protects almost anything but takes the most room. One crew packs the truck several ways, drives each version over the same bumpy road and counts the breakage. That is honest, but it costs one trip per packing plan. A second crew inspects every box once, writes a fragility score on a sticker, and then estimates breakage for any plan with arithmetic: fragility times the size of the jolt, squared, added up over the boxes. The Fisher screen is the second crew: it inspects the model once and prices every format plan with arithmetic.

In model terms, choosing a format adds a known error δ to every weight, because quantization rounds each weight onto a coarser grid; int4 usually introduces larger rounding errors than int8. What you do not know is how much the output cares about each error. The second-order expansion answers that: the KL between the original and the quantized model is approximately one half of δᵀFδ, where F is the Fisher information, built from the gradient of the log-likelihood on sequences the model sampled itself. The gradients cost one forward and one backward pass per calibration sequence. After that single pass, every candidate format costs a dot product per tensor, not a new model. The same expansion sits under Optimal Brain Surgeon, HAWQ-style bit allocation and GPTQ-family quantizers (modern methods); the paper's question is how far the number can be trusted.

Rankings held; the absolute number needed a trimmed average. Across a 16-fold range of perturbation sizes, int4 group sizes and rounding modes, relative predictions held: switching int4 to stochastic rounding raised the measured KL by 2.25× against a predicted 2.28×. The absolute number is harder because the per-sequence score is heavy-tailed: the top 5% of sequences carried up to 24% of the prediction. In the truck, that is a handful of boxes whose stickers dominate the total, so a plain average says more about those boxes than about a typical load. The paper therefore reports a trimmed mean that drops the top 5%, and a screener should treat its output as a ranking first and an absolute number second.

The sharpest finding is about the inspector's own scale. Computing the per-sequence gradients in bf16 can make the prediction four times too high. For most sequences bf16 and fp32 agree, but in a few the early-layer gradients blow up; in one sample set, one sequence's bf16 score came out more than 1,500 times its fp32 score, and squaring carries that into the average. The authors traced the error to bf16 arithmetic in the lower half of the backward pass of Qwen3-8B: running layers 0 to 15 of the backward in fp32 cut the error at the bottom layer to 0.017 of its all-bf16 value, while fp32 in the upper layers alone left it at 0.96. The recipe is short: compute the screen's gradients in fp32, or check a bf16 estimate against fp32 sequence by sequence.

The cheap diagonal form assumes rounding errors point in random directions; at 32B, in one tensor, they did not. Summing fragility times jolt squared box by box is right only if the jolts hit from unrelated sides, so they do not add up along the direction the output is most sensitive to. On Qwen3-32B with int4, the measured KL was 5.18× the diagonal prediction. Post hoc, almost all of the gap sat in one tensor, the last layer's down projection (the output matrix of its feed-forward block), and inside it in two of its 25,600 input columns, whose rounding errors lined up with the gradient. The fix costs nothing extra: the same pass records each tensor's projection of the gradient on its rounding error, which gives the tensor-block form, exact within each tensor, and the screener flags any tensor whose block prediction is far above its diagonal one and that carries a real share of the predicted KL. A flagged tensor is never assigned that format. Stochastic rounding of the same tensor removed the alignment.

With prices for every box, the crew can plan against a budget. The screener starts every decoder layer at int4 and repeatedly upgrades the layer that buys the largest drop in predicted KL per added byte, from int4 to int8 and then to bf16. It plans against the upper edge of a band, not the point prediction, where the band is built from the measured-to-predicted ratios of 17 archived configurations, which ran from 0.77 to 1.46. At the loosest budget on Mistral-7B-v0.3 it kept 12 of 32 layers at int4, nine of them a contiguous block in layers 21 to 29, and moved the rest, including the first and last layers, to int8. That matches the usual rule that the first and last layers are the sensitive ones, but here the rule is measured per model instead of assumed.

Here is what the arithmetic saves, with the model, the ladder and the calibration set held fixed: Mistral-7B-v0.3, three rungs per decoder layer (int4, int8, bf16), and 256 calibration sequences. With 32 layers there are 3³², about 1.9 × 10¹⁵, possible plans. In the paper, checking six configurations against exact KL took 8.7 GPU-minutes, about 1.5 minutes each (our division), so quantize-and-measure can only ever test a tiny sample of them. The screen took one 8.0 GPU-minute pass and then priced every plan as a sum. For a budget of a quarter of uniform int4's predicted KL (2.71 nats), it chose 7.28 bits per weight, predicted 1.64 nats and measured 1.74 nats, a ratio of 1.06 and well inside the budget. Its throughput model predicted 4,244 tokens per second in vLLM against 4,226 measured, and accuracy on LAMBADA, a word-prediction benchmark, was 74.5% against 74.8% for bf16.

Three limits are worth keeping in view. The study covers four dense models from two families (Qwen3 and OLMo-2) plus one mixture-of-experts model, on one GPU type (GH200); the screener was tested on one further dense model, Mistral-7B-v0.3. On that mixture-of-experts model the quadratic form missed badly, with int8 measuring about 4.1 times its prediction, because expert routing flips are not a smooth response; the screener deliberately stays away from such models. On Mistral-7B-v0.3 no tensor was flagged and the cheap diagonal form also landed in the band, so the block form's advantage there is a guard against the 32B case, not a measured gain. And with only eight calibration sequences the allocator was over-optimistic by 1.16 to 1.44 times, because it keeps at int4 exactly the layers that happened to look safe on those few sequences; it needs a few hundred. The design rule: use a Fisher screen to rank and to plan, compute it in fp32, report a band, and check int4 against a measured KL on the target model.

Goes deeper in: LLM Internals → Quantization → What to Quantize

Related explainers

• KronQ: Kronecker-factored Hessian — uses second-order sensitivity to decide how to round; the Fisher screen uses it to decide which format each layer gets.
• SOP: hardware-aware per-layer PTQ at FP6 — another per-layer format choice, picked by a quantizer rather than predicted in advance.
• Lost in Rounding: stochastic rounding for BF16 state — the same lesson from the other side: bf16 rounding error can compound, and stochastic rounding can break the pattern.
• Quantization study: correctness agreement — why a quality budget should be checked on outputs, not only on an aggregate score.