Spend quantization bits globally instead of repairing critical layers — Global quantization granularity — What does it mean?

The news. On September 1, 2026, researchers published The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally. They raised one quantized layer at a time back to 8-bit across 9 open-weight models in 4 architecture families, measured how much accuracy each layer recovered, and used that as ground truth to test three intuitive stories about where quantization damage lives: in task circuits (the subnetworks a task runs through), in the layers where the model does most of its computing, and in weight statistics. None of the three predicted which layers benefited from restored precision. Read the paper →

Picture the kitchen. Every station has an amount to write down, and the only way to write it down is to round it to the nearest notch on a scale. Quantization is that rounding, and the number of notches is the weight bit width — a 4-bit weight has 16 notches, full stop. What you can still choose is how much range one scale has to cover. Hand the whole kitchen a single scale and those 16 notches have to stretch across every station's range at once, so each notch is coarse and every amount lands a long way from one. Give each small group of stations its own scale and the same 16 notches only have to cover that group's much narrower range, so every amount lands close to a notch. That second choice is quantization granularity — the group size in the paper's terms, where group-128 means one scale per 128 weights.

Now suppose you can afford slightly more precision than plain 4-bit. There are two places to put it, and they are not the same purchase. You can spend it locally: pick the handful of layers you believe matter most and restore them to 8-bit, which is the lab balance for the one station you guessed. Or you can spend it globally: shrink the group size everywhere, so every station gets a scale sized to its own range. The paper's question is which of the two buys more accuracy for the same spend.

Two different things get called mixed precision, and only one of them is under test here. The first is component-level: weights at 4-bit, the KV cache at 8-bit, activations left in FP16, because those components really do have different tolerances. That is what what to quantize teaches, and this paper does not touch it. The second is layer-level selective repair: inside the weights, score the layers with a cheap proxy and hand the extra bits to the few that score highest — the ones carrying outliers, say, or the ones doing the heaviest computation. That second heuristic is the one the paper puts to a causal test, and it rests on an assumption worth stating out loud: that quantization damage is concentrated in a few layers you can identify in advance.

The paper's contribution is refusing to take a proxy's word for it. Instead of ranking layers by a cheap statistic, the authors ran a causal mixed-precision intervention: raise exactly one layer back to 8-bit, measure the accuracy that one layer recovers, put it back, move to the next. Run that across 9 open-weight models in 4 architecture families and you have a measured ranking of which layers actually pay, not a guess at it. Against that ranking they tested the three intuitive stories — damage in task circuits, damage where the model computes, damage visible in weight statistics — and none of the three predicted which layers benefited from restored precision.

What the intervention found instead is that recovery is spread out. For 8 of the 9 models, recovering 75% of the accuracy gap took roughly half the layers, each contributing a little, rather than a critical few carrying the loss. Only one model, Qwen3-8B, showed the sharply concentrated pattern the heuristic assumes. Back in the kitchen: no single station is ruining the batch, every station is off by a pinch, and one lab balance fixes one pinch out of many.

The arithmetic shows why a matched budget is the only fair comparison, because the two axes cost wildly different amounts. Hold three things fixed (illustrative): 4-bit weights, FP16 scale factors, and a 4096-wide row. With one scale for the whole row, the scale overhead is 16 bits spread over 4096 weights, about 0.004 bits per weight, so the model costs roughly 4.004 bits per weight. Move to group-128 and that same 16-bit scale now covers only 128 weights: 16 divided by 128 is 0.125 bits per weight, for a total near 4.125. Finer granularity everywhere therefore costs about 0.12 extra bits per weight, roughly 3% more memory. Price the local option at the paper's own recovery figure and it is a different order of spend: raising half the layers from 4 bits to 8 adds 4 bits to half the weights, averaging about 6 bits per weight, a 50% increase. So when the paper does hold the budget matched, granularity gets very much more of it — and it wins on all 8 group-128-compatible models, by 21 to 52 points. The ninth, OpenLLaMA, is excluded because its width rules out group-128 rather than because it disagreed.

Two limits keep this from being a universal rule, and the paper states both. The result holds within this budget setting — a small extra precision budget on top of a low-bit baseline — and push the budget higher and the question dissolves, because the authors report 8-bit is already near-lossless in their evaluation across RTN, GPTQ and AWQ. The residual damage is budget-limited, not structural. The location of peak recovery does correlate with architecture within a family, though not across families, so a per-family observation is not a per-model rule. The broader lesson is methodological: a cheap signal that correlates with quantization damage does not necessarily identify where restoring precision helps, and only a causal intervention tells you the difference.

Goes deeper in: LLM Internals → Quantization → What to Quantize

Related explainers

• QCA — Outlier injection for PTQ — the outlier story this paper's causal test puts to the question.
• KronQ — Kronecker-factored Hessian — a sensitivity proxy of exactly the kind the intervention checks.
• SOP — PTQ where FP6 beats FP8 — the other axis of the same budget question, format instead of granularity.