The news. On July 9, 2026, researchers posted a study (arXiv 2607.08734) arguing that the very numbers teams use to sign off on a quantized model — accuracy and perplexity — can stay flat while the model's behavior shifts underneath them. They analyze quantization as a structural operator on the attention weights, measure the distortion layer by layer from 8-bit down to 2-bit, and report non-linear breakpoints at low bit-widths plus higher sensitivity in the query and key projections than in value and output. Their proposed fix is correctness agreement: a decision-level metric for how often the base and quantized model are right on the same inputs. Read the paper →
Picture two students who sat the same exam and both scored 80%. On paper they look identical. But lay their answer sheets side by side and a different picture appears: they aced different questions. One nailed the geometry and fumbled the history; the other did the reverse. Their scores agree; their answers don't. A quantized model is the second student — a stand-in that studied from compressed notes. Check only its final score and it looks like a faithful copy of the original. Line up its answers one by one and you find it is getting a different set of them right.
That side-by-side answer check is exactly what the paper's correctness agreement measures. Accuracy is an average — it asks how many a model gets right, never which ones. Two models can both land at 80% while overlapping on far fewer than 80% of the individual items, because the new mistakes and the new saves cancel out in the total. Perplexity has the same blind spot: it is a smooth average over tokens, and an average is precisely where a small, one-directional shift disappears. Correctness agreement throws the totals away and compares the models input by input, so a change that accuracy hides shows up as a drop in how often the two are right together.
| Metric | Question it answers | What it averages over | Sees decision drift? |
|---|---|---|---|
| Accuracy | how many answers are right | all items → one score | No — misses and saves cancel |
| Perplexity | how confident on held-out text | all tokens → one number | No — a smooth average hides small shifts |
| Correctness agreement | right on the same inputs | nothing — compared item by item | Yes — the paper's proposal |
So where does the drift come from? The authors treat quantization not as harmless rounding but as a structural operator that distorts the attention weights, and they measure that distortion one layer at a time. The result is uneven. The query and key projections — the parts of attention that decide where the model looks — come out more sensitive to the rounding than the value and output projections that carry what it reads out. In the exam metaphor, that maps to the knowing-where-to-find-the-answer questions drifting more than the pure-recall ones. Because the query and key are what set where attention goes, distorting them more can shift what the model decides even when the average score holds.
That uneven sensitivity is the practical hook, because it tells you which parts to compress hardest and which to protect. It is the same lesson the curriculum teaches about layer position: some pieces of a network tolerate coarse numbers, others break the whole output when you round them.
Here is why "the accuracy looked fine" is such a trap. Suppose the full model answers a 1,000-question benchmark and gets 900 right (illustrative numbers, to show the mechanism). Quantize it to 8-bit and it still gets 900 right — but 30 of the questions it now misses are ones it used to get, and 30 it used to miss it now gets. Accuracy is unchanged at 900, yet 60 individual decisions flipped: correctness agreement is 870 out of 900, not a perfect copy. Push down toward 2-bit and the paper's sharpest finding bites: the distortion does not grow smoothly but crosses a non-linear breakpoint at low bit-widths. That implies agreement can drop off a cliff there even while a coarse accuracy read still looks survivable — so the bit-width where a model looks fine and the bit-width where it behaves the same need not be the same number.
Goes deeper in: LLM Internals → Quantization → What to Quantize
Related explainers
- Gemma 4 QAT — Quantization-Aware Training — the other route: instead of measuring drift after the fact, teach the model about the coarse grid during training so its decisions survive compression.
- SOP — hardware-aware per-layer PTQ at FP6 — acts on the same "some layers are more sensitive than others" insight this paper quantifies, choosing precision per layer.
- Ternary Mamba — quantization-aware training to ternary weights — pushing toward the extreme low-bit end, exactly where this paper's breakpoint warning matters most.