The news. In July 2026, researchers released KronQ, a COLM 2026 paper on post-training quantization for LLMs. Its argument: second-order methods like GPTQ over-focus on input activation statistics, so it adds gradient covariance to make the quantization loss account for both the input and the output side of every weight. On LLaMA-3-70B at 2-bit weight-only, GPTQ and its variant GPTAQ blow past 2000 perplexity — noise — while KronQ holds at 7.93. Read the paper →
Picture an audio engineer mixing a song for a tiny phone speaker that can only play four fixed volume levels — no in-between. Every instrument's fader has to snap to one of those four notches. A rushed engineer sets each notch by how loud that instrument plays on its own: the booming bass gets careful treatment, the quiet triangle gets rounded off. But a quiet instrument can still be load-bearing — the triangle that lands the chorus matters more to what the listener finally hears than its solo volume ever suggests. Judge every fader by input loudness alone and you'll crush exactly the parts the song leans on.
That rushed engineer is GPTQ, today's standard for rounding a trained model onto a low-bit grid. GPTQ is a second-order method — it uses a Hessian to decide how carefully to round each weight — but it builds that Hessian from the input activation statistics only. At 4-bit that's good enough. At 2-bit, where each weight has just four possible values and the rounding error is brutal, the input side stops being the whole story: some weights are quiet on input yet swing the model's final output, and GPTQ's input-only score misjudges them. KronQ's fix is to build the quantization objective from both sides — how loud a weight's inputs are and how much its rounding moves the final loss — adding the missing output-side signal, the gradient covariance.
The catch is that the "true" sensitivity is a full input-by-output interaction, and for a 70-billion-parameter model that matrix is astronomically large to form directly. KronQ makes it tractable by writing the Hessian as a Kronecker product — one small input-covariance matrix multiplied by one small output-covariance matrix — so the two-sided score stays small enough to compute instead of forming a matrix the size of the whole layer. It reuses the same idea for outliers, too: GPTQ-style incoherence processing rotates the input side to spread out extreme values, and KronQ makes that rotation bidirectional, extending it to the output dimension using the gradient covariance.
The same two-sided view then decides where the bits go. KronQ reads a per-layer sensitivity from the traces of its gradient and activation Hessians, and spends more bits on the layers that need them — a mixed-precision budget set by measured impact rather than by position. The layers whose errors ripple furthest keep their precision; the robust middle layers absorb the deepest cuts — the same instinct a good mixing engineer has about which tracks to touch last.
Put numbers on it. Take LLaMA-3-70B: at 16-bit each weight is 2 bytes, so the weights alone are about ~140 GB (from the parameter count). Push every weight to 2-bit and that drops to roughly ~17.5 GB — an 8× shrink on the weights — small enough to change what hardware can serve the model. The trouble is quality: quantized GPTQ's way, WikiText-2 perplexity blows past 2000, meaning the 2-bit model has degenerated into gibberish. Rounded KronQ's way — scoring the rounding error by input and output sensitivity — the same 2-bit LLaMA-3-70B holds perplexity at 7.93: a usable model, not gibberish. Same 8× compression; the difference comes down to how the rounding error was scored — by input sensitivity alone, or by input and output together.
| How it decides rounding error | Signal it uses | 2-bit perplexity · LLaMA-3-70B |
|---|---|---|
| Round-to-nearest (RTN) | nothing — snap to the closest grid point | collapses at 2-bit (well-known) |
| GPTQ / GPTAQ | input activation statistics (one-sided Hessian) | >2000 (paper) |
| KronQ | input and output sensitivity (gradient covariance) | 7.93 (paper) |
The honest caveat is that KronQ is still post-training — it needs a calibration pass with a backward pass to estimate those covariances, so it does a little more work than a purely input-side method, and the gains are reported at the extreme 2-bit end, where the older methods had the most room to fail. But that is exactly the frontier that matters: at 2-bit the cheaper input-only methods fall short, and on LLaMA-3-70B the output side of the sensitivity is what carries a 2-bit model from noise back to a usable 7.93.
Goes deeper in: LLM Internals → Quantization → Modern Methods
Related explainers
- Ternary Mamba — ternary quantization-aware training — the other way to survive extreme low-bit: train on the grid instead of scoring the rounding after the fact
- UFP4 — E2M1 shrinkage bias — a different low-bit failure mode, where the number format itself rounds toward zero
- QCA paper — outlier injection across AWQ/GPTQ/GGUF — the block-scaling mechanics of GPTQ-family methods, and how one outlier can hijack them