SignRot is a sign-adjusted Hadamard rotation for low-bit LLM quantization. It flips the signs of selected columns of the rotation so that a massive outlier channel shifts every rotated activation in the same direction, then uses asymmetric quantization to absorb that shift. It needs one calibration forward pass and adds only element-wise sign flips at inference.

Why does Hadamard rotation fail on massive outliers?

Each rotated output receives plus or minus the outlier value divided by the square root of the channel count, with the sign set by the Hadamard row. When one channel holds more energy than all the others combined, the outputs split into two clusters with an empty gap, so a 4-bit grid spends most of its 16 levels on values that never occur.

Does SignRot slow down inference?

Very little. On LLaMA2-7B on an RTX 3090 the paper measures +1.37% prefill latency, +0.34% per-token decode latency and +0.43% end to end, compared with QuaRot using the same asymmetric activation quantizer.

How does SignRot relate to SpinQuant and other learned rotations?

SpinQuant learns its rotation matrices, but at the FFN down-projection it still has to use an online Hadamard transform, because a learned dense matrix there would cost a full matrix multiply. SignRot changes only that Hadamard transform, so it can be added on top of QuaRot, SpinQuant or DartQuant.

LLM· 2026-11-14

SignRot for 4-bit LLM quantization — Sign-adjusted Hadamard rotation — What does it mean?

TL;DR

What is it: SignRot (arXiv 2611.16502, posted November 12, 2026) is a sign-adjusted Hadamard rotation: a fixed set of sign flips added to the rotation that quantizers such as QuaRot apply to activations before rounding them to 4 bits.
Why it’s needed: Rotation is how 4-bit activation quantization survives outlier channels, but when one massive outlier dominates a token the rotation itself splits that token into two far-apart clusters, and this happens most in the FFN down-projection, where the rotation has to run at inference time.
vs previous: Plain randomized Hadamard rotation leaves those two clusters with an empty middle that wastes quantization levels; SignRot flips column signs so the outlier pushes every output the same way, then an asymmetric zero-point slides the 4-bit grid onto the one remaining cluster.

Jargon

Hadamard rotation: Multiplying a vector by a Hadamard matrix, whose entries are all plus or minus the same value. Every output becomes a signed mix of every input, so one large channel gets shared across all channels. Randomized versions also flip input signs at random. KVarN uses the same idea on the KV cache: KVarN explainer.
Massive outlier: An activation channel whose value is hundreds or thousands of times larger than the rest of its token. The paper's example token has one channel at 1510 while the others are mostly small.
Bimodal distribution: A spread of values with two separate peaks and a gap between them, instead of one bell curve.
Symmetric vs asymmetric quantization: Symmetric quantization centers its grid on zero and sizes it by the largest absolute value. Asymmetric quantization adds a zero-point so the grid can cover any interval, for example 13 to 16, instead of being forced to straddle zero.
Per-token activation quantization: Each token's activation vector gets its own scale (and zero-point), computed on the fly at inference time.
FFN down-projection: The last matrix of a transformer's feed-forward block. Its input comes right after a nonlinearity, so a rotation there cannot be pre-multiplied into the previous weights and must run online.
QuaRot, SpinQuant, DartQuant: Rotation-based post-training quantization methods. QuaRot uses fixed randomized Hadamard rotations; SpinQuant and DartQuant learn rotations, but still use online Hadamard rotations at positions like the down-projection.
Perplexity: How surprised a model is by real text, on average; lower is better. WikiText2 perplexity is the standard quick check that quantization did not damage a model.

The news. On November 12, 2026, a team from Hanyang University and UC Santa Barbara posted SignRot, a drop-in change for any Hadamard-based quantizer (QuaRot, SpinQuant, DartQuant). It needs one calibration forward pass and adds only element-wise sign flips at inference. On LLaMA3-70B, QuaRot with GPTQ goes from 6.60 to 5.88 WikiText2 perplexity and from 70.45 to 72.73 average zero-shot accuracy, beating SpinQuant's 6.38. Part of that gain comes from the asymmetric activation quantizer that SignRot requires: QuaRot with the same asymmetric quantizer already reaches 6.16. Read the paper →

Picture the photographer from the card above. The camera has only 16 columns of pixels, and the frame must hold every guest. A 4-bit quantizer is that camera: it gets 16 levels, spaced evenly from the smallest to the largest value of one token's activation vector, and every value is snapped to the nearest level (the scale-and-round step). One giant channel stretches the frame for everyone, which is the outlier problem: the normal values all end up sharing a few levels.

The standard fix is rotation. A Hadamard matrix has every entry equal to plus or minus 1/√n, so after multiplying by it each output channel is a signed sum of all n input channels. The giant channel's energy is spread across all of them, and for an ordinary token the result looks like one bell curve that 16 levels can cover well. Rotation works because many small inputs, mixed with random signs, average into one bell shape — and that argument breaks when a single input is most of the total.

Outliers force INT8 range to span –60 to 60 · most of the 256 grid slots fall on empty space

Here is how it breaks. If one channel carries outlier value M, every output receives a share of ±M/√n from it, and the sign is whatever that row of the Hadamard matrix says: plus for half the outputs, minus for the other half. The guests split into two crowds at opposite walls. The paper shows the split appears when the outlier channel holds more energy than all other channels combined. Its real example is a token entering the down-projection of layer 2 in LLaMA2-7B: n = 11,008 and M = 1510, so the shift is 1510/√11,008 ≈ 14.39. After rotation every output lies between 13.13 and 15.63 in magnitude, so no output at all falls between −13.13 and +13.13. The frame must still span about −15.6 to +15.6, so most of its 16 levels cover an empty middle.

The down-projection is also where this hurts most, and where rotation cannot be swapped for a learned one. Massive outliers concentrate there, and because the rotation cannot be folded into the previous weights it must run at inference time, using the fast Walsh–Hadamard transform (a way to apply a Hadamard matrix in n·log n steps instead of a full n × n matrix multiply). A learned dense rotation, the SpinQuant approach, would need a full matrix multiply in that position, so it cannot simply replace the Hadamard matrix there.

SignRot asks one crowd to walk across the hall. During one calibration forward pass it finds the channel with the largest absolute activation, which stays the same channel across inputs: in the first massive-outlier layer of every model tested, that channel held 87–99.9% of the outlier tokens' energy. It then multiplies each column j of the Hadamard matrix by the sign of that row's entry, so the outlier row becomes all +1/√n and the outlier adds the same +M/√n to every output: one crowd, not two. A diagonal sign matrix is its own inverse, so the matching flip is folded into the weights offline. At inference the only new work is an element-wise sign flip.

The flip alone does nothing for a symmetric quantizer. Flipping signs keeps every absolute value, and a symmetric grid is sized by the largest absolute value, so the frame still spans −15.6 to +15.6. The gain comes from pairing the flip with an asymmetric zero-point that pans the frame onto the one crowd. The paper's ablation confirms it: with symmetric activations, SignRot changes LLaMA2-7B average accuracy from 66.54 to 66.33. The zero-point costs little in the matrix multiply, because its correction is the zero-point times a precomputed column sum of the weights. Measured end to end on LLaMA2-7B on an RTX 3090, SignRot adds 0.43% latency.

Worked example (illustrative arithmetic on the paper's token, no clipping). Hold three things fixed: the LLaMA2-7B token above, 4-bit per-token quantization, and the reported range of 13.13 to 15.63 in magnitude. After plain rotation, a symmetric grid with levels from −7 to +7 steps covers ±15.63, so one step is 15.63 / 7 ≈ 2.23 and a value can be off by up to about 1.12. The occupied values only reach the levels at ±13.40 and ±15.63, so 11 of the 15 levels sit in the empty middle. Asymmetric quantization without the flip does not help much: the range is still −15.63 to +15.63, and 31.26 / 15 ≈ 2.08 per step. After the flip plus a zero-point, the range is 13.13 to 15.63, so one step is 2.50 / 15 ≈ 0.167 and the worst rounding error is about 0.083 — roughly 13× finer. Neither the flip nor the zero-point helps much alone; together they shrink this token's step from about 2.23 to about 0.167.

LLaMA3-70B, GPTQ weights	Activation quantizer	WikiText2 PPL ↓	Avg. 0-shot ↑	Source
FP16 baseline	none	2.86	79.95	Table 1
QuaRot	symmetric	6.60	70.45	Table 1
QuaRot	asymmetric	6.16	72.28	Table 4, single run
QuaRot + SignRot	asymmetric	5.88	72.73	Table 1
SpinQuant (learned rotation)	asymmetric	6.38	71.15	Table 1

The gains are real but uneven, which tells you where to expect them. On LLaMA2 with round-to-nearest weights (plain rounding, no error compensation), the accuracy gain falls from 2.70 points at 7B to 0.39 points at 70B, because bigger models lose less to quantization in the first place. On top of DartQuant, which already learns its rotations, LLaMA3-70B accuracy moves from 76.39 to 76.02, a small drop. The method also fixes only one channel per layer: in the last layers of Mistral-7B-v0.3 and Qwen3-30B-A3B the largest channel changes from token to token, so those tokens keep their split. SignRot is a targeted repair for one known failure of rotation — a single fixed giant channel at the down-projection — not a replacement for rotation, which is also why it composes with GPTQ (how GPTQ compensates for rounding error) instead of competing with it.

Goes deeper in: LLM Internals → Quantization → The Outlier Problem

Related explainers

KVarN squeezes the KV cache to 2 bits — Hadamard rotation — the same rotation trick applied to the KV cache instead of activations
Decompose W4A4 quantization error into correctable components — which part of W4A4 error a weight solver can absorb, and which needs a transform

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based