RoleQ is a quantization framework that converts an existing LLM checkpoint to 1.59-bit ternary weights in a few GPU-hours. Its key idea is to place scales by the role of each vector in the Transformer instead of by matrix layout.

Why does scale placement matter most for ternary weights?

With codes limited to -1, 0 and +1, the scale is a unit's only continuous degree of freedom. The paper reports the benefit is largest at ternary and persists to the largest tested model in each family.

How does RoleQ relate to GPTQ and rotations?

RoleQ-T uses a learned rotation to tame outliers, a ternary adaptation of GPTQ to fit codes, and brief scale distillation. The placement rule sits on top of them and depends on the rotation to show its advantage.

LLM· 2026-10-18

RoleQ converts LLMs to 1.59-bit ternary — Role-aligned scales — What does it mean?

Q: What does role-aligned scale placement mean?

Vectors that read from the residual stream get one scale per output unit, and vectors that write to it get one scale per input unit. The rule needs no search, calibration or stored per-layer choice.

Jargon

Ternary weights: Weights restricted to -1, 0 or +1, about 1.59 bits each, so a matrix product needs only additions and subtractions. See how numbers shrink.
Residual stream: The running vector that every transformer layer reads from and adds its output to. It is the shared channel the RoleQ rule is built around.
Scale (per-OC, per-IC): The number that multiplies a group of codes to rebuild real weights. Per-OC means one per output channel (matrix row); per-IC means one per input channel (matrix column).
Post-training quantization (PTQ): Shrinking an already-trained model with a short calibration step instead of training it again. See the quantization process.
GPTQ: A PTQ solver that quantizes a matrix one column at a time and pushes each rounding error onto the columns not yet quantized. See modern methods.
Fisher saliency: An estimate of how strongly changing a unit's weights would change the loss. RoleQ uses it to pick which units get extra bits.
Perplexity: How surprised the model is by held-out text; lower is better. It can look fine even when knowledge is lost.
MMLU: A four-choice knowledge benchmark, so random guessing scores about 25%.

The news. On October 15, 2026, Johannes Knittel, Alec Helbling and Hanspeter Pfister posted RoleQ, a framework that converts existing checkpoints to 1.59 bits per weight in a few GPU-hours rather than by training from scratch. On Qwen3-8B-Base the full recipe reaches 11.33 perplexity, 60.3 zero-shot accuracy and 47.6 MMLU. Read the paper →

Picture a mixing desk with one shared bus. Every block of the transformer is a channel strip. Some strips listen to the bus: the attention query, key and value rows and the MLP's input-side rows. Others send onto the bus: the attention output columns and the MLP's output-side columns. The bus is the residual stream, the running vector that every layer reads from and adds to. On a well-built desk each listener and each sender has its own volume knob, so turning one strip never changes the loudness of another.

Quantization splits each weight into a small code and a scale (see how numbers shrink). With ternary weights the code is only -1, 0 or +1, so the scale is the one continuous knob a unit has left. Conventional formats attach one scale per matrix row, chosen by how the matrix is stored. For a listener a row is one unit, so that works. For a sender a row is a residual coordinate that many senders share, so a per-row scale ties unrelated units to a single knob. RoleQ's rule fixes this: vectors that read from the residual stream get one scale per output unit, and vectors that write to it get one scale per input unit. The role decides the axis, so there is no search, no calibration pass and no stored flag per layer. A writer's scale sits on the input side, so at inference it simply rescales the incoming activations once per token, at a cost that grows with the input width.

Three more pieces turn the rule into a full conversion recipe. First, a rotation of the residual basis spreads outliers (see the outlier problem); a random Hadamard rotation alone lowers 1.7B ternary perplexity from 1245.9 to 119.8, and RoleQ's scale axis is untouched by it. Second, a ternary variant of GPTQ (see modern methods) fits each vector's scale and codes with an exact formula and carries rounding error forward. Third, scale distillation retrains only the scales and norm gains, 0.02–0.08% of the parameters, against the full-precision model's outputs while the codes stay frozen. At 8B that last stage takes about 2 hours on one A100.

Scale placement (Qwen3-14B-Base, ternary)	One scale per	WikiText2 perplexity
Full precision (reference)	not applicable	6.38
Per-OC (conventional)	matrix row	37.89
Per-IC	matrix column	42.60
RoleQ	output unit (readers), input unit (writers)	25.25

Hold three things fixed: Qwen3-14B-Base, ternary codes at 1.59 bits per weight, and the same solver, rotation and seeds, as in the paper's matched-settings comparison. Change only where the scale sits. The full-precision model scores 6.38 WikiText2 perplexity. Per-row scales give 37.89, per-column scales give 42.60, and role-aligned scales give 25.25, about a third lower than the conventional per-row choice. The sharper gap is on knowledge: per-row scales land at 25.4 on MMLU, which is chance for a four-choice test, while role-aligned scales reach 38.5 at the same bit count. The bit count did not change; only the placement of the scales did.

Ternary conversion still loses facts first, so RoleQ optionally adds a sparse second ternary plane to a few chosen units, which keeps every matrix product multiplication-free. Which units? Ranking by raw quantization error can do worse than random choice at a tight bit budget, so RoleQ ranks by per-unit Fisher saliency, an estimate of how much each unit's error moves the loss. At 8B, the full recipe with a single ternary plane per weight reaches 47.6 MMLU at 1.59 bits; adding planes for 0.57 extra bits per weight (2.16 total) lifts it to 62.8, against 76.8 at full precision, so a fraction of a bit buys back about half of the MMLU that conversion cost. The paper reports these on Qwen3 and Llama models only up to 14B, so treat the result as evidence about post-training ternary conversion at that scale, not about every architecture.

Goes deeper in: LLM Internals → Quantization → Modern Methods

Related explainers

Trellis-coded weight quantization — another way to squeeze weights below four bits
Ternary Mamba quantization-aware training — the training-based route to ternary weights that RoleQ avoids
Global quantization granularity — how group size choices change what survives

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based