OnQ is a post-training method from the paper "Right On Q" for recovering reasoning in 2-bit and 3-bit quantized LLMs. It first calibrates the quantized model block by block, then trains it on its own sampled responses with the full-precision model as teacher.

Why train a quantized model on its own responses?

A low-bit model drifts onto token sequences that a fixed training set never contains, and the mismatch grows with response length. Supervising the teacher's distribution on the student's own prefixes closes that gap, and the ablation attributes most of the gain to it.

What does Dynamic KL do differently from forward or reverse KL?

It sets a separate mix of forward and reverse KL for each candidate token using the weight p / (p + q), where p is the teacher's probability and q the student's. Underpredicted tokens are mostly corrected by forward KL and overpredicted ones by reverse KL.

How does OnQ relate to ordinary quantization-aware training?

It is a form of QAT, since the forward pass is quantized while the weights train. The difference is that it distills on student-generated prefixes rather than fixed response sequences. See [how quantization works](/tracks/llm-internals/quantization?step=process).

LLM· 2026-10-23

OnQ — on-policy distillation with Dynamic KL for 2-bit LLMs — What does it mean?

Jargon

QAT: Quantization-aware training: keep training the model while its forward pass already uses the low-bit weights, so the weights adapt to the rounding. See how the quantization process works.
Student and teacher: In distillation the student is the model being trained (here the 2-bit model) and the teacher is the frozen model whose output probabilities it is trained to match (here the original full-precision model).
On-policy: Training on text the student itself sampled, so the training prefixes are the ones it will meet at inference time, instead of text someone else wrote.
Forward KL and reverse KL: Two ways to measure the gap between the teacher's next-token probabilities and the student's. Forward KL weights the gap by the teacher's probabilities, so it punishes tokens the student underpredicts. Reverse KL weights it by the student's, so it punishes tokens the student overpredicts.
Block-wise calibration: A first, cheap stage that trains one transformer block at a time so its quantized output matches the full-precision block's output. It gives the student a working starting point.
RTN: Round-to-nearest: quantization with no training, where each weight is simply rounded to the closest value on the low-bit grid.
Truncation: A response that hits the token limit before it reaches a final answer, so the answer never arrives within the budget.

The news. On October 21, 2026, a team from the University of Michigan, UC Santa Barbara and Meta AI posted "Right On Q" (arXiv 2610.29378), which introduces OnQ, a post-training method for keeping reasoning intact after aggressive 2-bit and 3-bit weight quantization. On Qwen3-1.7B, 4B and 8B it reports the highest average over nine reasoning benchmarks among the methods the authors tested. Read the paper →

Picture teaching someone to drive a car whose steering wheel has only four positions. The usual recipe, fixed-response distillation, plays the instructor's recorded drive and asks the learner to copy it. Every training example is a spot on the instructor's road: a prompt plus a response that the full-precision model or a dataset already wrote. But a 2-bit model, like the learner with the stiff wheel, does not stay on that road. It drifts onto its own route, and at inference time it is conditioned on its own earlier tokens, which no recorded drive ever showed it. The paper calls this a mismatch between training prefixes and generation prefixes, and reports that the gap grows as responses get longer. In its analysis of Qwen3-1.7B at 2 bits on MATH-500, QAT baselines recover little of the full-precision accuracy on questions where the full-precision model writes long answers, and they often hit the token limit while the full-precision model finishes inside it. For the numeric side of this, see how weights are rounded onto a low-bit grid.

OnQ runs in two stages. Stage one, block-wise calibration, only gets the student onto the road. Starting from round-to-nearest (RTN) quantization with clipped per-group ranges, it trains one transformer block at a time so the block's output matches the full-precision block's output (a mean-squared-error loss, updating the weights, scales and zero points). The paper follows the Block-AP recipe from EfficientQAT, with 4,096 calibration windows of 2,048 tokens and two epochs per block, and shows that RTN alone severely degrades 2-bit accuracy. Stage two puts the instructor in the passenger seat. The quantized student samples its own response to a prompt, and the frozen full-precision teacher reads the same prefix and returns its next-token distribution at every position, including positions that follow the student's own earlier mistakes. The loss is averaged over every generated token, and training updates the weights and scales while the forward pass stays quantized. The weights use uniform asymmetric quantization with group size 128 and BF16 activations.

Which loss compares the two distributions? Both models output a probability for every candidate next token, and the paper observes that at the same prefix the calibrated student does two things at once: it underpredicts some candidates and overpredicts others, more often at high-entropy prefixes where many tokens are plausible. In its local-update analysis, forward KL moves more of the severely underpredicted candidates toward the teacher, and reverse KL moves more of the severely overpredicted ones, where severe means a probability ratio of at least 20. Candidate-wise Dynamic KL gives every candidate its own mix of the two, using the weight α = p / (p + q), where p is the teacher's probability for that token and q is the student's, computed at each prefix and treated as a constant for backpropagation. When the student underpredicts a token (p > q), α is above 1/2 and forward KL dominates; when it overpredicts (q > p), α is below 1/2 and reverse KL dominates. The paper adopts Dynamic KL from earlier work and evaluates a top-K approximation of the sum over the vocabulary. Back on the road, α is the instructor choosing which note to give at each spot: "you missed that turn" or "you are drifting toward the ditch".

A worked example of the weight (illustrative numbers, not from the paper). Take one prefix and three candidate next tokens. Token A: the teacher gives p = 0.40 and the student q = 0.05, so α = 0.40 / 0.45 ≈ 0.89, which is 0.89 forward KL and 0.11 reverse KL. The student has badly underpredicted a token the teacher likes, so the forward term does almost all the work. Token B: p = 0.02 and q = 0.30, so α = 0.02 / 0.32 ≈ 0.06, which is 0.06 forward KL and 0.94 reverse KL. The student is putting 15 times too much probability on a token the teacher considers unlikely, so the reverse term dominates. Token C: p = q = 0.10 gives α = 0.50, an even split, and since the two distributions agree there it adds no loss. A single fixed-direction loss would apply the same rule to all three, and that is the failure the paper's analysis points at.

Training signal (Qwen3-1.7B, 2-bit)	MATH-500	HumanEval+	MMLU-Redux	Source
Fixed responses, forward KL	59.00	39.63	52.28	Table 3
Fixed responses, Dynamic KL	61.00	42.07	52.68	Table 3
Own responses, Dynamic KL (OnQ)	69.40	48.78	55.75	Table 3

The ablation separates the two ideas. All three runs start from the same calibrated checkpoint and use the same number of response tokens. Switching from forward KL to Dynamic KL on fixed responses adds up to 2.44 points (the largest gap, on HumanEval+), but switching those responses to the student's own adds up to 8.40 more (on MATH-500, 61.00 to 69.40). So most of the gain comes from where the training prefixes come from, and the loss choice adds a smaller second step. Against other on-policy objectives on a shared start, Dynamic KL improves MATH-500 by 1.0 points over forward KL (GKD) and 4.2 over reverse KL (vanilla on-policy distillation).

One practical detail keeps the early steps from wasting supervision. Right after calibration the student repeats itself late in long answers: for Qwen3-1.7B at 2 bits, 6.7% of tokens are repeats in the first 4K response tokens and 21.1% beyond 8K, falling to about 1% at every position by step 100. Because the loss weights every token equally, long early rollouts would spend supervision on repeated text, so the response cap grows with training: 4,096 tokens for steps 1–50, 8,192 for steps 51–100 and 16,384 after that.

On results, OnQ posts the highest nine-benchmark average in all six model and bit-width settings (Qwen3-1.7B, 4B and 8B at 2 and 3 bits) and the top score on 48 of 54 benchmark-setting pairs. At 2 bits the Qwen3-8B average is 10.26 points above the strongest baseline, and the 2-bit Qwen3-4B beats every 2-bit baseline built on the twice-larger Qwen3-8B. The biggest wins land on long-horizon tasks such as AIME and LiveCodeBench. On Qwen3-4B LiveCodeBench, accuracy rises from 28.34% for ReasoningQAT to 46.64% while the mean response gets more than a fifth shorter. Treat this as one paper's evidence: one model family, weight-only quantization with BF16 activations, and comparisons against the baselines the authors ran. For where this sits among other methods, see modern quantization methods and what to quantize.

Goes deeper in: LLM Internals → Quantization → Modern Methods

Related explainers

Quantization-Aware Healing — also distills a low-bit student from a full-precision teacher, but is about which teacher to use.
DOPD: dual on-policy distillation — another on-policy distillation recipe, aimed at token-level signal routing.
CoPD: co-evolving policy distillation — on-policy distillation where specialist models teach each other.

Continue in trackQuantization: how modern methods keep a low-bit model usable

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based