**Multi-teacher Adaptive On-policy Self-Distillation.** The model learns from its own sampled responses, and for each token a teacher is selected from a pool of seven copies of the model that saw different slices of a reference solution.

Why would every token need a different teacher?

In the paper's pilot, the best teacher at two positions of the same response matched only about half the time for Qwen3-1.7B. A teacher that helps at one token can be a poor fit at another, so one fixed teacher per response leaves supervision on the table.

How is it different from fixed-teacher OPSD and Paint?

Fixed-teacher OPSD uses one full-reference teacher for all tokens. Paint adapts the reference exposure, but per response. MTA-OPSD adapts per token, using a **majority vote and then the largest KL** from the student.

What results does the paper report?

On AMC23, AIME24 and AIME25 with Qwen3 models, MTA-OPSD had the best average at 1.7B, 4B and 8B, with gains over the base model of +5.9, +1.5 and +2.8 points. The evidence covers one model family and math reasoning only.

LLM· 2026-11-01

Every Token Needs Its Own Teacher (MTA-OPSD) — Per-token teacher selection — What does it mean?

Jargon

OPSD: On-policy self-distillation. The student writes a response itself, and a teacher, which is the same model with a reference answer added to its prompt, gives a learning signal on every token of that response.
Privileged reference: Extra information only the teacher sees, such as a worked reference solution. It is information the student does not have when it writes the response being trained on.
On-policy: The responses used for training are sampled from the model being trained, not copied from a fixed dataset, which avoids training on text the model would not write itself.
Next-token distribution: The probability the model assigns to every possible next token. From Logits to Probabilities shows how logits become this distribution.
KL distance: A measure of how far one probability distribution is from another. Here it is how far a teacher's next-token distribution is from the student's.
RLVR: Reinforcement Learning with Verifiable Rewards. It scores a whole response as right or wrong, so the signal is one sparse number per response rather than one per token.
Exposure ratio: The fraction of the reference solution a teacher is allowed to see, for example the first half only. The final answer is kept in every variant.

The news. On October 29, 2026, researchers at TU München, Kyoto University and Aalto University posted Every Token Needs Its Own Teacher (arXiv 2610.40574). It proposes MTA-OPSD, which the authors describe as the first token-level adaptive method for on-policy self-distillation, and reports gains on math-reasoning benchmarks with Qwen3 models. Read the paper →

Picture a student reading an essay aloud while tutors mark it. Each tutor has seen a different slice of the answer key: one saw all of it, one only the first half, one only the last half, one a random excerpt. Standard OPSD hires one tutor and lets that tutor mark every word. The tutor is the same model as the student; the only difference is that its prompt also contains the reference solution.

A tutor who has seen the whole key can be too sharp. The paper ties degraded reasoning in OPSD to references that are too informative: the teacher's distribution becomes overly peaked and leaves little room for the student's own uncertainty. Earlier fixes, such as Paint, change how much of the key the tutor sees, but they change it per response. Every token inside one response still gets the same tutor.

The authors first test whether that is actually a problem. In a teacher-intervention pilot they keep the student's prefix, let one candidate teacher write a short continuation at a chosen position, hand control back to the student, and measure how much the final answer improves. That gain is a proxy for how well the teacher suits that position. Comparing the preferred teacher at the 25%, 50% and 75% marks of the same response, Qwen3-1.7B showed only 48% to 52% overlap between any two marks (pilot Table 1). So the best tutor changes inside a single answer.

MTA-OPSD turns that observation into a per-token rule. It builds a pool of seven teachers, all the same model as the student, differing only in how the reference is cut: the full reference, plus prefix, suffix and random-span truncations at two exposure ratios each. Then, for every token the student produced, two steps run:

Majority vote. Each teacher votes on whether it gives the sampled token more probability than the student does (compare their next-token distributions). The larger side becomes the majority subset. In the pilot, the best teacher was inside the majority subset in roughly 78% to 91% of cases (Qwen3-1.7B and 4B, regular and extreme token positions), and majority teachers gained +3.34% versus +1.88% for minority ones in one Qwen3-4B setting.
Largest KL inside the majority. Among the majority teachers, pick the one whose distribution is farthest from the student's. In the pilot, the farthest teacher gained more than the nearest one at regular positions (+2.42% versus +0.59% on Qwen3-1.7B, +2.78% versus +1.21% on Qwen3-4B).

The order matters. The authors do not treat KL as a quality score by itself: the vote first decides whether the pool backs or opposes the student's token, and only then does KL choose the most informative teacher on that side. The chosen teacher's distribution replaces the single fixed teacher in the usual per-token distillation loss, so different tokens of one response are trained against different teachers.

Walk one token (illustrative numbers, not from the paper). The student sampled the token "therefore" with probability 0.40. The seven teachers give it 0.55, 0.62, 0.50, 0.35, 0.45, 0.30 and 0.70. Five are above 0.40 and two are below, so five teachers vote to support the token and two are outvoted. Among the five, say the teacher giving 0.70 has the distribution farthest from the student's (KL 0.9, against 0.2 to 0.5 for the other four). That teacher alone supervises this token, and the other six are ignored for it. At the next token the same procedure can select a different teacher, so one response is trained against a patchwork of teachers instead of one.

What it bought. Trained on DeepMath-103K for 300 steps and tested on AMC23, AIME24 and AIME25, MTA-OPSD had the best average at all three Qwen3 sizes the paper reports: +5.9 points over the base model on Qwen3-1.7B, +1.5 on 4B and +2.8 on 8B (best checkpoint, Avg@12).

Method	Which teacher supervises a token	Qwen3-1.7B average (AMC23, AIME24, AIME25)	Source
Base model	none, no distillation	51.1	Table 5
OPSD	one full-reference teacher, all tokens	54.0	Table 5
Paint	one teacher per response, exposure set from the student-teacher gap	54.0	Table 5
MTA-OPSD	one teacher per token, from a pool of seven	57.0	Table 5

Read the scope before generalizing. The evidence is one model family, math reasoning only, with reference solutions as the privileged information. The pool and the two-step rule are tuned to that setup, and we did not see cross-domain results. One implication of the method, which is our reading and not a figure the paper reports in the parts cited here: each token's supervision needs next-token distributions from several teacher prompts, not one.

Goes deeper in: LLM Internals → Text Generation → From Logits to Probabilities

Related explainers

CoPD — co-evolving policy distillation — a different multi-model distillation setup, between parallel experts.
SIS — selective importance sampling — another post-training method that works token by token.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based