What is Direct-OPD (Direct On-policy Distillation)?

Direct-OPD (arXiv 2607.05394) is a weak-to-strong post-training method that improves a stronger reasoning model without running reinforcement learning on it. It runs cheap RL on a small, weak teacher, computes the log-ratio between that teacher after RL and before RL, and uses that ratio as a dense per-token reward while the stronger student generates its own answers. It reports a 1.7B model rising from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100 GPUs.

Running RLVR directly on a large model is slow and GPU-expensive because the reward is sparse — one right/wrong signal at the end of a long answer. Direct-OPD lets a small model pay that RL cost, then reuses the shift it learned as a dense reward for a bigger model. That turns an expensive sparse-reward loop into a cheap, transferable coaching signal, and it needs no separate reward model.

How is it different from ordinary distillation or from DOPD?

Ordinary distillation copies the weak teacher's finished outputs, which caps the student at the teacher's ceiling. Direct-OPD instead copies the change RL induced — the log-ratio of the teacher after vs before RL — applied to the student's own on-policy rollouts, so it transfers a direction of improvement rather than answers. DOPD is a different member of the same on-policy distillation family: it routes each token between a privileged teacher and student by advantage, whereas Direct-OPD's signal is the pre/post-RL log-ratio.

Direct-OPD transfers weak-model RL gains as log-ratio rewards — Weak-to-strong reward transfer

TL;DR

What is it: The paper Direct On-policy Distillation (Direct-OPD) (arXiv 2607.05394) is a weak-to-strong post-training trick: it improves a stronger reasoning model by reusing how reinforcement learning already changed a smaller, weaker one — without running that RL on the big model at all.
Why it’s needed: Running RLVR directly on a large model is slow and expensive because the reward is sparse — one right/wrong at the very end of a long answer; reusing a small model’s already-paid-for RL shift is a cheap way to lift a bigger one.
vs previous: Instead of rerunning sparse-reward RLVR on the target (or copying the weak teacher’s finished answers, which caps the student at the teacher’s ceiling), Direct-OPD distills the change RL induced — the log-ratio of the teacher after vs before RL — as a dense per-token reward.

Jargon

RLVR: Reinforcement Learning with Verifiable Rewards — RL where the reward is an automatic pass/fail check (did the math answer match, did the code run) rather than a human rating. It works, but the signal is sparse: one bit at the end of a whole answer.
On-policy distillation (OPD): The student learns from a teacher’s signal on the student’s own attempts, not on the teacher’s finished answers, so the lessons stay relevant to what the student actually does. More on OPD →
Policy shift: The difference between a model’s token choices before and after training — which moves RL made more likely and which it made less likely. Direct-OPD treats this shift, not the final model, as the thing worth copying.
Log-ratio (implicit) reward: The natural log of the post-RL probability divided by the pre-RL probability for a token. It is positive where RL raised that choice and negative where RL lowered it — a dense, per-token reward you get for free, with no separate reward model.
Weak-to-strong: Using a weaker model’s signal to improve a stronger one. The surprise is that a small model’s learned direction of improvement can still point a bigger model the right way.
Sparse vs dense reward: A sparse reward arrives once, at the end (right/wrong); a dense reward scores every token. Dense signals train far faster because the model learns which specific choices helped.

The news. On July 6, 2026, the Direct On-policy Distillation paper (arXiv 2607.05394) proposed a weak-to-strong post-training route: instead of rerunning expensive RLVR on a large target model, it diffs a small teacher before and after RL and transfers that policy shift onto a stronger student’s own on-policy states. It reports Qwen3-1.7B rising from 48.3% to 62.4% on AIME 2024 after just 4 hours on 8 A100 GPUs. Read the paper →

Picture a rookie who just came back from a cheap weekend chess camp. The camp genuinely helped — the rookie now plays some openings differently, avoids a few traps, presses harder in the endgame. You also coach a champion who is already far stronger than the rookie. The expensive move would be to send the champion to their own elite camp. Direct-OPD does something cheaper: it never sends the champion anywhere — it copies what the rookie’s camp changed.

Why not just run the training on the champion directly? Because reasoning models are usually improved with RLVR, and its reward is sparse — the model writes a long solution and only learns right or wrong at the very end. On a large model that trial-and-error is slow and burns a lot of GPU time. The rookie, being small, can run that same RL cheaply. So the trick is to let the rookie pay the RL bill, then harvest the lesson rather than the bill.

Here is Direct-OPD’s move, and it is the whole idea. Take the weak teacher after its RL and before it, and for each token compare the two probabilities it would assign. Direct-OPD’s reward is the log-ratio of the teacher after RL versus before — positive where RL made a choice more likely, negative where it made a choice less likely. That per-token scorecard is a dense reward, and it needs no separate reward model, no human labels, and no sparse-reward RL on the big model.

Now hand that scorecard to the champion — but apply it on the champion’s own board. The champion generates its own answers, token by token, and Direct-OPD nudges each of those tokens up or down by the rookie’s log-ratio. Because the student learns from its own attempts, the borrowed signal stays on-policy and never just asks the champion to imitate the small teacher’s ceiling — it transfers the direction of improvement, not the weaker model’s answers.

Walk two tokens through it (illustrative numbers). On one move the rookie’s probability went from 0.30 before camp to 0.60 after, so the reward is ln(0.60 / 0.30) = +0.69 — push the champion toward that move. On another move the rookie dropped from 0.50 to 0.20, so the reward is ln(0.20 / 0.50) = −0.92 — push the champion away. Same camp, opposite signs: the size and direction of every nudge is simply the log of how much camp moved that choice. On the real benchmark, that transfer took a 1.7B model from 48.3% to 62.4% on AIME 2024 — a +14.1-point jump — in 4 hours on 8 A100 GPUs, with no RL run on the target model at all.

Approach	Runs on the big model?	Reward signal	What it transfers
RLVR on the target	yes — sparse-reward RL	verifiable pass/fail, once at the end (sparse)	real gains, but slow and GPU-expensive
Distill the teacher’s outputs	no	imitate the weak teacher’s final tokens	cheap, but caps the student at the teacher’s ceiling
Direct-OPD (arXiv 2607.05394)	no	log-ratio of the teacher after vs before RL, per token (dense)	the RL policy shift — the direction of improvement

The takeaway is a small shift in what you choose to copy: you can distill the change RL induced, not just the model RL produced. A weak model’s cheap reinforcement learning becomes a dense, reusable coaching signal for a stronger one — and because it lands on the student’s own rollouts, the gains transfer instead of merely being imitated.

Goes deeper in: LLM Internals → Text Generation → From Logits to Probabilities

Related explainers

DOPD — Dual on-policy distillation — another on-policy distillation fix, routing each token between a privileged teacher and student
OPID — On-policy skill distillation — the same family, mining reusable skills from an agent’s own runs
CoPD — Co-evolving policy distillation — keeps every teacher on-policy instead of frozen
Distillation in pre-training — non-monotonic teacher strength — when a stronger teacher can actually hurt the student

Continue in trackLLM Internals — Text Generation: from logits to probabilities

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based