What is targeted guidance in agentic RL?

It is a training recipe that adds a self-distillation loss to ordinary RL. When a turn fails a judge's criterion, a feedback string is placed before that turn, the same model run with that string acts as an expert, and the live policy is pulled toward it with a per-token KL.

Why distill only on failing turns?

Masking the loss on passing turns keeps the pressure small and, once every criterion passes, leaves only the plain policy-gradient loss. The paper's ablation with always-on feedback shows a drop in coding pass-rate.

Why keep the RL reward if distillation already teaches the behavior?

Distillation gives a direction but not stable convergence on its own. In the paper, distillation without the per-criterion RL reward learned the behavior but was unstable, so the reward is what retains and refines it.

What does it cost compared with plain RL?

The expert is used only for logits, never for generation, so the main added cost is one extra forward pass per update. Rollout sampling, usually the dominant cost, does not change.

Agent· 2026-11-05

Targeted guidance in agent RL — Failure-gated self-distillation — What does it mean?

Jargon

Policy: The model being trained, viewed as a function from the conversation so far to a distribution over the next tokens.
Rollout: One full attempt by the agent: a sequence of turns (messages and tool calls) from the task start to the end.
GRPO: A policy-gradient method that samples a group of rollouts for the same prompt and rewards each one relative to the group average. If every rollout scores the same, nobody gets a learning signal.
Criterion: One binary check from a judge model's rubric, for example "the final message includes a markdown table". Each criterion becomes a reward and also carries a feedback string.
KL divergence: A measure of how far one next-token distribution is from another. Here it measures how far the live policy is from its feedback-conditioned expert; zero means they already agree.
On-policy self-distillation (OPSD): Training a model to match a copy of itself that was given extra context, using the model's own rollouts. See MTA-OPSD for a variant that picks a teacher per token.
Stop-gradient (sg): A marker meaning "treat this value as a constant". The expert's output is a fixed target within one update; only the live policy moves.
Entropy collapse: The policy becomes overly certain and stops exploring varied behaviors, which stalls later learning.

The news. On November 3, 2026, researchers at Cursor and UC Berkeley posted "Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning". They post-train a Qwen3-Coder-30B coding agent and show it can learn style and tool-format criteria it never produced before, without losing coding pass-rate. Read the paper →

Picture a student driver on a test route. Plain RL is an examiner with a clipboard who only reports the final score. If the student never once takes the correct exit, the score never changes, so there is nothing to learn from. That is the coverage problem in agent training: a reward cannot reinforce a behavior the policy never performs, and a judge model's written explanation gets squeezed into a single number before the policy sees it.

Targeted guidance puts an instructor in the passenger seat who says nothing on correct turns. When a turn fails a criterion, the instructor whispers feedback just before that turn, and the paper builds a stand-in for "what the student would do after hearing it" without a second model: the same weights, run again with the feedback string added to the context. That second run is the expert. The loss is a per-token KL between the live policy and that expert, so the gradient lands hardest on the few tokens where the two disagree, such as the delimiter in a new tool-call format.

Three choices keep the whisper from wrecking the drive. The feedback is inserted right before the offending turn, not into a global prompt, and the KL is masked on every turn that passes. Because of the masking, once all criteria pass the distillation term is zero and what remains is the ordinary policy-gradient loss, which the paper uses to argue the method still converges to the optimal policy when one exists (an argument, not a proof of the experiments). The distillation term is added to, not substituted for, a per-criterion RL reward, which is the exam score that keeps the new behavior stable. The KL is also averaged over the tokens in a rollout, so a long, looping rollout with huge KL cannot dominate the update.

The combined objective is GRPO loss + λ · Σ KL(π(·|s) ‖ sg(π(·|s, f))), summed over only the failing turns. The only extra cost is one forward pass to read the expert's logits, because the expert is never asked to generate text. Rollout sampling, the expensive part of RL, is unchanged. The paper also describes a procedure for vetting each feedback string: query the base model with it and keep it only if average performance beats the contextless model, because an "expert" that is not better than the policy is no improvement direction (a design heuristic the authors note is not guaranteed). The wider picture of how to analyze what an agent got wrong is the same discipline applied to training, where the judge's explanation becomes the signal.

Method	Markdown criterion met	Coding pass-rate	Source
Baseline (no style reward)	0.00	0.33	Table 1
GRPO with per-criterion rewards	0.00	0.33	Table 1
OPSD (feedback distillation only)	0.77	0.30	Table 1
Online SFT	0.01	0.26	Table 1
LaTRO	0.00	0.14	Table 1
Verifree	0.01	0.20	Table 1
Targeted guidance	0.85	0.33	Table 1

Values are rates averaged over the last 32 training steps of one Qwen3-Coder-30B setup, as reported in Table 1 of the paper; the paper also reports ± standard deviations and seven further criteria. The markdown column is one of eight style criteria, chosen here because the base policy scores 0.00 on it.

Here is how the pieces compose, using the paper's batch of 32 groups of 32 rollouts, so 1,024 rollouts per step. Hold the criterion fixed as "include a markdown table", which the base policy meets in 0.00 of rollouts. Under plain GRPO every rollout in a group gets reward 0 for that criterion, so the group-relative advantage is zero for all 1,024: a full step of sampling with no signal about the table. With targeted guidance, assume (illustrative) that nearly every one of those rollouts fails the check on its final turn, so each failing turn gets feedback and a KL term. Suppose (illustrative) a rollout has 20 turns and only the last fails: the distillation loss touches 1 turn in 20, and the other 19 are masked. As the table appears in more rollouts, the failing set shrinks and the term fades. The measured outcome is 0.00 → 0.85 on the markdown criterion while coding pass-rate stays at 0.33, the same as the baseline, whereas OPSD reaches 0.77 but falls to 0.30.

Limits worth keeping in mind: every result comes from one 30B coding agent in one software-engineering harness with verifiable style and format criteria, and the feedback strings are hand-written per criterion. The paper reports that always-on feedback, and distillation without the RL reward, both degrade stability or pass-rate in its ablations. Whether this carries over to other domains or larger models is not shown here.

It connects to agent design through the same loop the agent track teaches at inference time: a retry with the failure explained in context works because the model can use a hint. Targeted guidance moves that hint from the prompt into the weights, so the agent no longer needs the hint at run time.

Goes deeper in: AI Agents → Evals & Diagnostics → Error Analysis First

Related explainers

MTA-OPSD — another on-policy self-distillation variant, where each token picks its own teacher.
CORD — what to do when the distillation gradient and the RL gradient pull in opposite directions.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based