CORD is a gradient-combination rule for training with two losses at once. It protects one loss's gradient, checks whether the other gradient opposes it, and if so removes only the opposing component before adding the two.

Why not just lower the distillation weight?

A positive weight rescales a gradient but never changes the sign of its inner product with the other gradient, so a conflicting pair stays conflicting. CORD changes the direction of the candidate gradient only when there is a conflict.

How is CORD different from PCGrad?

PCGrad modifies both gradients when they conflict. CORD keeps the protected gradient exactly as it is and modifies only the other one, which preserves the protected loss's first-order progress.

How strong is the evidence?

On Qwen2.5-3B-Instruct with one training seed, CORD raised ALFWorld average success for all three tested recipes (by 1.5 to 15.9 points) and WebShop success for two of three. One WebShop pair dropped by 0.3 points, so the gain is not universal.

Agent· 2026-10-23

CORD paper — Conflict-aware gradient projection — What does it mean?

TL;DR

What is it: The CORD paper (arXiv, October 21, 2026) introduces conflict-aware gradient projection: when an agent's reinforcement-learning gradient and its self-distillation gradient point against each other, CORD deletes only the opposing part of one of them before adding the two.
Why it’s needed: Joint RL plus distillation gives an agent both sparse task reward and dense teacher guidance, but in the paper's WebShop log 48.2% of mini-batch gradient pairs had a negative cosine, so conflict was common, and the paper separately reports that conflicting distillation gradients reduce first-order RL progress.
vs previous: A mixing coefficient (a fixed weight on the distillation loss) only rescales the distillation gradient and cannot selectively remove the part that opposes the RL gradient, whereas CORD removes the conflicting component and leaves aligned gradients untouched.

Jargon

Gradient: The direction in parameter space that increases a loss the fastest. Training subtracts it, so the update moves the opposite way.
Cosine / inner product: A number that says whether two gradients point the same way (positive), at right angles (zero), or against each other (negative). A negative value is a gradient conflict.
GRPO: Group Relative Policy Optimization. It samples several attempts at the same task and rewards each attempt by how much its return beats the group average. It gives no signal when all attempts tie.
OPD: On-policy distillation. A teacher scores the tokens the student itself just sampled, which gives dense token-level feedback. See CoPD and DOPD for other takes.
Protected objective: The one loss whose gradient CORD keeps exactly as it is. The other loss is the candidate, and only the candidate is modified.
Mixing coefficient (lambda): The fixed weight placed on the distillation loss before the two losses are added. It rescales a gradient but cannot remove the part that opposes the other one.
ALFWorld / WebShop: Two agent benchmarks. ALFWorld is a text-based household task suite; WebShop asks an agent to find and buy a product on a simulated shopping site.

The news. On October 21, 2026, researchers from CUHK, Tencent and other groups posted CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents. Added to three existing joint-training recipes, it raised ALFWorld success by 1.5 to 15.9 percentage points, and on WebShop it improved two of the three recipes. Read the paper →

Picture two people steering one shopping cart. The head pusher shoves toward the checkout, and that is the reward signal from the task. A helper pushes too, and that is the teacher's guidance. Most of the time they push in similar directions and the cart speeds up. Sometimes the helper shoves partly backward, and the two pushes cancel each other. Telling the helper to push softer does not fix it, because a softer backward shove is still backward.

That is the situation CORD addresses. Training an agent with GRPO (rewards from the task outcome) and OPD (token-level guidance from a teacher) produces two gradients per mini-batch, which the paper calls the RL gradient and the distillation gradient. Their sum is the usual update. The authors measured how often those two disagree on WebShop: 48.2% of mini-batch gradient records had a negative cosine, and a conflicting distillation gradient measurably cut the first-order progress made on the RL loss. The usual knob, the mixing coefficient, multiplies a gradient by a positive number, so it changes the size of the clash but cannot turn a backward shove into a forward one.

CORD's rule has two parts. First, pick a protected gradient a and a candidate gradient b. Second, test their inner product. If it is zero or positive, add them as usual. If it is negative, replace b with b - (a·b / a·a) a, the closest vector to b that no longer pushes against a. In cart terms, on a conflict the helper keeps only the sideways part of the push and drops the backward part. The paper proves this replacement is the nearest feasible point, and that the protected loss then makes first-order progress equal to the squared length of a, exactly what it would make alone. Only b is changed: the older PCGrad method modifies both gradients on a conflict, while CORD keeps the protected one intact. CORD also combines the gradients once per optimizer mini-batch, after accumulating over micro-batches, because projecting each micro-batch separately gives a different update.

A worked example with small numbers

Hold two things fixed: an RL gradient a = (3, 0) that is protected, and a distillation gradient b = (-2, 4) after weighting (illustrative numbers, not from the paper). Their inner product is 3 × -2 + 0 × 4 = -6, which is negative, so they conflict. The plain sum is (1, 4), and its inner product with a is 3 × 1 = 3, so the RL loss gets only 3 units of first-order progress out of the 9 it would make alone. CORD subtracts (-6 / 9) times a from b, which gives (0, 4): the backward component vanishes and the sideways component survives. The final update is (3, 4), and its inner product with a is 3 × 3 = 9, so RL progress rises from 3 to the full 9. The trade-off is on the other side: the distillation gradient's alignment with the update falls from 14 to 10, out of 20 for its own gradient alone. CORD does not remove the trade-off, it decides who wins it.

Joint-training recipe	ALFWorld average, before → with CORD	WebShop success, before → with CORD	Source
Skill-SD	76.4 → 77.9	55.8 → 59.4	Table 1, Qwen2.5-3B-Instruct, one training seed
GRPO + OPSD	65.0 → 80.9	64.8 → 64.5 (slightly lower)	Table 1, Qwen2.5-3B-Instruct, one training seed
SDAR	73.8 → 77.7	60.7 → 64.6	Table 1, Qwen2.5-3B-Instruct, one training seed

The table shows what the evidence does and does not say. CORD helped every recipe on ALFWorld and two of three on WebShop. But the headline table uses one training seed with four evaluation passes, one 3B model, and two benchmarks, so treat them as a promising signal rather than a general law. The paper also reports that the protected direction and the conflict trigger must be chosen together, and that the distillation loss and its weight still matter after projection. The first-order guarantee also holds for the ideal gradient only: gradient clipping and an optimizer such as AdamW change the final update direction.

This is the same habit that good agent evaluation teaches: measure the failure before you fix it. The authors first logged how often the gradients disagree and what it cost, and only then built the projection. The Error Analysis First step makes that argument for agent behavior, and Compounding Errors shows why small per-step losses, like a cancelled update, add up over a long training run.

Goes deeper in: AI Agents → Evals & Diagnostics → Error Analysis First

Related explainers

CoPD paper — Co-evolving Policy Distillation — another take on distillation between policies.
DOPD — Dual on-policy distillation — a different fix for unreliable teacher guidance.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based