CoPE is an agent-training method that adds a multi-step action forecasting loss to standard outcome-based reinforcement learning. The same model that acts is also trained to write the next few actions that followed each history in a successful rollout.

Agent rollouts are expensive and a single end-of-episode reward leaves most of the intermediate steps unused. CoPE turns the actions inside successful rollouts into extra training targets, and the paper reports up to 2.0× higher sample efficiency than GRPO on ScienceWorld.

How does CoPE differ from value or observation targets?

Value targets copy the final outcome to every step, and observation targets ask the model to predict long, noisy next observations one step ahead. CoPE predicts short action sequences that actually followed each step, so the target is specific to the step and low in noise.

Does CoPE slow down the agent at run time?

No. The forecasting instruction is used only during training. At inference the agent acts one step at a time with the same backbone, so the paper reports no additional model and no inference overhead compared with a GRPO-trained policy.

Agent· 2026-10-20

CoPE trains agents to forecast their own next actions — Multi-step action forecasting — What does it mean?

Jargon

Agentic RL: Reinforcement learning where a language model acts over many steps (tool calls, clicks, commands) and is rewarded for how the whole episode ends.
Rollout / trajectory: One complete episode: the history of observations and actions from the first step until the task ends or times out.
Terminal (sparse) reward: A single score, such as success or failure, that arrives only at the end of a rollout. Every step in the rollout shares it.
GRPO: Group Relative Policy Optimization. It samples a group of rollouts for the same task and normalizes each rollout's reward by the group's mean and standard deviation, so no separate critic model is needed. Reward-shaping view
Value target: An estimate of how likely a given step is to lead to success. Usually it is derived from the same one final outcome.
Observation target: The environment's next observation (a web page, a terminal log) used as something to predict. It is long and noisy. Related: a world model as a simulator
Auxiliary loss: An extra training objective added beside the main one. It shapes the model's internal representation but is not used when the model runs.
Temporal ensemble: Several predictions made at different times about the same thing, averaged or blended. Here the "blending" happens inside the shared model weights during training.

The news. On October 14, 2026, researchers from the University of Illinois Urbana-Champaign, UC San Diego and Meta AI posted CoPE, which they report improves on GRPO and on baselines with predictive auxiliary targets across the simulated environments (ALFWorld, ScienceWorld, WebShop), and improves overall performance on the two realistic applications (τ²-bench and AppWorld). Read the paper →

Picture a chess coach who gets a pile of won games and a stingy scoresheet: it says only "win" at the bottom, nothing about individual moves. The usual way to learn from that sheet is to nudge the student toward every move in a winning game and away from every move in a losing one. That is what GRPO does, and it wastes most of what the game contains. CoPE keeps that rule and adds a second exercise: pick any position in a won game and ask the student to name the next few moves the winner really played. The student is graded on moves, not on a score. Because the target is short and written by the agent's own successful play, it is cheap to build and it is specific to that position.

The reason this belongs in agent training is the shape of an agent's loop: it observes, picks an action, observes again (the loop itself). An action is short and decision-relevant, while the observation that follows it can be a whole web page full of irrelevant tokens. Predicting actions instead of observations drops the noise but keeps the structure of the task. It also ties directly to a problem named in the planning module: an agent that commits to one step at a time with no sense of what comes next is exactly the single-shot failure.

How the recipe works

CoPE uses one model for both jobs. The same backbone that picks the next action is also asked, when a forecasting instruction is appended to the history, to write out the chunk of actions that followed. No second model is trained, stored or served. Each training iteration has three steps:

Collect rollouts with the current policy, in groups per task, as in GRPO, and train the policy with the usual outcome-based RL loss.
Keep only the successful rollouts. For every history inside them, the target is the realized chunk of the next several actions. Failed rollouts are left out on purpose, because fitting them would pull the shared backbone toward the behaviour that GRPO is trying to suppress.
Add the forecasting loss to the RL loss. The paper averages the forecasting loss over the successful rollouts of each group, so the extra gradient does not grow just because the policy succeeds more often as training goes on.

At inference the forecasting instruction is simply not used. The agent acts one step at a time, exactly like a GRPO-trained policy, so the paper reports no inference overhead and no additional model.

Auxiliary target	Built from	How far ahead	Main weakness the paper names
Value target	The final outcome of the rollout, copied to each step	Whole rollout	Every step in a trajectory gets the same label, so it says little about any individual step
Observation target	The environment's next observation (web page, terminal log)	One step	Long and noisy, with many task-irrelevant tokens
CoPE action target	The actions that actually followed this history in a successful rollout	Several steps (a chosen horizon)	Needs successful rollouts to exist; far-ahead actions depend on observations the forecast has not seen

Why forecasting also helps acting

The paper's argument is that the two jobs co-evolve. Acting helps forecasting because the rollouts that supply the targets come from the current policy, and success filtering keeps only those that worked. Forecasting helps acting through what the authors call implicit temporal ensembling: a single action in a successful rollout appears in the forecast chunk of several earlier histories, so the shared backbone is taught that action from several vantage points instead of only at the moment it was taken. Those earlier forecasts do not contradict acting, because they are all conditionals of the same policy; each is, at the optimum, the plain GRPO target averaged over the observations not yet seen.

The horizon is a real tradeoff. A short horizon keeps the forecast close to what the history can support, and a long one adds noise, because the action several steps ahead depends on observations the forecast cannot see. In the paper's ablation, every horizon they tried beat GRPO, while the longest horizon tested declined slightly as noise from distant predictions began to dominate.

A worked example (illustrative)

Take a successful rollout of 6 actions and a forecast horizon of 3 (both illustrative numbers, not the paper's settings). At history 1 the target chunk is actions 1 to 3; at history 4 it is actions 4 to 6; near the end chunks are truncated at the terminal token, so history 5 targets actions 5 and 6, and history 6 targets action 6 only. Count the targets: 3 + 3 + 3 + 3 + 2 + 1 = 15 action targets from one rollout, against 6 for plain GRPO, which supervises each action once, at the moment it is taken. Action 4, for example, is a target from histories 2, 3 and 4, so it is learned three times from three different vantage points.

Be careful with the arithmetic: 2.5 times more action targets does not mean 2.5 times faster learning. The paper's measured gain is smaller and it depends on the task: on ScienceWorld it reports up to 1.9× higher sample efficiency than ECHO and 2.0× higher than GRPO. Those figures come from the paper's own experiments with Qwen2.5-7B-Instruct on the simulated environments; the AppWorld experiments use a 14B model, and a repeat on Qwen3-8B improved both score and success rate on WebShop and ScienceWorld.

Where it earns its keep, and where it may not

The method fits settings where rollouts are expensive and some of them succeed: shopping, household and science tasks, customer-service dialogues, and API-composing tasks all fit. The same design implies a limit that the paper's filter makes visible: if a task is so hard that almost no rollout succeeds, the success-filtered forecasting loss has little or nothing to train on, so CoPE depends on the policy first stumbling into some wins. The paper evaluates five benchmarks; the architectural specifics of how well it holds on much longer-horizon production agents are not documented in the source.

Goes deeper in: AI Agents → Planning & Reflection → The Single-Shot Failure

Related explainers

Tev-Gym — progress-potential reward shaping vs terminal-only reward — another way to get a denser signal than one end score
Qwen-AgentWorld — a world model as a decoupled RL simulator — the observation-prediction route that CoPE contrasts with

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based