What is dual-role self-play in Role-Agent?

It is a training setup where a single LLM plays both the agent and the environment in its own training loop. As the agent it takes actions; as the environment it produces the resulting states (World-In-Agent) and reshapes its own training data from its failures (Agent-In-World). The paper (arXiv 2606.10917) reports an average gain of more than 4% over strong baselines across several benchmarks.

Why have one LLM be both agent and environment?

Because the two things agent RL usually needs — a realistic environment to act in and a reward model to score the actions — are the expensive, slow-to-build parts. If the model can serve as its own environment and generate its own reward signal, you can bootstrap self-improvement without standing up that external scaffolding.

How does Role-Agent get a reward without a reward model?

Through agreement. In the World-In-Agent half the model predicts the next state, then compares its prediction to the observed state; the closeness of the match becomes a dense, per-step process reward. That replaces a separate reward model and gives a signal at every step instead of only at the end of the trajectory.

Role-Agent paper — One LLM as agent and environment

TL;DR

What is it: A new paper, Role-Agent (arXiv 2606.10917), trains an LLM agent with a single model playing both sides of the loop — it is the agent that acts and the environment that reacts, a setup the authors call dual-role self-play.
Why it’s needed: Building the two things agent training usually needs — a realistic environment to act in and a reward model to score the actions — is the expensive bottleneck, so a method that conjures both from the agent itself could make self-improvement far cheaper to bootstrap.
vs previous: Most agent-RL setups bolt on a hand-built environment plus an external reward model that typically scores only the final outcome; Role-Agent folds both into one model and grades each step with a dense process reward drawn from how well its own prediction of the next state matches what happens.

Jargon

Role-Agent: The paper's method: one LLM is trained by alternately playing the agent (taking actions) and the environment (producing what happens next), so it learns without an external world to act in.
World-In-Agent: The reward half of the loop. The model acts as a world model — it predicts the next state, and the agreement between its prediction and the observed state becomes a per-step reward.
Agent-In-World: The data half of the loop. The model diagnoses its own failure patterns, retrieves similar tasks, and rebalances what it trains on next.
Process reward: A reward given at every step of a trajectory rather than once at the end. It is denser than an outcome reward, so the learning signal can point at which step went wrong.
World model: A model that predicts how a situation evolves — given a state and an action, what the next state will be. Here the same LLM serves as its own world model.
Reward model: A separate model normally trained to score an agent's behavior. Role-Agent's point is that it needs no such external scorer — the agreement signal stands in for it.
Self-play: Training where a model improves by competing against or interacting with copies of itself, instead of against a fixed external opponent or dataset.

The news. On June 9, 2026, researchers posted Role-Agent (arXiv 2606.10917), a method that bootstraps an LLM agent by having one model play both sides of the loop. In its World-In-Agent half the model predicts the next state and is rewarded when its prediction matches what actually happens; in its Agent-In-World half it analyzes its own failures and retrieves similar tasks to reshape its training data. The paper reports an average gain of more than 4% over strong baselines across several benchmarks. Read the paper →

Picture a boxer shadowboxing. There is no opponent in the ring, yet the boxer throws a real combination and, in the same motion, imagines how an opponent would slip and counter. They are the fighter and the imagined opponent at once. That double role is exactly Role-Agent's trick: a single LLM is both the agent choosing actions and the environment that decides what those actions lead to. Agent training usually needs a real environment someone built plus a separate reward model to grade the actions — the two pieces that are slow and costly to stand up. Role-Agent's claim is that you can get both agent and environment out of the one model you were going to train anyway.

The reward comes from the boxer calling the counter before it lands. In the paper's World-In-Agent half, the model predicts the next state, then sees what the state actually becomes; the agreement between the two is turned into a reward. Crucially this is a dense process reward — a point at every exchange — not a single thumbs-up at the final bell. That matters because agent errors compound step by step: a sparse outcome reward only tells you the whole round went badly, while a per-step signal can localize which punch you mistimed. (The abstract frames this agreement signal as the reward; the exact way one model time-shares the predicting and acting roles is not fully detailed.)

The other half, Agent-In-World, is the post-session review. Instead of just replaying every round, the model diagnoses its own failure patterns, retrieves tasks similar to the ones it flubbed, and feeds itself more of those next time — reshaping its own training-data distribution toward its weak spots. The obvious risk is the one that haunts every self-teaching system: a boxer who is also their own coach can drift into bad habits and never notice, the same progressive collapse other self-evolving agents suffer. Role-Agent's bet is that the prediction-agreement reward keeps the world-model honest enough that the loop improves rather than rots.

Why prefer a process reward to an outcome reward at all? Walk a short example (illustrative numbers — the paper reports the headline gain, not these). Take a 20-step task. An outcome reward emits 1 signal — pass or fail at the end — so if the agent fails, all 20 steps share the blame equally and the gradient is mush. A process reward emits 20 signals, one per step: that is 20× denser feedback. Say the model's predicted state matches the observed state on 16 of the 20 steps; the 4 mismatches now flag exactly where the trajectory went off the rails, and Agent-In-World can route more practice to those. One model, three jobs — actor, world, and scorer — and no external scaffolding to build.

Ingredient	Standard agent RL	External-env approach (e.g. EnvFactory)	Role-Agent
Environment	hand-built, external	built outside the model	the model itself (World-In-Agent)
Reward signal	typically outcome-level	scored outside the model	prediction-agreement, process-level
Training data	fixed distribution	often fixed after it's built	self-reshaped toward failures (Agent-In-World)
External scaffolding	environment + reward model	an environment to build	none

Goes deeper in: AI Agents → Planning & Reflection → Retry

Related explainers

EnvFactory — synthetic envs for tool-use agents — the approach Role-Agent replaces: build the environment outside the model
Self-evolving agents collapse over iterations — the failure mode dual-role self-play has to avoid
MLEvolve — Monte Carlo Graph Search — the search side of agents improving themselves, vs this paper's training side

Continue in trackAI Agents — Planning & Reflection: how an agent diagnoses a failure and decides what to try next

Related explainers

Frequently Asked Questions