ActObs unmasks the environment's half of an agent trajectory — Observation-token supervision — What does it mean?

The news. On 17 September 2026, researchers at AWS AI Labs published Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL, introducing ActObs. The change is one line of the training loop: extend the supervised fine-tuning loss mask to cover the observation tokens every agent trajectory already contains. Data, model, context and training procedure stay fixed. On Qwen3-4B the resulting policy, after GRPO, reaches higher pass@k than its action-only counterpart at every evaluated sampling budget on Terminal-Bench 2.0; on Qwen3-8B it trades some pass@1 for +3.4 pp at pass@16. Read the paper →

Give an actor only their own lines and they will deliver them beautifully. Ask them to play the scene and it falls apart, because a performance is a reaction — you have to know roughly what is coming back at you to know how to land what you say next. An agent trained on action tokens alone is that actor: fluent in commands, blank on consequences. The blacked-out half of the script was never decoration. It is the half that tells you what your own line does.

An agent trajectory is literally that two-part script. The paper writes it as (x, a1, o1, a2, o2, ..., aT, oT): a task prompt x, then an alternation of assistant turns at carrying reasoning and commands, and environment responses ot — in this setting, terminal output inserted verbatim into the next context. That alternation is the tick every agent loop runs, and the transcript it leaves behind is the state object the agent carries forward. Standard agent fine-tuning applies its loss to the at tokens and masks the ot tokens, using them as context but never as a prediction target. Prompts stay masked under both objectives.

ActObs changes exactly one thing. It scores both token sets in a single loss, weighting observations by a factor the paper calls lambda (λ) and dividing by |A| + λ|O| so the per-example loss scale does not drift as λ moves. λ = 0 recovers the ordinary action-only objective, and ActObs's default is λ = 1, giving action and observation tokens equal weight. Nothing else moves: no new demonstrations, no extra rollouts, no added parameters, no additional forward passes. The observations were already being read on every step. Which means the trajectories you already keep for trace replay are two training examples rather than one — a policy example and a transition example, over the same tokens.

So why should predicting text the deployed agent never emits change anything? The paper's answer is measured at the gradient. At every saved fine-tuning checkpoint the authors compute the action-token and observation-token gradients separately, on the same 256 held-out trajectories, for both methods — so under action-only training the observation gradient is a pure diagnostic, masked during training and never applied. Two quantities describe how the two streams relate: their cosine similarity — are the two updates pointing the same way? — and their norm ratio r = ||g_obs|| / ||g_act||, which is simply how big one is next to the other.

At the pretrained checkpoint the two streams still largely agree — cosine similarity starts at 0.83 — and then falls to the measurement noise floor within 10 to 20 fine-tuning steps. Once the cosine is near zero the observation gradient is essentially all orthogonal to the action gradient, so r reports the size of a signal pointing somewhere training never goes. Under action-only fine-tuning that ratio climbs to about 41; under ActObs it stays near 0.5. Read the 41 literally: by the end of the standard recipe there is a coherent learning signal roughly forty times the length of the one being followed, at right angles to it, and the loss mask throws all of it away.

The bill arrives where you would expect. Teacher-forced on 300 held-out validation trajectories, ActObs predicts observations better than the base model — not only on the fixed New Terminal Output: wrapper, which is the easy part, but on shell prompts, error lines and other payload text. Action-only fine-tuning ends up worse than the base model across most of the sequence. It did not merely fail to learn consequences; it unlearned them.

Those are Terminal-Bench 2.0 pass@k figures in percent, measured after GRPO from each initialization, on 89 tasks with 16 attempts per task and one bootstrap standard error. The third row is the one worth pausing on. The timing control receives the same total amount of observation supervision as ActObs, just sequenced — a full epoch of observations, then a full epoch of actions — and it finishes below both. Having seen the environment's replies at some point is not the mechanism. Fitting decisions and outcomes jointly, so the same representation serves both, is.

Here is the accounting that makes the change look close to free. Hold three things fixed, exactly as the paper does: one corpus of 50,000 multi-turn terminal trajectories totalling 0.71B tokens, one model, and one training run of 781 steps at batch size 64. Observation tokens are about 45% of that corpus, so roughly 0.32B tokens are terminal output and roughly 0.39B are agent actions. Now count what each objective pays for against what it learns from. Both push all 0.71B tokens through the forward pass, because the observations sit in the context either way — that cost is identical and unavoidable. Action-only fine-tuning then applies loss to about 0.39B of them; ActObs applies loss to all 0.71B. Same data, same forward passes, about 1.8× the supervised tokens (0.71 ÷ 0.39). The 0.32B observation tokens were already being paid for on every single step. The only open question was whether anything was learned from them.

This is not a free uniform win, and the paper is careful about that. On Qwen3-4B, GRPO from ActObs beats its action-only counterpart at every evaluated k. On Qwen3-8B it does not: pass@1 after GRPO is 11.0 against 12.3 for action-only, while pass@16 is 27.0 against 23.6 — the +3.4 pp gain at sixteen attempts is bought with a 1.3 pp loss at one. Sweeping λ with the data and the GRPO recipe held fixed reproduces the same shape: as λ rises, the post-GRPO pass@8 advantage grows monotonically while pass@1 moves the other way. λ is a dial between one-shot reliability and multi-sample coverage, not a uniform improvement.

That has a concrete reading for anyone deciding whether to adopt it. If your agent gets one attempt and no retry, an action-only initialization may still be the right call. If your harness can pause, observe and try again, the extra coverage is the more valuable currency. The token-level measurement even says where that coverage comes from: after GRPO both policies are equally sharp on the first tokens of a command, and the gap opens later in the command — on the arguments, flags and paths. ActObs does not make the agent hesitant about which tool to reach for. It keeps more variation in how the chosen command gets filled in. One ActObs-only success in the paper's appendix uses exactly that freedom, sweeping QEMU launch parameters one at a time until it finds a memory configuration that works.

Goes deeper in: AI Agents → Agent Loop & State → Inside a Tick

Related explainers

• Role-Agent — Dual-role self-play — the other way to use state prediction: have one model act as both agent and environment, and turn the agreement between predicted and observed state into a per-step reward.
• Qwen-AgentWorld — World model as a decoupled RL simulator — what it looks like to go further and train a language model to stand in for the environment altogether.
• Observation-effect separation — why the observation a tool returns and the effect it actually had are not the same thing.