What is Critical-State RL?

A method from Salesforce AI Research for training tool-calling agents. It first measures which model call in a multi-turn conversation would improve if trained, using **nested continuation sampling**, and then applies reinforcement learning to that one call while the surrounding calls receive no gradient.

Why is a mix of wins and losses not enough to decide where to train?

Because the final reward depends on later turns too. The same action can be followed by a win or a loss purely through randomness downstream, so a mixed group can carry **zero action-dependent signal**. The paper's pilot found a group where every response was the same refusal, yet rewards still swung between 0 and 1.

How does it relate to turn-level credit assignment methods such as TRIAGE?

Credit-assignment methods decide how much reward each action in a trajectory should receive. Critical-State RL asks an earlier question — which call is worth training at all — by splitting the reward variance at each candidate call into an action part and a noise part, then trains only the call with the largest action part.

Agent· 2026-09-24

Critical-State RL picks which model call to train — Nested continuation sampling — What does it mean?

TL;DR

What is it: The Critical-State RL paper from Salesforce AI Research (arXiv 2609.24985) trains a tool-calling agent on one selected model call per trajectory, and it picks that call with nested continuation sampling: try several actions from the same point, then replay each action's aftermath several times.
Why it’s needed: In a multi-turn tool conversation the final reward depends on many calls, so a win or a loss does not tell you which call to fix — and training the wrong call wastes compute or makes the agent worse.
vs previous: A DAPO-style dynamic-sampling filter used with GRPO keeps any prompt whose rollouts mixed wins and losses, but that mix can come entirely from randomness in later turns; nested sampling splits the reward spread into the part the current action controls and the part it does not, and trains only where the first part is large.

Jargon

Tool call: A model response that invokes a function, for example book_flight(...), instead of answering in text. A multi-turn task chains many of them. See the tool schema.
Prefix: Everything in the conversation before the call being judged: the user messages, earlier tool calls and their results. Holding it fixed is what makes two actions comparable.
Continuation: Everything that happens after the judged call: later model turns, tool results and user replies. It is random, so the same action can be followed by a win or a loss.
GRPO: Group Relative Policy Optimization. It samples a group of answers to one prompt and scores each answer against the group average, so it needs no separate value model. A group where every answer scores the same teaches nothing.
Dynamic sampling (DAPO): A filter that drops prompt groups whose outputs are all correct or all wrong, keeping only groups with mixed results. The paper shows a mixed group can still carry no action signal.
Law of total variance: A statistics identity: total spread = spread of the per-action averages + average spread within each action. The first term is the signal that comes from the current action; the second is noise from what happens later.
Contextual bandit: A one-step learning setup: see a context, pick one action, get a reward, update. Critical-State RL trains the selected call this way instead of back-propagating one reward through the whole conversation.
BFCL: Berkeley Function Calling Leaderboard, a benchmark for tool calling. Its v4 multi_turn suite includes missing-function and missing-parameter tasks, where the right move at one turn is to wait.

The news. On September 21, 2026, Salesforce AI Research posted Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use to arXiv. The paper asks one question before any training: which model call in a conversation would actually improve if trained? It answers with a training-free diagnostic, then applies reinforcement learning to the selected call only. On the BFCL v4 missing-function task, training the call the diagnostic picked took a Gemma-4-26B-A4B model from 0.14 to 0.283 accuracy, while training the other candidate call moved it down to 0.095. Read the paper →

Picture a chess coach reviewing a lost game. The student lost, but the loss does not say which move lost it — a blunder on move 40 can hide a perfectly good move 12. So the coach sets up one position on the board, tries several candidate moves from it, and for each move plays the rest of the game out several times against varied replies. If move A wins every play-out and move C loses every one, the position is worth drilling: the choice made here decides the game. If every move wins about half the time, the play-outs give no evidence that a different move here would do better, however dramatic the game looked, and drilling it is likely wasted time.

A multi-turn tool-use trace has the same structure. The agent makes a chain of calls, and a reward arrives at the end. Standard trajectory-level RL spreads that one reward over every call, which is the compounding-errors problem turned inside out: one bad call sinks a run, but the training signal cannot see which one. Critical-State RL treats "which call to train" as a measurement to take first, not an assumption to make.

The measurement is nested sampling: it separates the differences between action averages from the variation inside each action's continuations. It runs on the model's own rollouts. Fix a prefix — the conversation up to one candidate call. Sample several actions at that call. Then hold each action fixed and resample its continuation, the turns after it, several times. The law of total variance splits the reward spread at that point into two parts: the spread of the per-action average rewards, which the paper calls action-dependent variance, and the average spread within one action, which is continuation noise. Only the first part is signal that comes from what the model does at this call.

The paper's own pilot shows why this matters. In a Gemma missing-function run, every sampled response at the decision turn was the same clean refusal with no tool call, yet the rewards still swung between 0 and 1 with a within-group standard deviation of 0.477. A mixed-outcome filter would keep that group as a good training example. The rewards differed, but the actions did not — so there was nothing at that call for training to learn.

Before ranking, each candidate call must also pass two gates: its local reward must track task success (the paper calls this action-sufficiency), and some action's expected local reward must exceed a reference policy's mean there, establishing headroom to improve. The call with the largest action-dependent variance among the survivors is selected. Training then updates only that call's generated tokens as a contextual bandit; the calls around it supply context or reward but receive no gradient.

Put numbers on it (illustrative, one position, binary win/loss reward). Take two candidate calls, each with 2 actions × 4 continuations = 8 rollouts, and each showing 4 wins and 4 losses — the same mixed group, the same total variance of 0.25, so a mixed-outcome filter cannot tell them apart. At call 1, action A wins 3 of 4 play-outs and action B wins 1 of 4: the action means are 0.75 and 0.25, their variance around the overall 0.5 is 0.0625, and the remaining 0.1875 is continuation noise. At call 2, both actions win 2 of 4: the action means are 0.5 and 0.5, the action-dependent variance is 0, and all 0.25 is noise. Same wins, same losses, same total — but only call 1 has anything to train. The paper also subtracts a finite-sample correction, because a few noisy continuations can make action means look different by chance; this example skips it.

BFCL v4 cell	Diagnostic role	Corrected action variance	Accuracy, start → trained	Source
missing-function / recovery turn	selected	0.0267	0.14 → 0.283 (+14.3 pp, 4 seeds)	Table 1
missing-function / decision turn	alternative	0.0103	0.14 → 0.095 (−4.5 pp, single run)	Table 1
missing-parameter / decision turn	selected	0.0361	0.435 → 0.473 (+3.8 pp, 4 seeds)	Table 1
missing-parameter / recovery turn	alternative	0.0000	0.435 → 0.445 (+1 pp, single run)	Table 1

Read the table by task, not by row. In missing-function tasks the needed tool is withheld for one turn: the agent must pause at the decision turn, then make the call at the recovery turn once the tool appears. For Gemma, the diagnostic picked recovery, and training it gained 14.3 points while training the decision turn cost 4.5. In missing-parameter tasks the diagnostic picked the opposite turn. The diagnostic selected the improving turn in both categories; either fixed turn rule missed one. Always training the decision turn, or always training the recovery turn, helps one category and leaves the other flat or worse. The choice also depends on the model: for Nemotron on the same missing-function task, the paper trains the decision turn instead.

Two controls in the paper check that the gain comes from where the signal is measured, not merely from which tokens get a gradient. Keeping one trajectory-level reward and only changing which turns receive gradient left missing-parameter accuracy near its start. Full multi-turn training with Monte-Carlo return-to-go fit its training environment far better (0.997 validation versus 0.77) but scored 0.45 on BFCL in a single-seed comparison, below the 0.473 from training the one selected call. This is error analysis first, applied to the training loop: locate the failure before spending gradient on it.

Goes deeper in: AI Agents → Evals & Diagnostics → Error Analysis First

Related explainers

TRIAGE — Role-typed credit assignment — the other answer to one reward for many actions: grade every action by its role, rather than measure which single call is trainable.
CacheRL — Cached rollouts for agent RL — the cost side of the same loop: nested sampling multiplies rollouts, and this is one way to make each rollout cheaper.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based