What is gradient streaming in ThunderSyncRL?

It is a training schedule that runs each trajectory's backward pass as soon as that trajectory's inputs are fixed, instead of waiting for the full batch. For GRPO that means as soon as its own reward arrives; for on-policy distillation it means as soon as the teacher has scored a turn.

How can GRPO compute a gradient before the group's rewards are all in?

GRPO weights each trajectory's gradient by `(r − μ)/σ`, and that sum rearranges into `(1/σ) · (Σ r·g − μ · Σ g)`. Keeping the two running sums lets the learner apply the group mean and standard deviation at the end with a cheap scalar step, giving exactly the batch-synchronous update.

How is this different from asynchronous RL?

Asynchronous RL overlaps rollout and learning across updates, so some trajectories come from an older policy, which can call for off-policy corrections or staleness limits. Gradient streaming overlaps work inside one update and still takes a single synchronous optimizer step, so policy staleness is zero.

When does gradient streaming help the most?

When rollout time and learner time are similar, because it can only hide learner work behind rollout work. In the paper, a 27B model near that balance got about **1.9×** shorter steps, while a model with only about 3B active parameters gained less.

LLM· 2026-10-11

ThunderSyncRL — Gradient streaming without policy staleness — What does it mean?

TL;DR

What is it: Gradient streaming is the scheduling idea in ThunderSyncRL, an October 2026 paper on training AI agents with reinforcement learning: it starts the gradient computation for each agent attempt (a trajectory) the moment that attempt's own reward arrives, instead of waiting for the whole batch.
Why it’s needed: Agent rollouts are long and uneven, so in synchronous RL training the paper measured learner GPUs sitting idle for 49.5% of each step; streaming fills that gap and reached the same SWE-bench Verified score up to 1.9× faster.
vs previous: Asynchronous RL also reduces the idle time, but it trains on trajectories sampled by an older version of the policy (policy staleness); gradient streaming moves only the gradient math earlier and keeps one synchronous optimizer step with zero staleness.

Jargon

GRPO: Group Relative Policy Optimization. An RL method that samples a group of answers per prompt and scores each one against the group, instead of training a separate value model. Background: RLVR explained.
Advantage: How much better one trajectory did than its group: its reward minus the group mean, divided by the group's standard deviation. It sets how hard that trajectory pushes the update.
Backward pass: The expensive half of training compute: running the model in reverse over a trajectory's tokens to get its gradient, the direction that makes those tokens more or less likely.
Policy staleness: Training on data that an older version of the model sampled. It can call for corrections such as importance weighting, filtering or explicit staleness limits.
Trajectory: One complete attempt by the agent at a task: every model output, tool call and observation, ending in a reward from a verifier.
OPD: On-policy distillation. The student model generates its own trajectories and a frozen teacher scores each of the student's tokens. Related: dual on-policy distillation.
pass@3: The share of benchmark tasks solved in at least one of three attempts.
GPU-hour: One GPU running for one hour. The paper's node has eight B200 GPUs, so 16 hours of training = 128 GPU-hours.

The news. On October 5, 2026, researchers from Stanford University, Yonsei University, Korea University and Bespoke Labs posted ThunderSyncRL (arXiv 2610.05935). They trained Qwen-3.8-27B as a coding and terminal agent with GRPO and with on-policy distillation, on one node of eight NVIDIA B200 GPUs, and evaluated on SWE-bench Verified and Terminal Bench 4.0. The method reached the synchronous baseline's pass@3 targets up to 1.9× faster, and at a fixed 128-GPU-hour budget it scored 1.27 to 2.47 percentage points above asynchronous training. Read the paper →

Picture a teacher who grades a class's exam on a curve. A strict teacher reads no paper until the last student hands in, because no grade is final until the class average is known. When some students finish in ten minutes and others take three hours, that teacher sits idle for most of the exam. Synchronous agent training is the strict teacher: the learner GPUs compute no gradient until the slowest trajectory in the batch has finished and been scored. It is the same wait-for-the-longest problem as static batching at inference time, where one long sequence holds a whole batch. In the paper's GRPO run, each step spent 650.0 s on rollout and verification and then 646.4 s on learner work, one strictly after the other.

The obvious fix is to start grading next week's exam while this week's is still running, using last week's answer key. That is asynchronous RL: rollout and learning overlap across updates, but some trajectories were sampled by a policy that has since changed. This policy staleness can call for off-policy corrections or explicit staleness limits. The curve seems to block anything better. In GRPO, a trajectory's weight in the update is its advantage, its reward minus the group mean divided by the group's standard deviation, and neither number exists until the last trajectory in the group reports its reward.

ThunderSyncRL's key observation is that the curve is applied by multiplication, so the expensive marking can happen first and the curve last. Call one trajectory's gradient g, its reward r, and the group's mean and standard deviation μ and σ. GRPO treats the advantage as a constant while differentiating, so the group update Σ (r − μ)/σ · g rearranges exactly into (1/σ) · (Σ r·g − μ · Σ g). (This simplified form leaves out the paper's division by group size and the small ε it adds to σ for numerical stability; neither changes the argument.) The learner therefore runs each trajectory's backward pass as soon as its reward arrives and adds the result to two running tallies, Σ g and Σ r·g. When the group closes, μ and σ are two scalars and combining the tallies costs almost nothing.

After every group is combined, the system takes one optimizer step, and publishes the new weights to the rollout servers before the next batch begins. No trajectory is ever trained against a policy that changed while it was running. The paper's summary is that the method is asynchronous in execution and synchronous in policy updates.

Trajectory finishesits own reward arrives

Backward pass nowgradient g, no group wait

Update two talliesΣ g and Σ r·g

Group closesμ and σ become known

One optimizer stepthen publish weights

The same rule, start each piece of work once its inputs are fixed, also covers on-policy distillation (OPD). A frozen teacher scores the student's tokens one agent turn at a time. The learner can run a turn's backward pass while that turn's tool call is still executing in the sandbox. That overlap fills the gap between model output and the next observation in one agent tick. The only quantity OPD cannot know early is the total number of scored tokens in the batch, which normalizes the loss, so ThunderSyncRL divides by it at the end. This is the same move continuous batching makes at inference: act on each sequence when it is ready instead of at a batch boundary. The paper proves both equivalences and confirms them numerically on B200 and H200 GPUs.

Schedule	When learner work starts	Policy staleness	Price paid
Sync	After the whole batch and every reward	None	Learner idle 49.5% of each step in the paper's GRPO and OPD runs (paper, Fig. 3)
Async	Overlaps the next rollout batch	One update	Trains on stale-policy data; may need correction or staleness limits
Fully Async	No batch barrier; weights change while trajectories run	Variable	Trains on stale-policy data; may need correction or staleness limits
ThunderSyncRL	Per trajectory (GRPO) or per turn (OPD), once inputs are fixed	None	Extra memory for partial gradients: peak 117.0 → 122.8 GB per learner GPU at most (paper, §4.3)

Where the step time goes. Hold the paper's reference setup fixed: Qwen-3.8-27B, GRPO with 32 prompts × 8 trajectories = 256 trajectories per update, one eight-GPU B200 node. Rollout and verification take 650.0 s in both schedules. Synchronous training then adds 646.4 s of learner work, so its step is about 1,296 s before publishing weights. ThunderSyncRL has done almost all of that learner work while rollouts were still running: only 20.5 s remains after the last reward, and with publication its whole step is 687.7 s. 1,296 ÷ 687.7 ≈ 1.9× shorter steps at zero policy lag. Over a run this compounds: the GRPO target of 74.01% pass@3 on SWE-bench Verified arrived at 121.3 GPU-hours instead of 233.5.

The gain depends on balance: streaming can only hide learner work behind rollout work, so it pays most when the two take similar time. The 27B model is close to balanced. GLM-4.7-Flash-31B activates only about 3B parameters, so its learner work is short, rollout bounds the step, and there is less to hide. The paper also notes that objectives whose gradient inputs resolve early, such as turn-level OPD, leave more learner work to overlap than objectives that wait on group-level signals.

Goes deeper in: LLM Internals → Batching → Static Batching

Related explainers

Rollout-trainer separation with asynchronous weight sync — the asynchronous design this paper compares against, and how it bounds staleness instead of removing it
Reinforcement Learning with Verifiable Rewards (RLVR) — the reward setup that GRPO training of coding agents relies on

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based