The news. On July 6, 2026, the Selective Importance Sampling paper (arXiv 2607.04728), titled Turning Off-Policy Tokens On-Policy, proposed a plug-in for RL post-training that reuses off-policy rollouts without the usual variance blow-up. It views the old model as a proposal distribution, then runs a token-level rejection test: tokens the current model would likely have produced anyway are accepted as on-policy, and only the rest keep the importance-sampling correction. The authors report it consistently improves results across both dense and mixture-of-experts models on math and agent benchmarks. Read the paper →
Picture a friend hands you a grocery receipt from last year and asks what it would cost today. You cannot just read off the total — some prices moved. So you go line by line and multiply each item’s old price by a conversion factor: today’s price divided by last year’s. The trouble is the receipt is long, and to value the whole thing you multiply every one of those factors together — so a few items that swung hard can throw off the entire total.
That receipt is exactly the problem in RL post-training. To save compute, you train on answers the model already generated a few steps ago — but the model has moved on since, so those rollouts are now off-policy. To count an old answer fairly under the current model, importance sampling reweights each token by its importance ratio: the current model’s probability for that token divided by the old model’s. One token is a single line item; a full answer is the whole receipt.
Here is the catch, and it is the whole reason SIS exists. A sequence-level weight is the product of every token’s importance ratio, so on a long answer you multiply many factors — and even when each is close to 1, the product swings hugely from one sampled answer to the next. That run-to-run swing is variance, and high variance makes the policy gradient noisy and the training unstable. The longer the answer, the worse it gets.
Now the SIS move. Selective Importance Sampling runs a token-level rejection test that accepts the tokens where the old and current models basically agree, treats each of those as on-policy, and pins its importance ratio to exactly 1 — leaving the correction only on the tokens it rejects. Back on the receipt: for the items whose price barely changed, you just keep the old number instead of bothering to convert it, and you only re-price the few items that actually moved. Crucially, SIS changes only the importance ratio inside the policy loss — nothing else in your RL algorithm — so it drops in as a low-overhead plug-in.
Walk it through (illustrative numbers). Take a 200-token answer where 195 tokens have importance ratios jittering between 0.95 and 1.05 — each almost 1, none exactly. Any single one is harmless, but the sequence weight is all 200 multiplied together, and that product drifts a different way every time you sample: one rollout might multiply out to 0.6, the next to 1.7 — a ~2.8× swing driven by noise alone. SIS accepts those 195 near-1 tokens as on-policy and pins each to a ratio of exactly 1, so instead of 200 jittering factors you multiply only the 5 tokens that truly differ. Its variance drops sharply, because far fewer factors multiply — at the price of only a small approximation on the tokens it pins to 1. The paper frames this as shrinking the gap between the noisy token-level estimator and the stabler sequence-level one.
| Approach | What it does to each token | Variance on long answers |
|---|---|---|
| Reuse off-policy rollouts, no correction | use the old model’s answer as-is | low variance, but a biased estimate — you are optimizing the wrong distribution |
| Standard importance sampling | multiply every token by its ratio (new prob ÷ old) | unbiased, but variance explodes as the product of many ratios grows with length |
| Selective Importance Sampling (arXiv 2607.04728) | accept agreeing tokens as on-policy (ratio pinned to 1), correct only the rest | far lower variance — far fewer factors multiply, with only a small approximation on the accepted tokens |
The takeaway is a small, surgical change in where you spend your correction. You do not need to reweight every token to reuse off-policy data — you keep the correction on the tokens the test rejects, and accept the rest as on-policy. Because SIS touches only the importance ratio in the loss, it slots into existing RL post-training with almost no extra cost, and it keeps reused rollouts from destabilizing the very model they are meant to improve.
Goes deeper in: LLM Internals → Text Generation → From Logits to Probabilities
Related explainers
- RLVR — Reinforcement Learning with Verifiable Rewards — the pass/fail post-training loop SIS plugs its correction into
- VPO — vector-reward advantage vs GRPO scalar — another fix to the estimator inside policy-gradient post-training
- DRPO — smooth trust-region penalty — a different way to keep off-policy updates from destabilizing training
- Reasoning Arena — Bradley-Terry trace ranking — what to do when the RLVR reward itself is too coarse