What is two-stage SLERP merging of per-axis GRPO experts?

It is a post-training recipe with two halves. First, each capability axis — instruction-following, function-calling, the internal data distribution — gets its own GRPO run, so their reward signals never compete inside one optimization. Second, the resulting checkpoints are combined in weight space by spherical linear interpolation applied twice, two checkpoints at a time, producing one model that carries all three capabilities.

Why merge the experts instead of training one model on every objective at once?

Because a serving fleet wants one checkpoint, but one optimization carrying every reward pulls the weights in competing directions. The study reports diagnosing reward failures it names semantic collapse, over-calling and verbosity hacking before splitting the axes apart. Training them separately removes the direct competition; the merge is what puts the fleet back down to a single model.

Why spherical interpolation rather than averaging the weights?

Averaging two vectors element by element cuts across the chord between them rather than following the arc, so for two vectors of the same length the result is pulled shorter than both and sits off the sphere they occupy — the compass version is adding two bearings and dividing. SLERP rotates along the arc instead, keeping the blend's length and direction well-formed. The study reports using two-stage SLERP but does not publish a measured comparison against element-wise averaging.

How much traffic does the merged model actually serve?

The study reports the merged model handling 50% of production traffic across about 200 applications, equal to 116 million requests per month, and scoring 69.6 aggregate against 65.8 for a roughly 7x larger baseline.

LLM· 2026-09-04

Merge per-task GRPO experts to serve 116M monthly requests — Two-stage SLERP merging of per-axis experts — What does it mean?

TL;DR

What is it: A production study post-trains its serving model by training one GRPO expert per capability axis and then combining those checkpoints with two-stage SLERP merging: spherical interpolation run twice, two checkpoints at a time, over the models' weights.
Why it’s needed: A serving fleet wants one model, not three, but the three jobs it must do — following instructions, calling functions, answering in-house questions — have reward signals that pull the weights in different directions; merging lets one checkpoint carry all three without running one fleet per job.
vs previous: The usual route is a single multi-objective GRPO run with every reward mixed together; this study instead reports diagnosing reward failures it names semantic collapse, over-calling and verbosity hacking, then training the axes apart so the signals never compete, which moves the blending out of the optimizer and into a post-training merge.

Jargon

GRPO: Group Relative Policy Optimization. A reinforcement-learning method for post-training a language model: it samples a group of candidate answers and scores each one relative to the rest of its group, instead of against a separately trained value model. See VPO on what a single scalar reward costs.
SLERP: Spherical linear interpolation. A way to blend two vectors — here, two trained checkpoints' weights — by rotating along the arc between them rather than averaging component by component, so the blend keeps the same length and stays on the sphere the two originals sit on.
Capability axis: One job you want the model to be good at, treated as its own training target. In this study the axes are instruction-following, function-calling and the organization's internal data distribution.
Reward interference: What happens when two reward signals in the same optimization pull the weights in opposite directions, so improving one job quietly degrades another. It is the failure mode the per-axis split is there to avoid.
Semantic collapse: A reward-hacking failure where a model's answers converge on a narrow, safe phrasing that scores well but carries less information. The study names it as one of the reward failures it diagnosed.
Over-calling: Invoking tools more often than the task needs, because the reward pays for the shape of a tool call rather than for solving the problem. Also named as a diagnosed reward failure.
Verbosity hacking: Padding an answer with extra length because the reward happens to correlate with length. The third reward failure the study names.
Production-stratified evaluation: Building the eval set so its mix of task types matches the mix in real production traffic, instead of using an off-the-shelf benchmark blend that may weight the jobs quite differently from your users.

The news. On September 1, 2026, a production study described how a self-hosted model serving about 200 applications is post-trained and merged. Rather than one reinforcement-learning run carrying every objective, the team splits post-training into three capability experts — instruction-following, function-calling and the internal data distribution — each with its own GRPO run, then combines the resulting checkpoints with two-stage spherical interpolation. The merged model is evaluated against a set stratified by real production traffic and then routed into the serving fleet, where it now handles half of all requests. Read the paper →

Picture three navigators on the same bridge, each locked onto a different bearing, all reaching for the same wheel. That is a single multi-objective training run: one optimizer, three reward signals, and every gradient step is a compromise none of the three asked for. The study reports diagnosing three reward failures before it split the axes apart, and names them — semantic collapse, over-calling, and verbosity hacking. Each is a different way of scoring well while getting worse at the actual job.

So the study takes the hands off the wheel. Each capability axis gets its own GRPO run, so the reward signals never compete inside the same optimization — one navigator per bearing, each steering alone. What comes out is three separate checkpoints: three headings, no ship.

Now the blending problem, and it is where the word spherical earns its place. The obvious way to combine three checkpoints is to average the weights element by element. On a compass that is adding the bearing numbers and dividing, and because every bearing on a compass has the same length, the average always lands inside the ring rather than on it — a heading pulled shorter than the ones it sits between. SLERP instead turns along the ring: it interpolates on the arc between two weight vectors, so the blend keeps the length it is interpolating between and stays on the sphere the originals occupy.

SLERP takes two inputs at a time, which is why the merge has two stages: blend the first pair, then blend that result with the third expert. The study reports using this two-stage spherical interpolation rather than a single multi-objective run, but it does not publish a head-to-head against element-wise averaging — so read the arc-versus-chord contrast as what SLERP is, not as a measured margin from this paper.

3 GRPO runsone per axis

SLERP stage 1blend two

SLERP stage 2add the third

Serving fleet50% of traffic

Route	How the reward signals meet	What the study reports
One multi-objective GRPO run	All objectives inside one optimization, competing step by step	The alternative the study does not take; it reports diagnosing semantic collapse, over-calling and verbosity hacking (source)
Per-axis GRPO experts	Never — each axis is optimized alone	Three expert checkpoints: instruction-following, function-calling, internal distribution (source)
Two-stage SLERP merge	Only after training, in weight space, two checkpoints at a time	69.6 aggregate against 65.8 for a roughly 7x larger baseline (source)
Merged model in the fleet	Not at all — one checkpoint behind about 200 applications	50% of production traffic, 116 million requests per month (source)

A merge is only as trustworthy as the scoreboard that approves it, and this is the third leg of the study's pipeline: the evaluation set is stratified by real production traffic before the merged model is routed into the fleet. A generic benchmark blend would weight the three axes however the benchmark's authors happened to weight them; stratifying by traffic weights them the way this particular fleet's users do. That difference is exactly the online-versus-offline eval gap — an offline number can look fine while the mix it was measured on has little to do with what the inference engine actually receives.

Hold two of the study's reported numbers fixed and the fleet arithmetic falls out. The merged model carries 50% of production traffic, and that half is 116 million requests per month — so the fleet as a whole runs about 232 million requests a month, with the merged checkpoint answering roughly 116 million of them. Next the size comparison: the study reports the merged model scoring 69.6 where a roughly 7x larger baseline scores 65.8, a gap of 3.8 points in the smaller model's favour — so on this traffic the merge is not a compromise for the sake of consolidation. Push that one step further with an illustrative cost model: if serving cost tracked parameter count one-for-one, running those same 116 million requests on the 7x baseline would cost about 812 million requests' worth of the smaller model's compute instead of 116 million (illustrative — the study reports no cost figures, and real serving cost also depends on batching, quantization and hardware, which the agent cost profile breaks down).

Goes deeper in: Agent Engineering → Production Evals → Online vs Offline Evals

Related explainers

VPO paper — Vector-reward advantage vs GRPO scalar collapse — the other half of this problem: what a single scalar reward loses before you ever reach the merge.
DRPO: smooth trust-region regularizer replaces hard masks in LLM RL — how a GRPO-style run is kept stable while it trains.
MiniMax-M2 — Forge RL prefix-tree merging — merging in the same pipeline, but applied to rollouts during training rather than to weights after it.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based