Decoupling prefill and decode power saves 32.3% of a lane pair's electricity — Phase-decoupled power control — What does it mean?

The news. On September 10, 2026, a paper titled Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving was posted to arXiv. Its authors deployed NVIDIA's Max-Q inference profile on a disaggregated 8× B200 system and found the realized gain modest, model-dependent, and carrying a mean end-to-end latency cost that throughput-only evaluation never surfaces — and, more fundamentally, that the profile applies one setting to prefill and decode GPUs that operate in opposite hardware regimes. Their alternative sets one policy per lane. Read the paper →

Picture the oven and the slow cooker again, both wired to the same fuse box. The oven has one job everybody notices: get hot fast, so dinner is not late. The slow cooker has the opposite job — sit at a low, steady draw for hours. Fit a single dial to the fuse box and turn it down to save electricity, and you have made the oven late while barely touching the cooker, because the cooker was never the thing pulling hard.

That is the shape of modern LLM serving. Production systems now split a request across two pools of GPUs: a prefill lane that reads the whole prompt in one dense pass, and a decode lane that emits one token at a time. Those two phases sit on opposite sides of the roofline — prefill saturates the arithmetic units, decode waits on memory bandwidth with most of the compute idle. Prefill and decode run on the same silicon in opposite hardware regimes, so a single power setting is necessarily wrong for one of them.

So the controller gives each lane the rule that suits its appliance. The oven's rule is a floor: the prefill lane runs inside an SM-clock window whose lower bound is chosen so that the latency guarantee holds by construction. A window has two edges, and the one that matters here is the lower one: it makes the worst case a number you picked rather than a number you hope for.

The cooker's rule is a ceiling, and it only works because of disaggregation. A disaggregated decode lane draws flat, memory-bound power, so a cap binds continuously instead of fighting bursts — which is exactly why capping had been rejected before. When prefill and decode share a GPU, demand swings between the two regimes and a cap has to chase them; the paper names that reactive overshoot as the weakness that led POLCA to reject capping. Separate the lanes and the weakness is absent. The cap then sits there quietly, and the GPU's own power manager retains throughput under it.

The two numbers are settled in different ways. The decode cap is placed empirically: automatic calibration finds the lane's throughput/latency cliff — the point below which shaving another watt stops being free — and sets the cap just above it, under a runtime service-level-objective (SLO) guard. The prefill floor is justified differently: it is a latency guarantee by construction. Behind both sits the paper's real claim: the right power setting is a property of the deployed combination of model, quantization, engine and hardware, not of the GPU class. Because the cliff moves with the model, a profile shipped against a GPU SKU cannot be right for every deployment on it.

The evidence for the SLO guard is the part worth reading twice. On Qwen3-235B-A22B (NVFP4, a 4-bit number format), every one of the controller's operating modes met the ITL p99 target in every repetition, while both vendor profiles missed it. An efficiency setting that violates the tail is not an efficiency setting; it is a quietly broken SLO.

Both measured rows come from the same 8× B200 node serving Qwen3-Coder-480B in FP8 (the 8-bit equivalent) under agentic load (multi-step, tool-calling request traffic); on another model or engine the numbers move, which is the paper's point.

Hold that setup fixed and the two policies reduce to two ratios against one baseline — but they are ratios of efficiency, not of energy, and the difference is where the intuition usually breaks. For a fixed number of tokens, energy is the inverse of efficiency. Max-Q's +8.6% tokens/J means the job now needs 1 / 1.086 = 92.1% of the baseline's energy, a 7.9% cut. The phase-decoupled controller's +20.4% means 1 / 1.204 = 83.1% — a 16.9% cut in energy per token, more than twice what Max-Q recovers — and it gets there while adding less latency, +3.5% against +5.2%. A separate measurement, over a different horizon, puts a number on what that compounds to: a three-day sustained run saved 32.3% of a lane pair's electricity — one prefill lane plus the decode lane it feeds.

Two limits travel with the result. The paper scopes its claims to MoE serving: both tested models are mixture-of-experts, and a dense model recovered roughly 5× less energy benefit. The paper reports the gap without explaining it. The second limit is structural: this technique spends latency headroom. Where a service already runs at the edge of its SLO there is nothing to convert, and the paper does not report how calibration behaves in that case.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Full Disaggregation

Related explainers

• AMD ATOM + ATOMesh — Prefill/decode disaggregation on ROCm — the lane split this controller assumes, built out on another vendor's stack.
• NVIDIA AI Factories — Tokens-per-megawatt as a serving metric — the same energy question one level up, at the datacenter.
• Mix-Quant paper — NVFP4 prefill + BF16 decode — another knob set per lane, this time numeric precision instead of watts.