The news. On August 24, 2026, NVIDIA said it had put Groq 3 LPX into full production as an extension to Vera Rubin NVL72, The arrangement NVIDIA describes is that Rubin GPUs handle large-scale context processing while LPX accelerates latency-sensitive decode, with the two accelerator types jointly computing model layers. An Artificial Analysis benchmark on Gemma 4 31B at a 100,000-token context is reported at 3,400 output tokens per second, which NVIDIA puts at 4x the nearest alternative platform. A rack-scale deployment can reportedly include up to 256 LP30 accelerators. The same announcement covers Spectrum-X Multiplane, which splits each server connection across independent network planes and reroutes failures in hardware — quoted as scaling to 512,000 GPUs and retaining about 90% of bandwidth through a failed plane, with 11x faster recovery than software load balancing. Read the release →

Picture a building site with two machines on it. There is a tower crane, which exists to make one enormous lift: the whole steel frame goes up in a single swing, and what limits the crane is raw lifting power. There is also a forklift, which exists to make short round trips: one pallet from the store yard to the building, then back, then again, all day. What limits the forklift is not lifting power at all — it is how far away the store yard is. Buy a bigger crane and the forklift does not get one minute faster.

That site is a language model serving a request, and the two machines are its two phases. Prefill and decode are the same arithmetic run under opposite constraints. Prefill is the crane: the model reads the entire prompt in one parallel pass, so thousands of tokens ride along on a single read of the weights and the chip is genuinely busy doing arithmetic. Decode is the forklift: the model emits one token, and to do it must haul every weight out of memory again — an entire trip to the store yard for a single pallet.

The roofline model gives this its usual name. Every workload has an arithmetic intensity — operations performed per byte moved — and that one number decides which ceiling it hits. Prefill sits high on that axis and runs into the compute roof, while decode sits far to the left and runs into the bandwidth roof, well below the chip's peak arithmetic. Two dots, one plot, opposite sides.

Hold the model and the chip fixed and the gap is easier to see than to argue about. Take Gemma 4 31B, roughly 31 GB of weights at 8 bits each, on a device with 8 TB/s of memory bandwidth and 500 TFLOP/s of usable arithmetic. To produce one decode token, the chip must read all 31 GB — that is 31 / 8,000 = about 3.9 ms of pure fetching — and perform about two operations per parameter, or 62 GFLOP, which at 500 TFLOP/s takes about 0.12 ms. So the arithmetic occupies roughly 3% of the step and the other 97% is waiting on memory. Now run prefill over a 100,000-token prompt instead. The weights are still read once, still 3.9 ms. But the arithmetic is now 100,000 times larger — 6.2 PFLOP, about 12.4 seconds. Same weights, same chip: arithmetic goes from about 3% of the time to about 99.97% of it, purely because 100,000 tokens rode along on one read of the weights instead of one token (illustrative: the bandwidth and throughput figures are stand-ins, not NVIDIA's, and this ignores the KV cache, which only pushes decode further toward the memory side).

Once you accept that, the interesting question stops being how do I make one chip faster and becomes which chip should each phase run on. Splitting the phases is not new — chunked prefill interleaves them on one machine to stop long prompts stalling everyone's decode, and full disaggregation puts them on separate GPU pools and ships the KV cache between them. What is new here is that the two sides are no longer the same kind of chip: prefill stays on GPUs and decode moves to a different accelerator architecture, with both computing the same model's layers. Why that helps is an inference from the roofline argument above, not something the release quantifies: NVIDIA states which phase each accelerator is assigned, but publishes no memory-bandwidth or arithmetic specification for LPX, so treat "built for decode" as a description of its job rather than of its datasheet. Back on the building site: nobody was arguing about whether to use the crane and the forklift for different jobs. The change is that the site stopped trying to do both jobs with two cranes.

The honest caveats are worth stating, because a rack-scale claim is easy to over-read. The 3,400 tok/s figure comes from one benchmark, on one model, at one context length, and NVIDIA is quoting it about its own product; a workload with short prompts and long replies will sit somewhere different, because the whole argument turns on the ratio between prompt tokens and generated tokens. Splitting phases across chip types also adds a boundary that a single device does not have, and the release does not say what crosses it or how: "jointly compute model layers" is a looser description than the clean prefill-then-handoff of classic disaggregation, and it could mean either. Whatever the boundary turns out to be, the pairing only pays off where decode was the bottleneck and the cost of crossing stays small next to what decode was already waiting on — the goodput question, not the peak-throughput one. The durable part of this news is not the benchmark number and not the product; it is that inference has stopped being a workload you buy one chip for.

Goes deeper in: LLM Serving → Prefill vs Decode Disaggregation → Full Disaggregation

Related explainers

• AMD ATOM + ATOMesh — Prefill/decode disaggregation on ROCm — the same split, but across two pools of the same accelerator: the closest neighbour and the direct contrast
• OpenAI and Broadcom's Jalapeño — Inference ASIC vs GPU — the substitution answer to the same problem: one purpose-built chip replacing the GPU rather than sitting beside it
• NVIDIA Vera Rubin NVL72 — Rack-scale NVLink domain — the rack the LPX accelerators are being attached to, and the interconnect it is built around