The news. On August 24, 2026 a study measuring masked diffusion language model serving appeared on arXiv. Running LLaDA-8B-Instruct with a D2F LoRA adapter on a single NVIDIA H200, it reports that only 24% of a single request's wall-clock time is GPU computation; that sharing one forward pass per denoising step across a batch raises throughput 16.0× at batch size 16; that request difficulty falls into 11 discrete step-count levels whose best pre-generation predictor reaches only R² = 0.150; and that accuracy sits at 74–76% on GSM8K — a grade-school word-problem benchmark — at single-request scale. Read the paper →
Picture the kiln. Loading it, sealing it and bringing it up to temperature is most of the day's work, and none of that changes with how many pots are on the shelf. A potter who fires one mug at a time is not spending their day on mugs; they are spending it on kilns. The whole economics of the workshop turns on one fact — the expensive thing is the firing, not what is inside it — and the fix is not a faster kiln but a fuller one.
That kiln is a GPU serving a masked diffusion model, and the study's headline measurement is the potter's problem stated in numbers: 24% of a single request's wall clock is GPU computation, so roughly three quarters of the time is the loading and the heating — Python composing the call, the framework launching kernels, the host synchronizing and copying. This is not a new observation about serving in general; batching exists because GPUs are wasted on one request at a time. What is specific here is how much of the clock is overhead, and what the model's shape lets you do about it.
Naive: One at a Time
Step 1 — only R1 runs
Step 2 — only R2 runs
Step 3 — only R3 runs
GPU reads weights 3× for 3 requests
Engine: All at Once
Step 1 — all 5 run together
GPU reads weights 1× for all 5 requests
Gray = idle GPU capacity. Batching fills the GPU on every step.
The picture above is the familiar version of the argument, drawn for autoregressive models: run one request per step and most of the GPU sits idle, run five together and one pass serves all five. Read it here with one substitution — the quantity being shared is not only the weight read but the launch itself. That is the part the measurement moves to the front. When three quarters of a step is host-side overhead that does not grow with the batch, the batch is not just filling idle silicon; it is dividing a fixed bill.
Now the part that is genuinely diffusion's. An autoregressive server cannot simply line its requests up, because they are at different places: one is on token 12, another on token 900, a third has just arrived. Continuous batching is the machinery built to cope with that — requests join and leave the running batch every iteration, and the scheduler does real work each time to keep the batch composed. A masked diffusion model shrinks the problem rather than managing it: every request advances in the same unit — one denoising step, revealing masked positions across the whole sequence at once — instead of appending one token at the end. Requests still differ in how many steps they need in total, but at any given moment each one is waiting for the same kind of step. Because the unit of progress is common, the server can hold live requests at a step boundary and let one forward pass carry all of them — which is the pot-and-kiln arrangement exactly, and it is not available to a lathe.
Hold one step fixed and price it both ways. Say a denoising step takes 100 ms of wall clock and a request needs 100 of them (illustrative — the study reports the ratio and the speedup, not these absolutes). The measurement says 24 ms of that 100 is GPU math and 76 ms is everything else. Serve sixteen requests one at a time and each pays its own full step: 16 × 100 steps × 100 ms = 160 s. Now synchronize the sixteen on the step boundary so they enter the same pass. The 76 ms of overhead is paid once per firing, not once per pot, and at this size the GPU half absorbs the extra fifteen sequences without a proportional cost — so sixteen requests' worth of progress comes out of 100 shared steps: 10 s. Sixteen times the work in the same wall clock, which is the shape behind the reported 16.0× at batch size 16. A speedup that lands that close to linear is the signature of a cost that was fixed per step rather than per request — in this measured setup the kiln was not the bottleneck, the number of firings was.
That leaves the scheduler with a real problem, and the study names a rule for it. A batch amortizes best when it is full, so the admission rule — the fixed-fill timeout — decides how long to hold the door open for one more request before starting the step anyway. Sizing that wait would be easy if you knew how much work each request represents, and here you do not: the study finds request difficulty lands in 11 discrete step-count levels, and the best predictor available before generation starts reaches R² = 0.150. You can see, in other words, that requests differ sharply in cost, and the signal available before generation accounts for only about 15% of that variation — the glaze problem, where the raw clay tells you very little about how many firings it will need.
| Serving question | Autoregressive | Masked diffusion, as measured |
|---|---|---|
| What one pass produces | One more token per sequence (background) | One more denoising step over the whole sequence (source) |
| Are live requests aligned? | No — each sits at its own token position | At the step boundary, yes — all mid-step, though not at the same step count |
| How the batch is formed | Continuous batching: join and leave every iteration | Synchronize at a step boundary, one shared forward pass |
| Reported share of wall clock that is GPU math | Not measured here; see the async continuous-batching report | ~24% at single-request scale (one model, one H200, setup-dependent) |
| Reported throughput gain from sharing the pass | — | 16.0× at batch size 16 (source) |
| Can you predict a request’s cost up front? | Output length is also hard to predict | 11 discrete step levels; best predictor R² = 0.150 (source) |
The transferable idea is about where a serving system's time actually goes, and it generalizes past diffusion. Two systems can spend the same wall clock per request and need opposite fixes: if the clock is mostly GPU math, you make the math cheaper — better kernels, lower precision, less attention. If the clock is mostly the machinery around the math, those optimizations only reach the smaller share, and the levers that matter are different ones: doing more per launch, or overlapping the host work with the GPU work so the two stop taking turns. Measuring the split is therefore not a footnote to the optimization; it is the optimization, because it tells you which of the two systems you are running. What makes masked diffusion the clean case is that its common step unit hands the scheduler that alignment cheaply — elsewhere, sharing a pass means composing the batch first.
Goes deeper in: LLM Internals → Batching → Continuous Batching
Related explainers
- HuggingFace blog — Async continuous batching — the autoregressive answer to the same complaint: overlap the host work with the GPU work instead of sharing one pass
- PSD — Parallel speculative decoding for diffusion LLMs — speeding the same model class from inside the decoding algorithm rather than from the serving loop
- Diffusion Gemma — Parallel block decoding — where the parallel-position property this batching scheme relies on comes from