When encoding shares a worker with prefill, a text-only request waits behind somebody else's image — head-of-line blocking. Splitting the encoder out is what NVIDIA credits for up to 5× faster time to first token and 42.2% lower mean TTFT for text requests in mixed traffic.

How is it different from prefill/decode disaggregation?

Prefill/decode disaggregation splits the language-model work in two and leaves image encoding attached to prefill, so an image-heavy burst still stalls the prefill pool. EPD adds a third stage, at the cost of moving vision embeddings between workers over NIXL.

When does it not help?

When the encoder work you isolate is smaller than the coordination and transfer you add. NVIDIA is explicit that encode disaggregation helps only in certain scenarios: the gains are strongest for image-heavy inputs, shorter outputs and quantized MoE models, and the benefit disappears once coordination and transfer overhead outweigh the encoder work you isolated.

LLM· 2026-09-10

NVIDIA Dynamo — Encode-prefill-decode disaggregation — What does it mean?

Q: What is encode-prefill-decode disaggregation?

It is a serving layout that runs a multimodal model's vision encoder on its own pool of workers, scheduled independently of the prefill-decode workers that consume its embeddings. The three stages of the request stop sharing one queue.

Jargon

EPD disaggregation: Encode-prefill-decode disaggregation. Running the vision encoder on its own worker pool, scheduled independently of the workers that do prefill and decode.
TTFT: Time to first token: how long a user waits between sending a request and seeing the first word come back. It is dominated by queueing plus prefill, which is why encoder work landing in the same queue hurts it.
ITL: Inter-token latency: the gap between consecutive output tokens once generation has started. Benchmarks usually fix an ITL ceiling (here 100 ms) and then ask how much traffic the system can carry under it.
Goodput: Requests per second that actually met their latency target — their SLO, or service-level objective — as opposed to raw throughput, which counts every request whether it was fast enough or not. More on the difference
Head-of-line blocking: When the item at the front of a queue is slow, everything behind it waits, however cheap those items are. A queue-shape problem, not a compute problem.
NIXL: NVIDIA Inference Xfer Library, the transport Dynamo uses to move tensors between workers. In EPD it carries vision embeddings from the encoder worker to the prefill-decode worker.
NVFP4: A 4-bit floating-point format NVIDIA uses for weights on Blackwell-class GPUs. Smaller weights make the language-model half of the pipeline cheaper, which changes where the bottleneck sits.
MoE: Mixture of Experts: a model where each token is routed to a small subset of the available feed-forward experts, so only part of the network runs per token. It makes the language half cheaper per token, which shifts where the serving bottleneck sits.

The news. On September 9, 2026, NVIDIA published a walkthrough of encode-prefill-decode disaggregation in Dynamo, its open-source inference serving framework. Benchmarking Qwen3.5 122B A10B with NVFP4 weights on four GB200 GPUs under a 100 ms ITL target, NVIDIA reports up to 5× faster time to first token and up to 7× faster end-to-end response time — with the gains strongest for image-heavy inputs, short outputs and quantized MoE models, and explicitly conditional: NVIDIA's own summary is that encode disaggregation helps only in certain scenarios. Read the walkthrough

Stay in the hall for one more minute. Everyone is heading to the same gate, but they are not carrying the same thing. A passenger with a backpack needs thirty seconds at a desk. A passenger with three ski cases needs five minutes of lifting, weighing and tagging — and for those five minutes the backpack passenger behind them does nothing but stand there.

A multimodal request has exactly that shape, and the image is the ski case. Before a language model can read a prompt containing a picture, a separate vision encoder has to turn that picture into a block of embeddings: vectors the model can attend over the same way it attends over text tokens. That encoding is real GPU work — a full forward pass through the vision encoder — and in the ordinary layout it runs on the same worker that is about to do prefill for everybody else in the queue.

So a text-only request that needed nothing but a fast prefill sits behind a photograph it has no interest in. That is head-of-line blocking, and the fix is the one the airport already uses: give the heavy work its own desk.

encodeimage to embeddings

prefillread the whole prompt

decodeone token at a time

EPD adds that third desk: the vision encoder gets its own workers, scheduled independently of the prefill-decode workers that consume what it produces. Encoder workers take the media, run the vision encoder and emit embeddings; the prefill-decode workers pick those embeddings up and do the language half. Note what did not move — prefill and decode stay together here, so EPD composes with prefill/decode disaggregation rather than replacing it. Dynamo will run both roles on the same GPUs — colocated EPD, where the split buys you scheduling isolation and nothing has to travel — or place the encoders on cheaper GPUs such as the RTX 6000D and move the embeddings to the GB200 workers over NIXL.

The second arrangement is the interesting one, because it says something about the shape of the data. You can afford to put a pipeline stage on a different machine when the thing it hands downstream is small. A per-image embedding can be far smaller than the KV cache a long conversation accumulates, which is what makes the belt affordable here and why moving a KV cache between pools is the harder version of the same trick. Where the tensor has to cross, NVLink and PCIe bandwidth set the price.

In mixed traffic — the realistic case, where text and image requests share one endpoint — NVIDIA reports 42.2% lower mean TTFT for the text requests and 30.8% lower for the image requests. Note which number is bigger. The requests that gain most are the ones that were never the problem: the backpack passengers who were only ever waiting because of where the queue put them.

Put numbers on that. Take a stream that is 70% text and 30% image, and say the text requests average a 900 ms mean TTFT and the image requests 2,400 ms before the change (baselines illustrative — NVIDIA publishes the reductions, not the absolute means). Blended, that is 0.7 × 900 + 0.3 × 2,400 = 1,350 ms. Now apply the reported cuts: text drops to 900 × 0.578 ≈ 520 ms, image to 2,400 × 0.692 ≈ 1,661 ms, and the blend becomes 0.7 × 520 + 0.3 × 1,661 ≈ 862 ms. 1,350 ms to 862 ms, a 36% cut in blended mean TTFT — and more of that saving lands on the text users (about 266 ms of the 488 ms) than on the image users, even though the images were the heavy work. They were only ever paying for a queue they never joined.

Throughput: 10 req/s

Goodput: 3 req/s7 violated SLO

4×

5×

6×

7×

8×

9×

10×

Same system, same second — throughput looks healthy, goodput tells the truth

Latency is only half of it. The other half is how much traffic you can carry while still meeting the target, which is goodput rather than throughput. On the colocated EPD setup, NVIDIA reports same-SLO goodput rising from 1.78× to 2.64× once the language-model weights are in NVFP4. The same structural change is worth more when the language half gets cheaper and the encoder becomes a proportionally larger share of the work you isolated.

Layout	Where the vision encoder runs	What it costs	Reported effect
No disaggregation	Same worker as prefill and decode	Nothing to move	Baseline; image work blocks text requests (NVIDIA)
Prefill/decode split	Still attached to the prefill stage	KV cache transfer between pools	Separates prefill from decode; encoder work still lands on the prefill pool
Colocated EPD	Own worker role, same GPUs	Scheduling only	Same-SLO goodput 1.78× to 2.64× with NVFP4 weights (NVIDIA)
Disaggregated EPD	Separate, cheaper GPUs (RTX 6000D)	Embedding transfer over NIXL	Dynamo supports lower-cost RTX 6000D encoder GPUs and transfers embeddings to GB200 workers through NIXL (NVIDIA)

Which brings the honest part, and it is the reason this article is worth reading twice. NVIDIA's own framing is conditional: encode disaggregation reduces TTFT and raises same-SLO goodput only in certain scenarios. Encode disaggregation pays only when the encoder work you isolate is larger than the coordination and transfer you add. Short prompts with one small image, long generations where decode dominates anyway, a hall where almost nobody is carrying skis — in those the extra desk is a walk to a counter that had no queue. Building an oversized-baggage desk in an airport where everyone travels with a backpack does not speed anybody up; it just adds a walk.

That conditionality is the transferable lesson, not the 7×. Full disaggregation is a scheduling decision, and every scheduling decision is a bet on the shape of your traffic. Measure the shape first.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Full Disaggregation

Related explainers

AMD ATOM + ATOMesh — Prefill/decode disaggregation on ROCm — the two-stage version of the same idea, and the transport problem it creates
Gemma 4 12B — Encoder-free multimodal — the opposite bet: delete the vision encoder rather than give it its own pool

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based