How a Request Flows Through vLLM — Visual Source Walkthrough
The Whole Path in One Picture
What is the vLLM request path?
The vLLM request path is the eleven-phase journey one HTTP request takes from POST /v1/chat/completions to the last streamed token — crossing two operating-system processes on the way. An API server process renders and tokenizes your prompt, hands it over a socket to a separate engine-core process that owns the GPU, and then reads generated token ids back over a second socket to detokenize and stream them out.
That is the whole story. The rest of this module is coordinates: for each of the eleven phases, which process runs it and which file to open.
Why coordinates and not just concepts?
You can already describe what an inference server does. What you probably cannot do — and what stops most people from ever contributing to vLLM — is answer the question "where would I put a print statement to see that happen?"
So every claim in this track lands on a real file. Every code block is distilled from a pinned release (v0.26.0) and carries a footer with its upstream path, its symbol, and the true size of the file it came from. A 30-line excerpt from a 7,846-line file is a reading aid, not a summary — the footer keeps that honest.
The panel on the right is the map for the whole track. Eleven chips, one per phase, colour-coded by owning process: blue for the API server, green for the engine core, violet for the worker. Click any chip to see its phase title and the real symbol names that run there. The same strip sits above every step in this module as a "you are here" bar.
The eleven phases
| # | Phase | Process | Directory to open |
|---|---|---|---|
| 1 | HTTP arrival, chat-template rendering, tokenization | API server | vllm/entrypoints/openai/chat_completion/, vllm/renderers/ |
| 2 | Frontend request registration | API server | vllm/v1/engine/ |
| 3 | Submission across the process boundary | API server | vllm/v1/engine/ |
| 4 | Engine-side receive, decode, enqueue | engine core | vllm/v1/engine/, vllm/v1/core/sched/ |
| 5 | Admission, token budget, preemption | engine core | vllm/v1/core/sched/ |
| 6 | KV block allocation, prefix reuse, eviction | engine core | vllm/v1/core/ |
| 7 | Batch assembly, forward pass, logits | worker | vllm/v1/worker/ |
| 8 | Logits processors, sampling, token ids | worker | vllm/v1/sample/ |
| 9 | Scheduler state update, engine output built | engine core | vllm/v1/core/sched/ |
| 10 | Engine output back across the boundary | engine core → API server | vllm/v1/engine/ |
| 11 | Output processing, detokenization, streaming | API server | vllm/v1/engine/, vllm/entrypoints/openai/chat_completion/ |
Read the Process column and a shape appears: blue, blue, blue → green, green, green → violet, violet → green → grey → blue. Out and back. The two socket crossings are phases 3 and 10, and they are the only two places in the path where data leaves one process's memory for another's.
Three things about this path that surprise people
It is not one pass — the middle is a loop
Phases 1 through 4 happen exactly once per request. Phases 5 through 9 happen once per forward pass, and for a request that is already generating, each pass normally advances it by one token. (Not always one: a long prompt can be chunk-prefilled across several passes before it produces anything, and speculative decoding can land several tokens in one pass.) A 500-token answer means those five phases ran on the order of 500 times for your request, interleaved with every other request in flight.
The scheduler is therefore not deciding "which request runs next." It is deciding, before every single forward pass, which set of requests moves one step. That is continuous batching, and it is the reason a phase-5-to-9 loop exists at all.
Phases 7 and 8 are two separate calls into the worker, not one
The natural way to narrate a forward pass is "run the model, get a token." The code does not do that: running the model and choosing the token are two separate calls, and the engine does useful work in between.
You can see the seam in the right-hand panel — the connector between chips 07 and 08 is marked ⟲ 2 calls rather than a plain arrow. Step 5 puts the fifteen lines of real code in front of you and explains what the gap buys.
Nothing runs on the GPU until phase 7
Phases 1 through 6 are all CPU work: parsing JSON, applying a Jinja chat template, running a tokenizer, serializing a message, walking a queue, and bookkeeping block indices. On a short prompt that is microseconds; on a long one with images it is not. It is also why vLLM puts phases 1-3 in a different process from phases 4-9 — the topic of the next step.
Say it out loud
You should now be able to narrate the whole path without notes. Try it against the strip above the panes:
- HTTP request arrives at the API server; the chat template renders and the prompt is tokenized.
- The frontend registers the request's output-side state first, then submits it.
- The request is msgpack-encoded and pushed over a socket.
- The engine-core process receives it, decodes it, and enqueues it for the scheduler.
- The scheduler admits some subset of waiting and running requests under a token budget.
- KV blocks are allocated for whatever it admitted, reusing any blocks that already hold the same prefix.
- The worker assembles the batch, runs the forward pass, and parks the logits.
- A second worker call applies logits processors and samples the token ids.
- The scheduler folds those tokens back into its own state and builds an engine output.
- The engine output crosses the socket back to the API server.
- Token ids are detokenized into text and streamed to the client.
If any of those eleven sentences feels like a black box, the remaining steps open it. If you also want the file, keep reading — that is the part you cannot get from a diagram.
What vllm serve Actually Spawns
What does vllm serve actually start?
vllm serve is one command that becomes more than one operating-system process. On a default single-GPU launch it becomes two: the process you typed the command into, which runs the HTTP server, and a child process named EngineCore, which owns the GPU and runs the scheduler-and-forward-pass loop.
Add GPUs and that count changes — but only if you tell vLLM to use them, with a flag called --tensor-parallel-size. If that term is new, here it is in one breath: a 70-billion-parameter model does not fit in one GPU's memory, so tensor parallelism cuts every weight matrix in the model into N slices and puts one slice on each of N GPUs. All N work on the same request at the same time and combine their partial results after each layer — like four people each multiplying one quarter of the same big matrix, then adding the quarters up. N is the tensor-parallel size, and each GPU's numbered slot in that group is its rank: rank 0, rank 1, and so on.
One model, sliced across N GPUs
every weight matrix in the model
One GPU holds every matrix whole. Nothing is split, and nothing has to be combined.
real example — meta-llama/Llama-3.1-70B-Instruct, bf16
~70B parameters × 2 bytes = ~140 GB of weights ÷ 1 GPU = 140 GB per GPU
Will not start. 140 GB of weights is past the 72 GB vLLM claims on an 80 GB card — over budget before a single token is cached.
That is one of three ways to spread a model over more than one GPU. vLLM has a flag for each, and the quickest way to tell them apart is to ask what a single GPU ends up holding:
Four GPUs, three ways — what one GPU holds
The same 4-layer model and the same four GPUs each time. Shaded means this GPU has that part. A real 70B model has ~80 layers, so the pipeline case is really 20 layers per GPU, not one — the shape is what matters.
Tensor parallel
--tensor-parallel-size 4one GPU
4 layers, top to bottom
Holds a quarter of every layer
35 GB per GPU
The four must exchange partial results after every layer, so they want a fast interconnect between them.
Pipeline parallel
--pipeline-parallel-size 4one GPU
4 layers, top to bottom
Holds one whole layer, none of the others
35 GB per GPU
Each GPU hands its activations to the next, so traffic is light — but a GPU sits idle while it waits its turn.
Data parallel
--data-parallel-size 4one GPU
4 layers, top to bottom
Holds the entire model
140 GB per GPU
Nothing is shared and nothing is exchanged. You get four times the requests — not room for a bigger model.
The line that matters: tensor and pipeline parallelism both cut the per-GPU weight footprint from 140 GB to 35 GB — they make one model fit. Data parallelism leaves it at 140 GB on every GPU; it only helps once the model already fits.
The first two make one model span several GPUs. The third makes several independent copies. All three default to 1, which is why a plain vllm serve on one GPU is the simple case — and the rest of this module is about what happens as you raise them. Raise the tensor-parallel size past one and every rank gets its own process on top of those two — not every rank past the first, so --tensor-parallel-size 4 is six processes, not five.
Knowing the count matters for a practical reason: a stack trace, a py-spy dump, or an OOM kill only makes sense once you know which process it came from.
What vllm serve starts
One GPU holds the whole model. Raise the size to slice one copy of it across that many GPUs.
the process you launched — FastAPI routes, chat-template rendering, the tokenizer, AsyncLLM, OutputProcessor, detokenization, SSE streaming
EngineCoreProc, the Scheduler, the KV cache manager — and, at this size, the model itself
no worker process. World size 1 selects the uni executor, so the forward pass is a direct function call inside the engine-core process — not an IPC round-trip.
world size 1 → uni executor → no worker processes → 2 processes total
The default launch: two processes
vllm serve meta-llama/Llama-3.1-8B-Instruct
| Process | Log prefix | What runs inside it |
|---|---|---|
| the one you launched | (APIServer pid=…) | FastAPI routes, chat-template rendering, the tokenizer, AsyncLLM, OutputProcessor, detokenization, SSE streaming |
child, titled VLLM::EngineCore | (EngineCore pid=…) | EngineCoreProc, the Scheduler, the KV cache manager, and — at this size — the model itself |
vLLM stamps that bracketed prefix onto every line a process writes, not once per log call — it is re-emitted after every newline, so a multi-line traceback arrives fully attributed, line by line:
# illustrative: the prefix format is exact, the PIDs are not
(EngineCore pid=41402) ERROR … EngineCore encountered a fatal error.
(EngineCore pid=41402) Traceback (most recent call last):
(EngineCore pid=41402) File "…/core.py", in run_engine_core
(EngineCore pid=41402) engine_core.run_busy_loop()
(EngineCore pid=41402) torch.OutOfMemoryError: CUDA out of memory.
Both processes write to the same terminal, so without that per-line prefix only the first line of a traceback would name its process and the rest would be unattributable. Child processes also rename themselves via setproctitle, so ps names the same boundary the logs do — and the PID is the join key between the two views:
$ ps -eo pid,args | grep VLLM::
41402 VLLM::EngineCore ← the PID from the traceback above
Only one line, and the process you launched is missing from it — that is not a bug. run_server calls decorate_logs("APIServer") but never set_process_title, so the launching process prefixes its logs with (APIServer pid=…) while ps still shows the vllm serve … command line you typed. The correspondence is exact for every process vLLM spawns, and absent for the one you spawned yourself.
Two details in that table are easy to misread:
On a single GPU — all three parallelism sizes left at their default of 1 — there is no separate worker process at all. The model runs inside the EngineCore process.
The number that decides this is the world size: how many processes it takes to hold one copy of the model. It is the parallelism sizes multiplied together — tensor-parallel × pipeline-parallel — so leaving them at their defaults gives a world size of 1, and raising the tensor-parallel size to 4 gives a world size of 4. vLLM computes it once at startup and branches on it:
World size picks the executor
Same eleven phases either way. What changes is whether the worker is a function you call or a process you send a message to.
One GPU
TP 1 × PP 1 = world size 1
VLLM::EngineCore — one PID
Executor uni. Nothing is serialized and nothing crosses a process boundary — the worker is an object the engine core already holds.
Four GPUs
TP 4 × PP 1 = world size 4
VLLM::EngineCore — its own PID
one IPC round-trip per forward passserialize → shared-memory queue → deserialize
↔ separately, the four GPUs all-reduce with each other after every layer — a different exchange, and a far more frequent one
Executor mp. Five processes, and every forward pass now pays to cross that boundary — which is also where a hung worker or a backed-up queue becomes possible.
One catch: world size is TP × PP — data-parallel size is not in that product. vLLM still checks it first, so uni requires every axis at 1, data parallelism included. Distributed Execution takes that apart.
Chips 07 and 08 in the right-hand panel are still labelled worker because that is the role, not the process — run the model, sample a token, whoever does it. On one GPU that role is played inside the engine-core process, so the worker and the engine core share a PID.
The API server is a child only when you ask for more than one. With the default --api-server-count 1, the HTTP server runs in the launching process. Pass --api-server-count 4 and you get four child processes named ApiServer_0 through ApiServer_3, all talking to the same engine core.
Those four are the one place where the log name and the ps name genuinely disagree, so grep for the wrong one and you will find nothing:
(ApiServer_0 pid=41233) … # logs: multiprocessing name
41233 VLLM::APIServer_0 # ps: set_process_title()
run_api_server_worker_proc calls set_process_title("APIServer", str(server_index)) and then a bare decorate_logs(), and a bare call takes its name from current_process().name — which was set to ApiServer_{i} when the process was spawned. Two different strings for the same process, differing only in the capitalisation of Api. Everywhere else in vLLM the two names are built from one variable, which is why this is the exception worth remembering rather than a rule.
Tensor parallelism adds one process per rank
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4
World size is now 4, so vLLM defaults to the multiprocessing (mp) executor and spawns one worker process per rank. Six processes total:
APIServer the process you launched: HTTP, rendering, streaming
VLLM::EngineCore scheduler, KV cache manager, executor front-end
├── VLLM::Worker_TP0 rank 0: its shard of the weights, its slice of the KV cache
├── VLLM::Worker_TP1 rank 1
├── VLLM::Worker_TP2 rank 2
└── VLLM::Worker_TP3 rank 3
The workers are children of the engine core, not of the API server. Each rank — 0 through 3 here — holds one slice of the model weights and one slice of the KV cache. The rank suffix in the title is built from whichever parallelism axes are above 1, so a run using two axes at once produces titles like Worker_DP1_TP2: data-parallel replica 1, tensor-parallel rank 2.
This is where the per-line log prefix earns its keep. Four workers now write to the same terminal, and a traceback from one of them stays attributable all the way down:
# illustrative: the prefix format is exact, the PIDs are not
(Worker_TP2 pid=41455) ERROR … WorkerProc hit an exception.
(Worker_TP2 pid=41455) Traceback (most recent call last):
(Worker_TP2 pid=41455) File "…/multiproc_executor.py", in worker_busy_loop
(Worker_TP2 pid=41455) output = func(*args, **kwargs)
(Worker_TP2 pid=41455) torch.OutOfMemoryError: CUDA out of memory.
Every one of those lines came from rank 2 — the third of the four workers — and you can say so without reading a word of the message. Cross-check it against ps on the PID:
$ ps -eo pid,args | grep VLLM::
41402 VLLM::EngineCore
41453 VLLM::Worker_TP0
41454 VLLM::Worker_TP1
41455 VLLM::Worker_TP2 ← the PID from the traceback above
41456 VLLM::Worker_TP3
That one rank ran out of memory and the other three did not is the whole finding: an OOM confined to Worker_TP2 points at something uneven about that shard, not at the model being too big for the machine.
Inside the engine core: one loop, two IO threads
The engine-core process is not a single thread of control. It runs three:
| Thread | Symbol | Job |
|---|---|---|
| input IO (daemon) | EngineCoreProc.process_input_sockets | Poll the inbound socket, msgpack-decode, push onto an in-process input_queue |
| busy loop (main) | EngineCore.step | Drain input_queue, then schedule → execute → sample → update, forever |
| output IO (daemon) | EngineCoreProc.process_output_sockets | Pop from an in-process output_queue, msgpack-encode, push onto the outbound socket |
The reason for the split is written into the source as a comment on the thread creation: the IO threads exist so that ZMQ socket work — which releases Python's global interpreter lock — can overlap with the GPU, and so that serialization and deserialization can overlap with the forward pass. The busy loop never touches a socket; it only touches queues.
One process, three threads
Sockets live at the edges. The busy loop in the middle only ever touches queues.
⇢ inbound socket (ZMQ)
EngineCoreProc.process_input_socketspoll the socket · msgpack-decode
input_queuein-process queueEngineCore.stepschedule → execute → sample → update, forever
never touches a socket
output_queuein-process queueEngineCoreProc.process_output_socketsmsgpack-encode · push to the socket
⇢ outbound socket (ZMQ)
why split it — one forward pass, three threads running at once
ZMQ’s socket calls release Python’s global interpreter lock, so the two IO threads keep working while the busy loop is inside a forward pass — decoding the next request and encoding the last one during the span the GPU is busy, rather than before and after it.
This is the shape you should keep in your head for the rest of the track: sockets talk to queues, and queues talk to the loop. When a request appears to be "stuck in vLLM", the useful first question is which of those three hand-offs it is sitting in.
The annotated command
Here is the same launch with the flags that matter, each one pointing at the part of the engine it configures — and at the module of this track that opens that part up.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--max-num-seqs 256 \
--max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.90 \
--block-size 16 \
--enable-prefix-caching \
--api-server-count 2
| Flag | Phase it changes | Module of this track that owns it |
|---|---|---|
--max-num-seqs, --max-num-batched-tokens | 5 — admission | The Scheduler — Both Halves |
--gpu-memory-utilization, --block-size, --enable-prefix-caching | 6 — KV allocation | KV Cache Manager & Block Pool |
--max-model-len | 1 and 6 — validation, then block budgeting | KV Cache Manager & Block Pool |
--tensor-parallel-size, --data-parallel-size | 7 — process layout and sharding | Distributed Execution |
--structured-outputs-config | 8 — the logits-processor chain | Sampler, Logits Processors & Structured Output |
--compilation-config | startup, then 7 | Compilation & CUDA Graph Capture |
--kv-transfer-config | 6 and 9 — where KV comes from and goes | KV Connectors & Disaggregation |
--api-server-count | 1, 2, 11 — HTTP front-end | this module |
Those modules are still being written, so treat the right-hand column as a table of contents rather than a set of links. What you can do today is the thing the rest of this module teaches: take any flag above, and name the phase and the process it lands in.
In: Rendering and Tokenization
What is rendering in vLLM?
Rendering is the step that turns an OpenAI-shaped JSON body — a list of messages, maybe some tools — into the flat list of integer token ids the model will actually consume. It is two jobs stacked: apply the model's chat template to flatten the conversation into one string, then Tokenization that string.
vLLM gives rendering its own top-level package, vllm/renderers/, because it is where a surprising amount of "why did the model see something different from what I sent?" lives.
Rendering — JSON in, token ids out
Two jobs stacked. Watch what appears between them that you never sent.
what you sent — OpenAI JSON body
{
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "What is paged attention?" }
]
}render_messages_asyncone flat string — Llama-3-style template
<|begin_of_text|><|start_header_id|>system<|end_header_id|>You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>What is paged attention?<|eot_id|><|start_header_id|>assistant<|end_header_id|>6 of these 8 pieces were not in your JSON — the template added them, including the trailing assistant header that tells the model whose turn it is to speak.
tokenize_prompts_asyncinteger token ids — what the model actually consumes
illustrative ids — the real values are specific to the model’s tokenizer
Why this is its own package: your prompt was two sentences, and the model was handed a string with 6 pieces of scaffolding around them. When a token count, a stop sequence, or a tool call behaves strangely, the middle stage is almost always where the surprise was introduced — which is why vllm/renderers/ exists at the top level rather than hiding inside the API layer.
The inbound chain
Four hops get a request from the socket FastAPI is listening on to a tokenized prompt. The first two live in vllm/entrypoints/openai/chat_completion/, the last two in vllm/renderers/:
Four hops — a call stack, not a pipeline
Each hop calls the next and waits inside it. Nothing finishes until the innermost one does, and the tokenized prompt comes back out through all four.
⇢ HTTP request arrives
HTTP
Is this JSON or an SSE stream? Nothing about prompts.
↳ handler.create_chat_completion(...)
request policy
Request id, LoRA adapter, max_tokens, SamplingParams.
↳ self.online_renderer.render_chat(...)
chat rules
Tool-choice rules, Mistral quirks, is a custom template allowed?
↳ renderer.render_chat_async(...)
the actual work
Apply the chat template, then tokenize. Nothing chat-specific left.
⇠ token ids return up through all four
Why four and not one: each frame owns a different question, and the deeper you go the less it knows about HTTP. That is what lets the innermost one be reused — hop 4 has no idea it was reached from a chat endpoint, so completions and other entry points can call it too. Splitting them is what makes the bottom of the stack generic.
| # | Symbol and file | Does |
|---|---|---|
| 1 | create_chat_completionin api_router.py | The FastAPI route. Pulls the handler off app state, calls it, and wraps the result as JSON or as an SSE StreamingResponse |
| 2 | OpenAIServingChat._create_chat_completionin serving.py | Builds the request id, resolves the LoRA adapter, computes max_tokens, builds SamplingParams, and calls the engine |
| 3 | OnlineRenderer.render_chatin online_renderer.py | Chat-specific validation: tool-choice rules, Mistral quirks, whether a request-supplied chat template is even allowed |
| 4 | BaseRenderer.render_chat_asyncin base.py | The actual render-then-tokenize |
Note that the route file is per-endpoint. There is no single monolithic api_server module holding every handler — chat completions live in their own package alongside their own protocol definitions and their own serving class.
Render, tokenize, prepare
# distilled — real names, reduced bodyclass BaseRenderer: async def render_chat_async( self, conversations: Sequence[list[ChatCompletionMessageParam]], chat_params: ChatParams, tok_params: TokenizeParams | None = None, ):1 arrival_time = time.time() if tok_params is None: tok_params = self.default_chat_tok_params 2 rendered = [2 self.render_messages_async(conversation, chat_params)2 for conversation in conversations2 ]2 2 out_conversations = []2 dict_prompts = []2 for conv, prompt in await asyncio.gather(*rendered):2 out_conversations.append(conv)2 dict_prompts.append(prompt) 3 tok_prompts = await self.tokenize_prompts_async(dict_prompts, tok_params)3 3 eng_prompts = await asyncio.gather(3 *(self.process_for_engine_async(p, arrival_time) for p in tok_prompts)3 ) return out_conversations, eng_promptsvllm/renderers/base.py · BaseRenderer.render_chat_async · vLLM v0.26.0 · real file is 1,108 lines · verified 2026-07-26 · open the real fileThree things worth reading off that body:
arrival_time is stamped here, before tokenization. It rides along on the request all the way through, and the time-to-first-token and end-to-end latency vLLM reports for this request are both measured from it — not from the moment the scheduler admitted you.
Queue time is therefore inside the measurement, which is what you want, and also why a saturated server's reported latency climbs even when the forward pass itself did not get slower.
Everything is a list. The signature takes conversations, plural, and this body asyncio.gathers over them into two parallel lists. One HTTP request can legitimately carry several prompts, so the frontend's data structures are batched from the very first hop.
Rendering and tokenization are separate awaits. render_messages_async above produced a dict prompt; tokenize_prompts_async turns it into ids; process_for_engine_async does the last preparation (including multimodal work) before the engine sees it. If your token count is not what you expected, the first of these two lines is the one to instrument.
Registration happens before submission
Three of the four rows below are AsyncLLM methods, so it is worth one sentence on what that is. Despite the name, AsyncLLM does not run a model — it is class AsyncLLM(EngineClient), the engine's client, living in the API-server process. It holds three things: the IPC handle to the engine-core process, the OutputProcessor that will catch tokens coming back, and a background task that drains them. Its whole job is to make a two-process system look like one async for loop to the serving layer above it.
Once there are token ids, the API-server process registers the request with itself before it tells the engine anything. The order is the interesting part:
| # | Symbol | Does |
|---|---|---|
| 1 | AsyncLLM.generate | The entry point the serving layer awaits. Calls add_request, then loops pulling finished outputs and yielding them |
| 2 | AsyncLLM.add_request | Starts the background output handler if it is not running, creates a RequestOutputCollector for this request, and fans out child requests when n > 1 |
| 3 | AsyncLLM._add_request | Two statements: register with the OutputProcessor, then submit to the engine core |
| 4 | OutputProcessor.add_request | Builds the per-request state via RequestState.from_new_request and files it under the request id |
RequestState is where the request's output-side machinery lives: its incremental detokenizer, its logprobs processor, and the queue the streaming response will read from. None of that has anything to do with running the model. It exists purely so that when token ids come back, there is somewhere to put them.
The ordering in AsyncLLM._add_request is a correctness requirement, not a style choice. The engine core is a separate process with its own loop; the instant the submit line completes, that process is free to schedule the request and produce a token. If the OutputProcessor entry did not already exist, the first output to come back would arrive for a request id nobody is tracking — and be silently dropped. Registering the receiver before opening the sender is the same discipline as subscribing before you publish.
Phases 1 and 2 are both behind you now — arrival, rendering and tokenization, then frontend registration. The strip above these panes marks where each step opens, not everything it covers, which is why it sat on phase 1 for both halves. Phase 3, the submit line itself, is the next step — and it is where the request leaves this process for good.
Across the Boundary
What is the process boundary in vLLM?
The process boundary is the pair of ZeroMQ sockets that separate the API-server process from the engine-core process. Requests cross it going in, engine outputs cross it coming back, and everything on it is msgpack-serialized — because the two sides do not share memory.
Everything before this point was ordinary Python objects being passed by reference. From here on, the request is bytes on a socket.
Why split the processes at all?
Both halves want the CPU, and Python will only give it to one of them at a time. Rendering, tokenizing, detokenizing, and formatting SSE chunks are all CPU-bound Python. So is the scheduler's per-step bookkeeping. Put them in one process and the global interpreter lock forces them to take turns — meaning a burst of HTTP traffic can stall the loop that is feeding the GPU.
Splitting them means the engine-core process does one thing: drain a queue, step the engine, push outputs. It never parses JSON and never touches a tokenizer.
One lock per process — the reason for the boundary
The same kinds of work, over the same window of time. What changes is how much of it fits — and how often the GPU gets fed.
One process
one GILEvery Python block excludes every other one, so a burst of HTTP work and the engine's own bookkeeping take turns. The GPU is idle for most of the window — not because it is slow, but because nothing could reach it.
Two processes
one GIL eachTwo interpreters, two locks, two cores. Tokenizing a new request no longer competes with stepping the engine, so the passes come back to back.
The mechanism, in one line: the lock is per process, not per machine. That single fact is why the engine core parses no JSON and touches no tokenizer — the moment it did, that work would be back inside the same lock as the loop feeding the GPU.
The everyday version: this is a kitchen with a pass-through window. Front-of-house takes orders and plates food; the line cook only ever looks at tickets coming through the window. Neither is waiting on the other to finish talking.
The four hops
| # | Symbol | Process | Does |
|---|---|---|---|
| 1 | AsyncMPClient.add_request_async | API server | Stamps client_index on the request so replies can be routed back, then sends it as an ADD message |
| 2 | AsyncMPClient._send_input_message | API server | Prepends the engine identity and puts the msgpack frames on a ROUTER socket with send_multipart |
| 3 | EngineCoreProc.process_input_sockets | engine core (IO thread) | Polls a DEALER socket, decodes the frames, pushes onto input_queue |
| 4 | EngineCoreProc._handle_client_request | engine core (busy loop) | Dispatches on the message type; for ADD, calls EngineCore.add_request |
The inbound socket pair is ROUTER on the client side and DEALER on the engine side, which is ZeroMQ's shape for "one party talking to many identified peers." That is not over-engineering for the two-process case: under data parallelism there are several engine-core processes behind that one client socket, and the identity frame is how a request reaches the right one. The return leg needs no addressing, so it uses a different pattern — PUSH on the engine side into PULL on the client side.
It is worth being precise about what ZeroMQ is here, because "message queue" suggests something it is not. There is no broker: ZeroMQ is a socket library, and the buffering lives inside the sockets at each end — the process list from the previous step has no fourth process that owns a queue. And this is only one of three queue-shaped hand-offs in a running vLLM:
Three boundaries, three transports
Queue-shaped hand-offs happen in three places in a running vLLM. Only the first is ZeroMQ, and the differences between them are what decide whether anything gets serialized.
ROUTER→DEALER· requests in — the identity frame picks which enginePULL←PUSH· outputs back — no addressing needed, they just flow incarries msgpack bytes
Two different socket patterns, not one symmetric channel. And no broker: ZeroMQ is a socket library, so the buffering lives in the sockets themselves — there is no fourth process in the tree that owns a queue.
input IO thread→busy loop· input_queuebusy loop→output IO thread· output_queuecarries plain Python objects
One process, so nothing is serialized and nothing leaves memory — these are ordinary objects handed between threads. This is the queue the busy loop actually touches; it never sees a socket.
engine core→all ranks· rpc_broadcast_mqeach rank→engine core· worker_response_mqcarries serialized, via shared memory
Only exists above world size 1 — at tensor-parallel size 1 there are no worker processes to talk to, so this boundary is absent entirely.
The one that costs you: only the boundaries between processes pay for serialization. The queue in the middle is free, which is precisely why the engine core keeps its busy loop on that side of the line.
The receiving side
# distilled — real names, reduced bodyclass EngineCoreProc(EngineCore): def process_input_sockets( self, input_addresses, coord_input_address, identity, ready_event ): add_request_decoder = MsgpackDecoder(EngineCoreRequest, ...) generic_decoder = MsgpackDecoder(...) with ExitStack() as stack, zmq.Context() as ctx: input_sockets = [ stack.enter_context( make_zmq_socket(ctx, addr, zmq.DEALER, identity=identity, bind=False) ) for addr in input_addresses ] poller = zmq.Poller()1 ready_payload = msgspec.msgpack.encode(EngineCoreReadyResponse(...))1 for input_socket in input_sockets:1 input_socket.send(ready_payload)1 poller.register(input_socket, zmq.POLLIN)1 ready_event.set() while True: for input_socket, _ in poller.poll():2 type_frame, *data_frames = input_socket.recv_multipart(copy=False)2 request_type = EngineCoreRequestType(bytes(type_frame.buffer)) 3 if request_type == EngineCoreRequestType.ADD:3 req = add_request_decoder.decode(data_frames)3 request = self.preprocess_add_request(req)3 else:3 request = generic_decoder.decode(data_frames) self.input_queue.put_nowait((request_type, request))vllm/v1/engine/core.py · EngineCoreProc.process_input_sockets · vLLM v0.26.0 · real file is 2,407 lines · verified 2026-07-26 · open the real fileThis is the thread from the previous step's table, and its whole job is the decode call and the input_queue.put_nowait on the last line. It never schedules anything.
Three details, each marked in the gutter of the lines it is about:
The engine sends first. Before the loop starts, each socket sends a ready_payload and only then does ready_event.set() fire. That handshake is required by the ROUTER/DEALER pairing — the client's ROUTER socket cannot address a peer it has not heard from — and it doubles as the channel that reports the engine's real configuration back to the frontend: the number of GPU blocks it actually allocated, the block size, the dtype, the vLLM version. If startup hangs waiting on ready_event, this is the line it is hanging on.
The message type is its own frame. recv_multipart returns a list of frames, and the first one is split off as type_frame before anything is decoded. That is deliberate: the members of EngineCoreRequestType are single bytes — ADD is b"\x00", ABORT is b"\x01" — so routing a message costs one byte read, not a deserialization.
Only ADD gets the typed decoder. add_request_decoder is built for EngineCoreRequest specifically; aborts, utility calls and data-parallel wave starts ride the same socket with a different first byte and fall to generic_decoder. Two of the six types never arrive on the socket at all — WAKEUP and EXECUTOR_FAILED are pushed onto input_queue from inside this process, which is why the busy loop reads a (type, request) tuple rather than a decoded message: local sentinels and remote requests queue up in the same place.
Where the request finally lands
These last three hops have no excerpt on this page, so each name links straight to its file at the pinned ref. Back on the busy-loop thread, EngineCoreProc._handle_client_request unpacks the ADD tuple and calls EngineCore.add_request, which validates the request id's type, warns if it asked for a KV transfer that no connector can serve, and then hands off with a single line: self.scheduler.add_request(request). Inside Scheduler.add_request the request is appended to the waiting queue and filed in the scheduler's requests dict.
That is the end of phase 4, and it is worth being precise about the state of the world at this moment:
| True now | Not true yet |
|---|---|
| The prompt is tokenized | No KV blocks are allocated |
A Request object exists in the scheduler's waiting queue | Nothing has been sent to a GPU |
The API server has a RequestState ready to receive output | Not one token has been generated |
A request sitting here — decoded, enqueued, and completely idle — is the state behind most "vLLM is slow" reports that turn out not to be about the model at all. It is waiting for the scheduler's next admission decision, and whether it gets one depends on the token budget and on how many requests are already running. That decision is phase 5, and it is the first thing the next step shows you.
The Engine Loop
What is the vLLM engine loop?
The engine loop is the while loop in the engine-core process that runs forever, and on each pass does five things: schedule, allocate, execute, sample, update. For a request that is already generating, one pass normally advances it by one token. Your 500-token answer is on the order of 500 passes of this loop, sharing every pass with whatever else is in flight.
Phases 5 through 9 of the request path are that one pass. This step shows you the loop body and names the file that owns each hop. It does not teach the hops — each one is a module of its own, and cramming them in here would replace understanding with vocabulary.
The loop body
# distilled — real names, reduced bodyclass EngineCore: def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]: if not self.scheduler.has_requests(): return {}, False 1 scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())2 future = self.model_executor.execute_model(scheduler_output, non_block=True)2 grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)2 2 model_output = future.result()3 if model_output is None:3 model_output = self.model_executor.sample_tokens(grammar_output) 4 self._process_aborts_queue()5 engine_core_outputs = self.scheduler.update_from_output(5 scheduler_output, model_output5 ) return engine_core_outputs, scheduler_output.total_num_scheduled_tokens > 0vllm/v1/engine/core.py · EngineCore.step · vLLM v0.26.0 · real file is 2,407 lines · verified 2026-07-26 · open the real fileUnder twenty lines, and the whole serving loop is in them. The engine-core process's busy loop calls this repeatedly — drain the input queue, step, drain, step — and pushes whatever comes back onto the outbound queue. There is no other layer of orchestration hiding underneath.
The gutter marks each hop of the request path, plus one line that is not a hop at all:
Phase 5 — admission. Scheduler.schedule decides what runs on this pass. Phase 6 has no line of its own: KV allocation happens inside this call, because the scheduler cannot decide to run a request without also deciding where its keys and values will live. If allocate_slots cannot find blocks the request is not admitted — and an already-running request may be preempted to free some, which is what Scheduler._preempt_request is for. Admission and allocation are one atomic decision.
Phase 7 — the forward pass, launched but not awaited. non_block=True hands back a future straight away, which is what lets get_grammar_bitmask build the structured-output mask on the CPU while the GPU works. The last line of this mark, future.result(), is where the pass is finally joined.
Phase 8 — sampling, as a second call. For a generative model the forward pass returns None; this guard is what detects that and calls back into the worker to sample. Two calls, not one — the section below is about why that matters.
Not a hop. Aborts that arrived while the GPU was busy are drained here, before the output is processed. The inbound IO thread puts an ABORT onto aborts_queue and input_queue, so this line can apply it eagerly while the queue copy keeps the ordering — safe to do twice, because aborting in the scheduler is idempotent.
Phase 9 — state update. Scheduler.update_from_output folds the sampled tokens back into request state, marks finished requests, and builds the EngineCoreOutputs that will cross the boundary. Same file as phase 5, at the other end of the pass.
Which module owns each hop
| Phase | Symbols it reaches | Module of this track that owns it |
|---|---|---|
| 5 — admission | Scheduler.scheduleScheduler._preempt_request | The Scheduler — Both Halves |
| 6 — KV allocation | KVCacheManager.get_computed_blocksKVCacheManager.allocate_slotsBlockPool.get_new_blocks | KV Cache Manager & Block Pool |
| 7 — forward pass | GPUModelRunner._prepare_inputsGPUModelRunner.execute_model | Executor, Workers & the Model Runner |
| 8 — sampling | GPUModelRunner.sample_tokensSampler.apply_logits_processorsSampler.forward | Sampler, Logits Processors & Structured Output |
| 9 — state update | Scheduler.update_from_output | The Scheduler — Both Halves |
The scheduler owning two of those five rows is the point. It is not a queue — it is a state machine stepped once per forward pass, opened by schedule and closed by update_from_output. That is why the module about it is called "Both Halves."
The part that contradicts the usual narration
Look again at the lines marked 2 and 3:
# vllm/v1/engine/core.py — EngineCore.step
future = self.model_executor.execute_model(scheduler_output, non_block=True)
model_output = future.result()
if model_output is None:
model_output = self.model_executor.sample_tokens(grammar_output)
For a generative model, execute_model runs the forward pass — and returns None. It does not sample. Down in GPUModelRunner.execute_model, the last thing the method does before returning is stash its results, including the Logits, on the runner as ExecuteModelState. That is exactly what the if model_output is None guard is testing: a None means "the forward pass ran, the logits are parked, come back for them." Then sample_tokens — a second, separate call into the worker — unpacks that state, runs the logits processors, samples, and returns the token ids.
So phases 7 and 8 are two calls, not one combined "run the model and get a token" step. The right-hand panel marks this: the connector between chips 07 and 08 is ⟲ 2 calls, not an arrow.
Three qualifications, because the guard is a real branch and not decoration. execute_model returns a value directly — and the second call is skipped — when there is nothing to sample: a pass that scheduled no tokens returns an empty output, an embedding or other pooling model returns its pooled output, and under pipeline parallelism a non-final rank returns hidden states for the next rank instead. Module 4 walks all three. And whether "call into the worker" means an actual inter-process round-trip depends on the executor: with the default single-process executor at tensor-parallel size 1 these are two function calls inside the engine-core process, while with the multiprocessing executor they are two genuine round-trips to the worker processes.
Why split it? Because the gap between the two calls is useful. get_grammar_bitmask is computed on the CPU while the GPU is busy with the forward pass — the non_block=True on execute_model is what makes that overlap possible — and its result is passed into sample_tokens, not into execute_model. Structured output needs to mask the logits, and the mask can be built concurrently with the very forward pass that produces them.
This is the single most common thing people get wrong when describing vLLM from memory, and it is not a detail: it is why a grammar-constrained request does not pay the full cost of its mask, and it is the seam that speculative decoding and async scheduling both hook into. If you take one structural fact from this module, take this one.
What "one step" costs
Nothing in the loop body is per-request. schedule() returns one SchedulerOutput describing the whole batch; execute_model runs one forward pass for all of it; update_from_output walks the batch once. That is why the return value's second element is scheduler_output.total_num_scheduled_tokens > 0 — the loop's unit of work is tokens per step, not requests.
That framing is the bridge back to the LLM Serving track: the token budget you set with --max-num-batched-tokens is a ceiling on that number, and the loop's cost per pass is roughly fixed by the weights it has to read regardless of how full the batch is. Which is exactly why filling the batch is the whole game.
Out: Detokenization and Streaming
How does vLLM stream tokens back to the client?
Token ids leave the engine-core process the same way requests came in — msgpack-encoded on a ZeroMQ socket — and are turned back into text on the API-server side. A background task in the API server pulls engine outputs off that socket in a loop, runs each request's incremental detokenizer, and drops finished text into a per-request queue. Your HTTP handler is a separate task reading that queue and emitting data: lines.
One socket, one loop, one queue per request
Tokens come back batched for the whole server and leave as one stream per client. The split happens in the middle, and it is what the queue is for.
engine-core process
output_queue→PUSH· output IO threadmsgpack on a ZeroMQ socket · once per engine step
API-server process
PULL → _run_output_handler1 task, whole serverone EngineCoreOutputs in, carrying every request that stepped:
req-a1fits own queue
↓generate()
↓data: Paris
req-7c2its own queue
↓generate()
↓data: def
req-e90its own queue
↓generate()
↓data: Sure
3 tasks here — one per in-flight request, each awaiting only its own queue
abort_requests_asyncback across the same boundaryThe only traffic that runs the other way, and it exists for one reason: stop strings are detected here, in text that only exists on this side.
Seven hops back, against the four that carried the request in — and one genuinely surprising back-channel. All of it is below.
The seven hops back
Hop 1 runs in the engine-core process, on its output IO thread. Hops 2 through 7 all run in the API-server process.
| # | Symbol | Does |
|---|---|---|
| 1 | EngineCoreProc.process_output_sockets | Pops from output_queue, msgpack-encodes, sends on a PUSH socket |
| 2 | AsyncMPClient.get_output_async | Awaits the next decoded EngineCoreOutputs, which its own socket task reads off a PULL socket |
| 3 | AsyncLLM._run_output_handler | The background loop that ties 2 to 4, in chunks |
| 4 | OutputProcessor.process_outputs | Per request: stats, detokenize, logprobs, build output, push to that request's queue |
| 5 | BaseIncrementalDetokenizer.update | Appends the new token ids to text and checks stop strings |
| 6 | RequestState.make_request_output | Builds the RequestOutput the caller will see, honouring the stream interval |
| 7 | AsyncLLM.generate, thenOpenAIServingChat.chat_completion_stream_generator | Yields each output to the serving layer, which formats SSE chunks and finally data: [DONE] |
The background loop
# distilled — real names, reduced bodyclass AsyncLLM(EngineClient): def _run_output_handler(self):1 if self.output_handler is not None:1 return engine_core = self.engine_core output_processor = self.output_processor chunk_size = envs.VLLM_V1_OUTPUT_PROC_CHUNK_SIZE async def output_handler(): while True:2 outputs = await engine_core.get_output_async()2 num_outputs = len(outputs.outputs)2 iteration_stats = IterationStats() if num_outputs else None 3 for start in range(0, num_outputs, chunk_size):3 end = start + chunk_size3 processed_outputs = output_processor.process_outputs(3 outputs.outputs[start:end], outputs.timestamp, iteration_stats3 )3 if end < num_outputs:3 await asyncio.sleep(0)4 if processed_outputs.reqs_to_abort:4 await engine_core.abort_requests_async(4 processed_outputs.reqs_to_abort4 ) output_processor.update_scheduler_stats(outputs.scheduler_stats) self.output_handler = asyncio.create_task(output_handler())vllm/v1/engine/async_llm.py · AsyncLLM._run_output_handler · vLLM v0.26.0 · real file is 1,095 lines · verified 2026-07-26 · open the real fileFour things to notice, each marked on the lines it is about:
One loop for the whole server, not one per request. This guard is the reason. AsyncLLM.add_request calls _run_output_handler on every request, and every call after the first returns right here. The one asyncio task is created on the last line of the method and then drains the socket for everything in flight. A request's own generate coroutine never touches that socket — it waits on its private queue, which this loop fills.
One await, one engine step, every request. get_output_async hands back a single EngineCoreOutputs carrying one engine step's outputs — for all the requests that step produced tokens for, which can be hundreds. That is also why iteration_stats is built here, once per pull rather than once per request: this loop's unit of work is a step, the same unit the engine loop counts in.
The chunking is a fairness device. Detokenizing hundreds of requests in one synchronous pass would block the event loop — and the event loop is also the thing serving HTTP. So the batch is sliced, and await asyncio.sleep(0) between slices hands control back to the scheduler so FastAPI's own tasks can run. That sleep(0) is not a delay; it is a deliberate yield.
This is the back-channel. reqs_to_abort arrives from detokenization, and these lines are where the API server tells the engine core to stop working on a request — the one place in the whole path where anything travels that direction. The next section is why it has to exist.
The surprising back-channel: stop strings
The engine core knows about stop token ids. It does not know about stop strings — because a stop string like "\n\nUser:" may not correspond to any single token, and detecting it requires the detokenized text, which only exists on the API-server side.
So the check happens late, and its result has to travel backwards:
BaseIncrementalDetokenizer.updateappends the new token ids tooutput_textand calls the stop-string check against the newly added characters.- If a stop string matched,
OutputProcessor.process_outputstreats the request as finished even though the engine did not say so — and adds its id toreqs_to_abort. - The output handler sees that list and sends an abort back over the boundary, which reaches the scheduler and frees the request's KV blocks.
This is the one place in the request path where the API server tells the engine core to stop doing something, and it exists because detokenization and generation live in different processes. It also means a stop string cannot stop the engine as promptly as a stop token id can: the abort is asynchronous, so the engine may already have sampled further tokens by the time it arrives. A stop token id is checked inside the engine and has no such lag.
Why "incremental" detokenizer
A tokenizer does not decode token-by-token cleanly. A single multi-byte character can be split across tokens, and some tokenizers only produce the correct spacing once they can see the following token. Decoding the whole sequence from scratch on every step would be correct but quadratic.
So BaseIncrementalDetokenizer keeps state — the token ids so far, the text produced so far, an offset — and each update call only decodes the newly arrived ids and appends. The stop-string check then runs against a window of the newly added characters rather than the whole output. Incremental, in this file, means the detokenizer is stateful and per-request, which is exactly why the RequestState created back in phase 2 had to exist before the first token could arrive.
The last hop
RequestState.make_request_output decides whether this output is ready to be handed to the caller — it respects a stream interval, so a client asking for streaming does not necessarily get one SSE chunk per token — and puts the result on the request's queue. AsyncLLM.generate, still sitting in its while not finished loop from phase 2, picks it up and yields it. OpenAIServingChat.chat_completion_stream_generator turns each yielded RequestOutput into data: {...} and, at the end, data: [DONE].
That is the whole path. Eleven phases, two processes, and about a dozen files. Count the socket crossings and the loop shows up again: the request crosses inbound exactly once, but hops 1 and 2 above repeat for every engine step that produced output for you — so a 500-token answer crosses outbound many times, batched together with every other request's tokens. Plus a ready handshake at startup, and an abort travelling backwards whenever a stop string fires.
You can now name the file for every hop
The point of this module was never the eleven phases — you could have got those from a blog post. It was the fourth column of every table: the file. Test yourself before the knowledge check.
| If you want to change… | Open… |
|---|---|
| how the chat template is applied | vllm/renderers/base.py |
| which requests run this step | vllm/v1/core/sched/scheduler.py |
| how KV blocks are handed out | vllm/v1/core/kv_cache_manager.py, vllm/v1/core/block_pool.py |
| what the batch looks like on the GPU | vllm/v1/worker/gpu_model_runner.py |
| how a token is chosen | vllm/v1/sample/sampler.py |
| what crosses the process boundary | vllm/v1/engine/core.py, vllm/v1/engine/core_client.py |
| how text is streamed out | vllm/v1/engine/output_processor.py, vllm/v1/engine/detokenizer.py |