LAVLAV
AI Systems TracksvLLM TracksHandbookAI ExplainedInterview, ExplainedAI Knowledge Map

How a Request Flows Through vLLM — Visual Source Walkthrough

The Whole Path in One Picture

What is the vLLM request path?

The vLLM request path is the eleven-phase journey one HTTP request takes from POST /v1/chat/completions to the last streamed token — crossing two operating-system processes on the way. An API server process renders and tokenizes your prompt, hands it over a socket to a separate engine-core process that owns the GPU, and then reads generated token ids back over a second socket to detokenize and stream them out.

That is the whole story. The rest of this module is coordinates: for each of the eleven phases, which process runs it and which file to open.

Why coordinates and not just concepts?

You can already describe what an inference server does. What you probably cannot do — and what stops most people from ever contributing to vLLM — is answer the question "where would I put a print statement to see that happen?"

So every claim in this track lands on a real file. Every code block is distilled from a pinned release (v0.26.0) and carries a footer with its upstream path, its symbol, and the true size of the file it came from. A 30-line excerpt from a 7,846-line file is a reading aid, not a summary — the footer keeps that honest.

The panel on the right is the map for the whole track. Eleven chips, one per phase, colour-coded by owning process: blue for the API server, green for the engine core, violet for the worker. Click any chip to see its phase title and the real symbol names that run there. The same strip sits above every step in this module as a "you are here" bar.

The eleven phases

#PhaseProcessDirectory to open
1HTTP arrival, chat-template rendering, tokenizationAPI servervllm/entrypoints/openai/chat_completion/, vllm/renderers/
2Frontend request registrationAPI servervllm/v1/engine/
3Submission across the process boundaryAPI servervllm/v1/engine/
4Engine-side receive, decode, enqueueengine corevllm/v1/engine/, vllm/v1/core/sched/
5Admission, token budget, preemptionengine corevllm/v1/core/sched/
6KV block allocation, prefix reuse, evictionengine corevllm/v1/core/
7Batch assembly, forward pass, logitsworkervllm/v1/worker/
8Logits processors, sampling, token idsworkervllm/v1/sample/
9Scheduler state update, engine output builtengine corevllm/v1/core/sched/
10Engine output back across the boundaryengine core → API servervllm/v1/engine/
11Output processing, detokenization, streamingAPI servervllm/v1/engine/, vllm/entrypoints/openai/chat_completion/

Read the Process column and a shape appears: blue, blue, blue → green, green, green → violet, violet → green → grey → blue. Out and back. The two socket crossings are phases 3 and 10, and they are the only two places in the path where data leaves one process's memory for another's.

Three things about this path that surprise people

It is not one pass — the middle is a loop

Phases 1 through 4 happen exactly once per request. Phases 5 through 9 happen once per forward pass, and for a request that is already generating, each pass normally advances it by one token. (Not always one: a long prompt can be chunk-prefilled across several passes before it produces anything, and speculative decoding can land several tokens in one pass.) A 500-token answer means those five phases ran on the order of 500 times for your request, interleaved with every other request in flight.

The scheduler is therefore not deciding "which request runs next." It is deciding, before every single forward pass, which set of requests moves one step. That is continuous batching, and it is the reason a phase-5-to-9 loop exists at all.

CONTINUOUS BATCHINGLLM Internals → batching
Instead of locking a batch for a whole generation, the server re-decides the batch before every forward pass: finished requests leave, waiting requests join, and everyone still generating moves one token. Also called iteration-level scheduling, because the scheduling decision happens once per iteration rather than once per request.

Phases 7 and 8 are two separate calls into the worker, not one

The natural way to narrate a forward pass is "run the model, get a token." The code does not do that: running the model and choosing the token are two separate calls, and the engine does useful work in between.

You can see the seam in the right-hand panel — the connector between chips 07 and 08 is marked ⟲ 2 calls rather than a plain arrow. Step 5 puts the fifteen lines of real code in front of you and explains what the gap buys.

Nothing runs on the GPU until phase 7

Phases 1 through 6 are all CPU work: parsing JSON, applying a Jinja chat template, running a tokenizer, serializing a message, walking a queue, and bookkeeping block indices. On a short prompt that is microseconds; on a long one with images it is not. It is also why vLLM puts phases 1-3 in a different process from phases 4-9 — the topic of the next step.

Say it out loud

You should now be able to narrate the whole path without notes. Try it against the strip above the panes:

  1. HTTP request arrives at the API server; the chat template renders and the prompt is tokenized.
  2. The frontend registers the request's output-side state first, then submits it.
  3. The request is msgpack-encoded and pushed over a socket.
  4. The engine-core process receives it, decodes it, and enqueues it for the scheduler.
  5. The scheduler admits some subset of waiting and running requests under a token budget.
  6. KV blocks are allocated for whatever it admitted, reusing any blocks that already hold the same prefix.
  7. The worker assembles the batch, runs the forward pass, and parks the logits.
  8. A second worker call applies logits processors and samples the token ids.
  9. The scheduler folds those tokens back into its own state and builds an engine output.
  10. The engine output crosses the socket back to the API server.
  11. Token ids are detokenized into text and streamed to the client.

If any of those eleven sentences feels like a black box, the remaining steps open it. If you also want the file, keep reading — that is the part you cannot get from a diagram.

What vllm serve Actually Spawns

What does vllm serve actually start?

vllm serve is one command that becomes more than one operating-system process. On a default single-GPU launch it becomes two: the process you typed the command into, which runs the HTTP server, and a child process named EngineCore, which owns the GPU and runs the scheduler-and-forward-pass loop.

Add GPUs and that count changes — but only if you tell vLLM to use them, with a flag called --tensor-parallel-size. If that term is new, here it is in one breath: a 70-billion-parameter model does not fit in one GPU's memory, so tensor parallelism cuts every weight matrix in the model into N slices and puts one slice on each of N GPUs. All N work on the same request at the same time and combine their partial results after each layer — like four people each multiplying one quarter of the same big matrix, then adding the quarters up. N is the tensor-parallel size, and each GPU's numbered slot in that group is its rank: rank 0, rank 1, and so on.

One model, sliced across N GPUs

--tensor-parallel-size

every weight matrix in the model

rank 01/1

One GPU holds every matrix whole. Nothing is split, and nothing has to be combined.

real example — meta-llama/Llama-3.1-70B-Instruct, bf16

~70B parameters × 2 bytes = ~140 GB of weights ÷ 1 GPU = 140 GB per GPU

072 GB — vLLM’s default budget (0.90)80 GB H100

Will not start. 140 GB of weights is past the 72 GB vLLM claims on an 80 GB card — over budget before a single token is cached.

That is one of three ways to spread a model over more than one GPU. vLLM has a flag for each, and the quickest way to tell them apart is to ask what a single GPU ends up holding:

Four GPUs, three ways — what one GPU holds

The same 4-layer model and the same four GPUs each time. Shaded means this GPU has that part. A real 70B model has ~80 layers, so the pipeline case is really 20 layers per GPU, not one — the shape is what matters.

Tensor parallel

--tensor-parallel-size 4

one GPU

4 layers, top to bottom

Holds a quarter of every layer

35 GB per GPU

The four must exchange partial results after every layer, so they want a fast interconnect between them.

Pipeline parallel

--pipeline-parallel-size 4

one GPU

4 layers, top to bottom

Holds one whole layer, none of the others

35 GB per GPU

Each GPU hands its activations to the next, so traffic is light — but a GPU sits idle while it waits its turn.

Data parallel

--data-parallel-size 4

one GPU

4 layers, top to bottom

Holds the entire model

140 GB per GPU

Nothing is shared and nothing is exchanged. You get four times the requests — not room for a bigger model.

The line that matters: tensor and pipeline parallelism both cut the per-GPU weight footprint from 140 GB to 35 GB — they make one model fit. Data parallelism leaves it at 140 GB on every GPU; it only helps once the model already fits.

The first two make one model span several GPUs. The third makes several independent copies. All three default to 1, which is why a plain vllm serve on one GPU is the simple case — and the rest of this module is about what happens as you raise them. Raise the tensor-parallel size past one and every rank gets its own process on top of those two — not every rank past the first, so --tensor-parallel-size 4 is six processes, not five.

Knowing the count matters for a practical reason: a stack trace, a py-spy dump, or an OOM kill only makes sense once you know which process it came from.

What vllm serve starts

--tensor-parallel-size

One GPU holds the whole model. Raise the size to slice one copy of it across that many GPUs.

$ vllm serve meta-llama/Llama-3.1-8B-Instruct
↓becomes2 OS processes
APIServer(APIServer pid=…)

the process you launched — FastAPI routes, chat-template rendering, the tokenizer, AsyncLLM, OutputProcessor, detokenization, SSE streaming

VLLM::EngineCore(EngineCore pid=…)

EngineCoreProc, the Scheduler, the KV cache manager — and, at this size, the model itself

no worker process. World size 1 selects the uni executor, so the forward pass is a direct function call inside the engine-core process — not an IPC round-trip.

world size 1 → uni executor → no worker processes → 2 processes total

The default launch: two processes

vllm serve meta-llama/Llama-3.1-8B-Instruct
ProcessLog prefixWhat runs inside it
the one you launched(APIServer pid=…)FastAPI routes, chat-template rendering, the tokenizer, AsyncLLM, OutputProcessor, detokenization, SSE streaming
child, titled VLLM::EngineCore(EngineCore pid=…)EngineCoreProc, the Scheduler, the KV cache manager, and — at this size — the model itself

vLLM stamps that bracketed prefix onto every line a process writes, not once per log call — it is re-emitted after every newline, so a multi-line traceback arrives fully attributed, line by line:

# illustrative: the prefix format is exact, the PIDs are not
(EngineCore pid=41402) ERROR … EngineCore encountered a fatal error.
(EngineCore pid=41402) Traceback (most recent call last):
(EngineCore pid=41402)   File "…/core.py", in run_engine_core
(EngineCore pid=41402)     engine_core.run_busy_loop()
(EngineCore pid=41402) torch.OutOfMemoryError: CUDA out of memory.

Both processes write to the same terminal, so without that per-line prefix only the first line of a traceback would name its process and the rest would be unattributable. Child processes also rename themselves via setproctitle, so ps names the same boundary the logs do — and the PID is the join key between the two views:

$ ps -eo pid,args | grep VLLM::
41402 VLLM::EngineCore     ← the PID from the traceback above

Only one line, and the process you launched is missing from it — that is not a bug. run_server calls decorate_logs("APIServer") but never set_process_title, so the launching process prefixes its logs with (APIServer pid=…) while ps still shows the vllm serve … command line you typed. The correspondence is exact for every process vLLM spawns, and absent for the one you spawned yourself.

Two details in that table are easy to misread:

On a single GPU — all three parallelism sizes left at their default of 1 — there is no separate worker process at all. The model runs inside the EngineCore process.

The number that decides this is the world size: how many processes it takes to hold one copy of the model. It is the parallelism sizes multiplied together — tensor-parallel × pipeline-parallel — so leaving them at their defaults gives a world size of 1, and raising the tensor-parallel size to 4 gives a world size of 4. vLLM computes it once at startup and branches on it:

World size picks the executor

Same eleven phases either way. What changes is whether the worker is a function you call or a process you send a message to.

One GPU

TP 1 × PP 1 = world size 1

VLLM::EngineCore — one PID

Scheduler
↓direct function call — same process, nothing sent
Worker — forward pass, sampling

Executor uni. Nothing is serialized and nothing crosses a process boundary — the worker is an object the engine core already holds.

Four GPUs

TP 4 × PP 1 = world size 4

VLLM::EngineCore — its own PID

Scheduler
↓engine core → workers → back:
one IPC round-trip per forward passserialize → shared-memory queue → deserialize
Worker_TP0own PID
Worker_TP1own PID
Worker_TP2own PID
Worker_TP3own PID

↔ separately, the four GPUs all-reduce with each other after every layer — a different exchange, and a far more frequent one

Executor mp. Five processes, and every forward pass now pays to cross that boundary — which is also where a hung worker or a backed-up queue becomes possible.

One catch: world size is TP × PP — data-parallel size is not in that product. vLLM still checks it first, so uni requires every axis at 1, data parallelism included. Distributed Execution takes that apart.

Chips 07 and 08 in the right-hand panel are still labelled worker because that is the role, not the process — run the model, sample a token, whoever does it. On one GPU that role is played inside the engine-core process, so the worker and the engine core share a PID.

The API server is a child only when you ask for more than one. With the default --api-server-count 1, the HTTP server runs in the launching process. Pass --api-server-count 4 and you get four child processes named ApiServer_0 through ApiServer_3, all talking to the same engine core.

Those four are the one place where the log name and the ps name genuinely disagree, so grep for the wrong one and you will find nothing:

(ApiServer_0 pid=41233) …   # logs: multiprocessing name
41233 VLLM::APIServer_0     # ps:   set_process_title()

run_api_server_worker_proc calls set_process_title("APIServer", str(server_index)) and then a bare decorate_logs(), and a bare call takes its name from current_process().name — which was set to ApiServer_{i} when the process was spawned. Two different strings for the same process, differing only in the capitalisation of Api. Everywhere else in vLLM the two names are built from one variable, which is why this is the exception worth remembering rather than a rule.

Tensor parallelism adds one process per rank

vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

World size is now 4, so vLLM defaults to the multiprocessing (mp) executor and spawns one worker process per rank. Six processes total:

APIServer                 the process you launched: HTTP, rendering, streaming
VLLM::EngineCore          scheduler, KV cache manager, executor front-end
 ├── VLLM::Worker_TP0     rank 0: its shard of the weights, its slice of the KV cache
 ├── VLLM::Worker_TP1     rank 1
 ├── VLLM::Worker_TP2     rank 2
 └── VLLM::Worker_TP3     rank 3

The workers are children of the engine core, not of the API server. Each rank — 0 through 3 here — holds one slice of the model weights and one slice of the KV cache. The rank suffix in the title is built from whichever parallelism axes are above 1, so a run using two axes at once produces titles like Worker_DP1_TP2: data-parallel replica 1, tensor-parallel rank 2.

This is where the per-line log prefix earns its keep. Four workers now write to the same terminal, and a traceback from one of them stays attributable all the way down:

# illustrative: the prefix format is exact, the PIDs are not
(Worker_TP2 pid=41455) ERROR … WorkerProc hit an exception.
(Worker_TP2 pid=41455) Traceback (most recent call last):
(Worker_TP2 pid=41455)   File "…/multiproc_executor.py", in worker_busy_loop
(Worker_TP2 pid=41455)     output = func(*args, **kwargs)
(Worker_TP2 pid=41455) torch.OutOfMemoryError: CUDA out of memory.

Every one of those lines came from rank 2 — the third of the four workers — and you can say so without reading a word of the message. Cross-check it against ps on the PID:

$ ps -eo pid,args | grep VLLM::
41402 VLLM::EngineCore
41453 VLLM::Worker_TP0
41454 VLLM::Worker_TP1
41455 VLLM::Worker_TP2     ← the PID from the traceback above
41456 VLLM::Worker_TP3

That one rank ran out of memory and the other three did not is the whole finding: an OOM confined to Worker_TP2 points at something uneven about that shard, not at the model being too big for the machine.

Inside the engine core: one loop, two IO threads

The engine-core process is not a single thread of control. It runs three:

ThreadSymbolJob
input IO (daemon)EngineCoreProc.process_input_socketsPoll the inbound socket, msgpack-decode, push onto an in-process input_queue
busy loop (main)EngineCore.stepDrain input_queue, then schedule → execute → sample → update, forever
output IO (daemon)EngineCoreProc.process_output_socketsPop from an in-process output_queue, msgpack-encode, push onto the outbound socket

The reason for the split is written into the source as a comment on the thread creation: the IO threads exist so that ZMQ socket work — which releases Python's global interpreter lock — can overlap with the GPU, and so that serialization and deserialization can overlap with the forward pass. The busy loop never touches a socket; it only touches queues.

One process, three threads

Sockets live at the edges. The busy loop in the middle only ever touches queues.

⇢ inbound socket (ZMQ)

input IO thread(daemon)
EngineCoreProc.process_input_sockets

poll the socket · msgpack-decode

↓input_queuein-process queue
busy loop(main thread)
EngineCore.step

schedule → execute → sample → update, forever

never touches a socket

↓output_queuein-process queue
output IO thread(daemon)
EngineCoreProc.process_output_sockets

msgpack-encode · push to the socket

⇢ outbound socket (ZMQ)

why split it — one forward pass, three threads running at once

busy loop
schedule
execute + sample on the GPU
update
input IO
decode next
output IO
encode last

ZMQ’s socket calls release Python’s global interpreter lock, so the two IO threads keep working while the busy loop is inside a forward pass — decoding the next request and encoding the last one during the span the GPU is busy, rather than before and after it.

This is the shape you should keep in your head for the rest of the track: sockets talk to queues, and queues talk to the loop. When a request appears to be "stuck in vLLM", the useful first question is which of those three hand-offs it is sitting in.

The annotated command

Here is the same launch with the flags that matter, each one pointing at the part of the engine it configures — and at the module of this track that opens that part up.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --block-size 16 \
  --enable-prefix-caching \
  --api-server-count 2
FlagPhase it changesModule of this track that owns it
--max-num-seqs, --max-num-batched-tokens5 — admissionThe Scheduler — Both Halves
--gpu-memory-utilization, --block-size, --enable-prefix-caching6 — KV allocationKV Cache Manager & Block Pool
--max-model-len1 and 6 — validation, then block budgetingKV Cache Manager & Block Pool
--tensor-parallel-size, --data-parallel-size7 — process layout and shardingDistributed Execution
--structured-outputs-config8 — the logits-processor chainSampler, Logits Processors & Structured Output
--compilation-configstartup, then 7Compilation & CUDA Graph Capture
--kv-transfer-config6 and 9 — where KV comes from and goesKV Connectors & Disaggregation
--api-server-count1, 2, 11 — HTTP front-endthis module

Those modules are still being written, so treat the right-hand column as a table of contents rather than a set of links. What you can do today is the thing the rest of this module teaches: take any flag above, and name the phase and the process it lands in.

In: Rendering and Tokenization

What is rendering in vLLM?

Rendering is the step that turns an OpenAI-shaped JSON body — a list of messages, maybe some tools — into the flat list of integer token ids the model will actually consume. It is two jobs stacked: apply the model's chat template to flatten the conversation into one string, then Tokenization that string.

vLLM gives rendering its own top-level package, vllm/renderers/, because it is where a surprising amount of "why did the model see something different from what I sent?" lives.

TOKENIZATIONLLM Internals → tokenization
Turning a text string into the integer ids the model actually consumes, using a fixed vocabulary of sub-word pieces. The same sentence becomes a different id list under a different tokenizer, which is why the server — not the model — owns this step.

Rendering — JSON in, token ids out

Two jobs stacked. Watch what appears between them that you never sent.

what you sent — OpenAI JSON body

{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user",   "content": "What is paged attention?" }
  ]
}
↓1. apply the model's chat templaterender_messages_async

one flat string — Llama-3-style template

<|begin_of_text|><|start_header_id|>system<|end_header_id|>You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>What is paged attention?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

6 of these 8 pieces were not in your JSON — the template added them, including the trailing assistant header that tells the model whose turn it is to speak.

↓2. tokenize that stringtokenize_prompts_async

integer token ids — what the model actually consumes

1280001280069125128007267552726411190128009128006882128007…

illustrative ids — the real values are specific to the model’s tokenizer

Why this is its own package: your prompt was two sentences, and the model was handed a string with 6 pieces of scaffolding around them. When a token count, a stop sequence, or a tool call behaves strangely, the middle stage is almost always where the surprise was introduced — which is why vllm/renderers/ exists at the top level rather than hiding inside the API layer.

The inbound chain

Four hops get a request from the socket FastAPI is listening on to a tokenized prompt. The first two live in vllm/entrypoints/openai/chat_completion/, the last two in vllm/renderers/:

Four hops — a call stack, not a pipeline

Each hop calls the next and waits inside it. Nothing finishes until the innermost one does, and the tokenized prompt comes back out through all four.

⇢ HTTP request arrives

1create_chat_completionapi_router.py

HTTP

Is this JSON or an SSE stream? Nothing about prompts.

↳ handler.create_chat_completion(...)

2OpenAIServingChat._create_chat_completionserving.py

request policy

Request id, LoRA adapter, max_tokens, SamplingParams.

↳ self.online_renderer.render_chat(...)

3OnlineRenderer.render_chatonline_renderer.py

chat rules

Tool-choice rules, Mistral quirks, is a custom template allowed?

↳ renderer.render_chat_async(...)

4BaseRenderer.render_chat_asyncbase.py

the actual work

Apply the chat template, then tokenize. Nothing chat-specific left.

⇠ token ids return up through all four

Why four and not one: each frame owns a different question, and the deeper you go the less it knows about HTTP. That is what lets the innermost one be reused — hop 4 has no idea it was reached from a chat endpoint, so completions and other entry points can call it too. Splitting them is what makes the bottom of the stack generic.

#Symbol and fileDoes
1create_chat_completion
in api_router.py
The FastAPI route. Pulls the handler off app state, calls it, and wraps the result as JSON or as an SSE StreamingResponse
2OpenAIServingChat._create_chat_completion
in serving.py
Builds the request id, resolves the LoRA adapter, computes max_tokens, builds SamplingParams, and calls the engine
3OnlineRenderer.render_chat
in online_renderer.py
Chat-specific validation: tool-choice rules, Mistral quirks, whether a request-supplied chat template is even allowed
4BaseRenderer.render_chat_async
in base.py
The actual render-then-tokenize

Note that the route file is per-endpoint. There is no single monolithic api_server module holding every handler — chat completions live in their own package alongside their own protocol definitions and their own serving class.

Render, tokenize, prepare

# distilled — real names, reduced bodyclass BaseRenderer:  async def render_chat_async(      self,      conversations: Sequence[list[ChatCompletionMessageParam]],      chat_params: ChatParams,      tok_params: TokenizeParams | None = None,  ):1      arrival_time = time.time()      if tok_params is None:          tok_params = self.default_chat_tok_params 2      rendered = [2          self.render_messages_async(conversation, chat_params)2          for conversation in conversations2      ]2 2      out_conversations = []2      dict_prompts = []2      for conv, prompt in await asyncio.gather(*rendered):2          out_conversations.append(conv)2          dict_prompts.append(prompt) 3      tok_prompts = await self.tokenize_prompts_async(dict_prompts, tok_params)3 3      eng_prompts = await asyncio.gather(3          *(self.process_for_engine_async(p, arrival_time) for p in tok_prompts)3      )      return out_conversations, eng_prompts
Distilled from vllm/renderers/base.py · BaseRenderer.render_chat_async · vLLM v0.26.0 · real file is 1,108 lines · verified 2026-07-26 · open the real file

Three things worth reading off that body:

Note 1

arrival_time is stamped here, before tokenization. It rides along on the request all the way through, and the time-to-first-token and end-to-end latency vLLM reports for this request are both measured from it — not from the moment the scheduler admitted you.

Queue time is therefore inside the measurement, which is what you want, and also why a saturated server's reported latency climbs even when the forward pass itself did not get slower.

Note 2

Everything is a list. The signature takes conversations, plural, and this body asyncio.gathers over them into two parallel lists. One HTTP request can legitimately carry several prompts, so the frontend's data structures are batched from the very first hop.

Note 3

Rendering and tokenization are separate awaits. render_messages_async above produced a dict prompt; tokenize_prompts_async turns it into ids; process_for_engine_async does the last preparation (including multimodal work) before the engine sees it. If your token count is not what you expected, the first of these two lines is the one to instrument.

Registration happens before submission

Three of the four rows below are AsyncLLM methods, so it is worth one sentence on what that is. Despite the name, AsyncLLM does not run a model — it is class AsyncLLM(EngineClient), the engine's client, living in the API-server process. It holds three things: the IPC handle to the engine-core process, the OutputProcessor that will catch tokens coming back, and a background task that drains them. Its whole job is to make a two-process system look like one async for loop to the serving layer above it.

Once there are token ids, the API-server process registers the request with itself before it tells the engine anything. The order is the interesting part:

#SymbolDoes
1AsyncLLM.generateThe entry point the serving layer awaits. Calls add_request, then loops pulling finished outputs and yielding them
2AsyncLLM.add_requestStarts the background output handler if it is not running, creates a RequestOutputCollector for this request, and fans out child requests when n > 1
3AsyncLLM._add_requestTwo statements: register with the OutputProcessor, then submit to the engine core
4OutputProcessor.add_requestBuilds the per-request state via RequestState.from_new_request and files it under the request id

RequestState is where the request's output-side machinery lives: its incremental detokenizer, its logprobs processor, and the queue the streaming response will read from. None of that has anything to do with running the model. It exists purely so that when token ids come back, there is somewhere to put them.

The ordering in AsyncLLM._add_request is a correctness requirement, not a style choice. The engine core is a separate process with its own loop; the instant the submit line completes, that process is free to schedule the request and produce a token. If the OutputProcessor entry did not already exist, the first output to come back would arrive for a request id nobody is tracking — and be silently dropped. Registering the receiver before opening the sender is the same discipline as subscribing before you publish.

Phases 1 and 2 are both behind you now — arrival, rendering and tokenization, then frontend registration. The strip above these panes marks where each step opens, not everything it covers, which is why it sat on phase 1 for both halves. Phase 3, the submit line itself, is the next step — and it is where the request leaves this process for good.

Across the Boundary

What is the process boundary in vLLM?

The process boundary is the pair of ZeroMQ sockets that separate the API-server process from the engine-core process. Requests cross it going in, engine outputs cross it coming back, and everything on it is msgpack-serialized — because the two sides do not share memory.

Everything before this point was ordinary Python objects being passed by reference. From here on, the request is bytes on a socket.

Why split the processes at all?

Both halves want the CPU, and Python will only give it to one of them at a time. Rendering, tokenizing, detokenizing, and formatting SSE chunks are all CPU-bound Python. So is the scheduler's per-step bookkeeping. Put them in one process and the global interpreter lock forces them to take turns — meaning a burst of HTTP traffic can stall the loop that is feeding the GPU.

Splitting them means the engine-core process does one thing: drain a queue, step the engine, push outputs. It never parses JSON and never touches a tokenizer.

One lock per process — the reason for the boundary

The same kinds of work, over the same window of time. What changes is how much of it fits — and how often the GPU gets fed.

One process

one GIL
Pythonone lock
tokenize
step
SSE
tokenize
SSE
step
tokenize
GPU
pass
pass

Every Python block excludes every other one, so a burst of HTTP work and the engine's own bookkeeping take turns. The GPU is idle for most of the window — not because it is slow, but because nothing could reach it.

Two processes

one GIL each
API serverfront of house
tokenize
SSE
tokenize
SSE
tokenize
SSE
engine coreline cook
step
step
step
step
step
GPU
pass
pass
pass
pass
pass

Two interpreters, two locks, two cores. Tokenizing a new request no longer competes with stepping the engine, so the passes come back to back.

The mechanism, in one line: the lock is per process, not per machine. That single fact is why the engine core parses no JSON and touches no tokenizer — the moment it did, that work would be back inside the same lock as the loop feeding the GPU.

The everyday version: this is a kitchen with a pass-through window. Front-of-house takes orders and plates food; the line cook only ever looks at tickets coming through the window. Neither is waiting on the other to finish talking.

The four hops

#SymbolProcessDoes
1AsyncMPClient.add_request_asyncAPI serverStamps client_index on the request so replies can be routed back, then sends it as an ADD message
2AsyncMPClient._send_input_messageAPI serverPrepends the engine identity and puts the msgpack frames on a ROUTER socket with send_multipart
3EngineCoreProc.process_input_socketsengine core (IO thread)Polls a DEALER socket, decodes the frames, pushes onto input_queue
4EngineCoreProc._handle_client_requestengine core (busy loop)Dispatches on the message type; for ADD, calls EngineCore.add_request

The inbound socket pair is ROUTER on the client side and DEALER on the engine side, which is ZeroMQ's shape for "one party talking to many identified peers." That is not over-engineering for the two-process case: under data parallelism there are several engine-core processes behind that one client socket, and the identity frame is how a request reaches the right one. The return leg needs no addressing, so it uses a different pattern — PUSH on the engine side into PULL on the client side.

It is worth being precise about what ZeroMQ is here, because "message queue" suggests something it is not. There is no broker: ZeroMQ is a socket library, and the buffering lives inside the sockets at each end — the process list from the previous step has no fourth process that owns a queue. And this is only one of three queue-shaped hand-offs in a running vLLM:

Three boundaries, three transports

Queue-shaped hand-offs happen in three places in a running vLLM. Only the first is ZeroMQ, and the differences between them are what decide whether anything gets serialized.

API server ⟷ engine coreZeroMQ sockets
ROUTER→DEALER· requests in — the identity frame picks which engine
PULL←PUSH· outputs back — no addressing needed, they just flow in

carries msgpack bytes

Two different socket patterns, not one symmetric channel. And no broker: ZeroMQ is a socket library, so the buffering lives in the sockets themselves — there is no fourth process in the tree that owns a queue.

inside the engine corequeue.Queue
input IO thread→busy loop· input_queue
busy loop→output IO thread· output_queue

carries plain Python objects

One process, so nothing is serialized and nothing leaves memory — these are ordinary objects handed between threads. This is the queue the busy loop actually touches; it never sees a socket.

engine core ⟷ workersMessageQueue (shared memory)
engine core→all ranks· rpc_broadcast_mq
each rank→engine core· worker_response_mq

carries serialized, via shared memory

Only exists above world size 1 — at tensor-parallel size 1 there are no worker processes to talk to, so this boundary is absent entirely.

The one that costs you: only the boundaries between processes pay for serialization. The queue in the middle is free, which is precisely why the engine core keeps its busy loop on that side of the line.

The receiving side

# distilled — real names, reduced bodyclass EngineCoreProc(EngineCore):  def process_input_sockets(      self, input_addresses, coord_input_address, identity, ready_event  ):      add_request_decoder = MsgpackDecoder(EngineCoreRequest, ...)      generic_decoder = MsgpackDecoder(...)       with ExitStack() as stack, zmq.Context() as ctx:          input_sockets = [              stack.enter_context(                  make_zmq_socket(ctx, addr, zmq.DEALER, identity=identity, bind=False)              )              for addr in input_addresses          ]           poller = zmq.Poller()1          ready_payload = msgspec.msgpack.encode(EngineCoreReadyResponse(...))1          for input_socket in input_sockets:1              input_socket.send(ready_payload)1              poller.register(input_socket, zmq.POLLIN)1          ready_event.set()           while True:              for input_socket, _ in poller.poll():2                  type_frame, *data_frames = input_socket.recv_multipart(copy=False)2                  request_type = EngineCoreRequestType(bytes(type_frame.buffer)) 3                  if request_type == EngineCoreRequestType.ADD:3                      req = add_request_decoder.decode(data_frames)3                      request = self.preprocess_add_request(req)3                  else:3                      request = generic_decoder.decode(data_frames)                   self.input_queue.put_nowait((request_type, request))
Distilled from vllm/v1/engine/core.py · EngineCoreProc.process_input_sockets · vLLM v0.26.0 · real file is 2,407 lines · verified 2026-07-26 · open the real file

This is the thread from the previous step's table, and its whole job is the decode call and the input_queue.put_nowait on the last line. It never schedules anything.

Three details, each marked in the gutter of the lines it is about:

Note 1

The engine sends first. Before the loop starts, each socket sends a ready_payload and only then does ready_event.set() fire. That handshake is required by the ROUTER/DEALER pairing — the client's ROUTER socket cannot address a peer it has not heard from — and it doubles as the channel that reports the engine's real configuration back to the frontend: the number of GPU blocks it actually allocated, the block size, the dtype, the vLLM version. If startup hangs waiting on ready_event, this is the line it is hanging on.

Note 2

The message type is its own frame. recv_multipart returns a list of frames, and the first one is split off as type_frame before anything is decoded. That is deliberate: the members of EngineCoreRequestType are single bytes — ADD is b"\x00", ABORT is b"\x01" — so routing a message costs one byte read, not a deserialization.

Note 3

Only ADD gets the typed decoder. add_request_decoder is built for EngineCoreRequest specifically; aborts, utility calls and data-parallel wave starts ride the same socket with a different first byte and fall to generic_decoder. Two of the six types never arrive on the socket at all — WAKEUP and EXECUTOR_FAILED are pushed onto input_queue from inside this process, which is why the busy loop reads a (type, request) tuple rather than a decoded message: local sentinels and remote requests queue up in the same place.

Where the request finally lands

These last three hops have no excerpt on this page, so each name links straight to its file at the pinned ref. Back on the busy-loop thread, EngineCoreProc._handle_client_request unpacks the ADD tuple and calls EngineCore.add_request, which validates the request id's type, warns if it asked for a KV transfer that no connector can serve, and then hands off with a single line: self.scheduler.add_request(request). Inside Scheduler.add_request the request is appended to the waiting queue and filed in the scheduler's requests dict.

That is the end of phase 4, and it is worth being precise about the state of the world at this moment:

True nowNot true yet
The prompt is tokenizedNo KV blocks are allocated
A Request object exists in the scheduler's waiting queueNothing has been sent to a GPU
The API server has a RequestState ready to receive outputNot one token has been generated

A request sitting here — decoded, enqueued, and completely idle — is the state behind most "vLLM is slow" reports that turn out not to be about the model at all. It is waiting for the scheduler's next admission decision, and whether it gets one depends on the token budget and on how many requests are already running. That decision is phase 5, and it is the first thing the next step shows you.

The Engine Loop

What is the vLLM engine loop?

The engine loop is the while loop in the engine-core process that runs forever, and on each pass does five things: schedule, allocate, execute, sample, update. For a request that is already generating, one pass normally advances it by one token. Your 500-token answer is on the order of 500 passes of this loop, sharing every pass with whatever else is in flight.

Phases 5 through 9 of the request path are that one pass. This step shows you the loop body and names the file that owns each hop. It does not teach the hops — each one is a module of its own, and cramming them in here would replace understanding with vocabulary.

The loop body

# distilled — real names, reduced bodyclass EngineCore:  def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:      if not self.scheduler.has_requests():          return {}, False 1      scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())2      future = self.model_executor.execute_model(scheduler_output, non_block=True)2      grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)2 2      model_output = future.result()3      if model_output is None:3          model_output = self.model_executor.sample_tokens(grammar_output) 4      self._process_aborts_queue()5      engine_core_outputs = self.scheduler.update_from_output(5          scheduler_output, model_output5      )      return engine_core_outputs, scheduler_output.total_num_scheduled_tokens > 0
Distilled from vllm/v1/engine/core.py · EngineCore.step · vLLM v0.26.0 · real file is 2,407 lines · verified 2026-07-26 · open the real file

Under twenty lines, and the whole serving loop is in them. The engine-core process's busy loop calls this repeatedly — drain the input queue, step, drain, step — and pushes whatever comes back onto the outbound queue. There is no other layer of orchestration hiding underneath.

The gutter marks each hop of the request path, plus one line that is not a hop at all:

Note 1

Phase 5 — admission. Scheduler.schedule decides what runs on this pass. Phase 6 has no line of its own: KV allocation happens inside this call, because the scheduler cannot decide to run a request without also deciding where its keys and values will live. If allocate_slots cannot find blocks the request is not admitted — and an already-running request may be preempted to free some, which is what Scheduler._preempt_request is for. Admission and allocation are one atomic decision.

Note 2

Phase 7 — the forward pass, launched but not awaited. non_block=True hands back a future straight away, which is what lets get_grammar_bitmask build the structured-output mask on the CPU while the GPU works. The last line of this mark, future.result(), is where the pass is finally joined.

Note 3

Phase 8 — sampling, as a second call. For a generative model the forward pass returns None; this guard is what detects that and calls back into the worker to sample. Two calls, not one — the section below is about why that matters.

Note 4

Not a hop. Aborts that arrived while the GPU was busy are drained here, before the output is processed. The inbound IO thread puts an ABORT onto aborts_queue and input_queue, so this line can apply it eagerly while the queue copy keeps the ordering — safe to do twice, because aborting in the scheduler is idempotent.

Note 5

Phase 9 — state update. Scheduler.update_from_output folds the sampled tokens back into request state, marks finished requests, and builds the EngineCoreOutputs that will cross the boundary. Same file as phase 5, at the other end of the pass.

PAGEDATTENTIONLLM Internals → paged-attention
Borrowing the operating system's virtual-memory trick for the KV cache: each request's keys and values live in fixed-size blocks that need not be contiguous, and a per-request block table maps logical positions to physical blocks. That indirection is what lets two requests share a block and lets a request grow without reserving its worst-case length up front.

Which module owns each hop

PhaseSymbols it reachesModule of this track that owns it
5 — admissionScheduler.schedule
Scheduler._preempt_request
The Scheduler — Both Halves
6 — KV allocationKVCacheManager.get_computed_blocks
KVCacheManager.allocate_slots
BlockPool.get_new_blocks
KV Cache Manager & Block Pool
7 — forward passGPUModelRunner._prepare_inputs
GPUModelRunner.execute_model
Executor, Workers & the Model Runner
8 — samplingGPUModelRunner.sample_tokens
Sampler.apply_logits_processors
Sampler.forward
Sampler, Logits Processors & Structured Output
9 — state updateScheduler.update_from_outputThe Scheduler — Both Halves

The scheduler owning two of those five rows is the point. It is not a queue — it is a state machine stepped once per forward pass, opened by schedule and closed by update_from_output. That is why the module about it is called "Both Halves."

The part that contradicts the usual narration

Look again at the lines marked 2 and 3:

# vllm/v1/engine/core.py — EngineCore.step
future = self.model_executor.execute_model(scheduler_output, non_block=True)
model_output = future.result()
if model_output is None:
    model_output = self.model_executor.sample_tokens(grammar_output)

For a generative model, execute_model runs the forward pass — and returns None. It does not sample. Down in GPUModelRunner.execute_model, the last thing the method does before returning is stash its results, including the Logits, on the runner as ExecuteModelState. That is exactly what the if model_output is None guard is testing: a None means "the forward pass ran, the logits are parked, come back for them." Then sample_tokens — a second, separate call into the worker — unpacks that state, runs the logits processors, samples, and returns the token ids.

So phases 7 and 8 are two calls, not one combined "run the model and get a token" step. The right-hand panel marks this: the connector between chips 07 and 08 is ⟲ 2 calls, not an arrow.

Three qualifications, because the guard is a real branch and not decoration. execute_model returns a value directly — and the second call is skipped — when there is nothing to sample: a pass that scheduled no tokens returns an empty output, an embedding or other pooling model returns its pooled output, and under pipeline parallelism a non-final rank returns hidden states for the next rank instead. Module 4 walks all three. And whether "call into the worker" means an actual inter-process round-trip depends on the executor: with the default single-process executor at tensor-parallel size 1 these are two function calls inside the engine-core process, while with the multiprocessing executor they are two genuine round-trips to the worker processes.

LOGITSLLM Internals → generation
The raw, unnormalised score the model emits for every token in the vocabulary at a given position — one float per vocabulary entry. Sampling is the separate step that turns those scores into one chosen token id, after temperature, penalties, and any other logits processors have reshaped them.

Why split it? Because the gap between the two calls is useful. get_grammar_bitmask is computed on the CPU while the GPU is busy with the forward pass — the non_block=True on execute_model is what makes that overlap possible — and its result is passed into sample_tokens, not into execute_model. Structured output needs to mask the logits, and the mask can be built concurrently with the very forward pass that produces them.

This is the single most common thing people get wrong when describing vLLM from memory, and it is not a detail: it is why a grammar-constrained request does not pay the full cost of its mask, and it is the seam that speculative decoding and async scheduling both hook into. If you take one structural fact from this module, take this one.

What "one step" costs

Nothing in the loop body is per-request. schedule() returns one SchedulerOutput describing the whole batch; execute_model runs one forward pass for all of it; update_from_output walks the batch once. That is why the return value's second element is scheduler_output.total_num_scheduled_tokens > 0 — the loop's unit of work is tokens per step, not requests.

That framing is the bridge back to the LLM Serving track: the token budget you set with --max-num-batched-tokens is a ceiling on that number, and the loop's cost per pass is roughly fixed by the weights it has to read regardless of how full the batch is. Which is exactly why filling the batch is the whole game.

Out: Detokenization and Streaming

How does vLLM stream tokens back to the client?

Token ids leave the engine-core process the same way requests came in — msgpack-encoded on a ZeroMQ socket — and are turned back into text on the API-server side. A background task in the API server pulls engine outputs off that socket in a loop, runs each request's incremental detokenizer, and drops finished text into a per-request queue. Your HTTP handler is a separate task reading that queue and emitting data: lines.

One socket, one loop, one queue per request

Tokens come back batched for the whole server and leave as one stream per client. The split happens in the middle, and it is what the queue is for.

engine-core process

output_queue→PUSH· output IO thread

msgpack on a ZeroMQ socket · once per engine step

API-server process

PULL → _run_output_handler1 task, whole server

one EngineCoreOutputs in, carrying every request that stepped:

req-a1freq-7c2req-e90
req-a1f

its own queue

↓

generate()

↓

data: Paris

req-7c2

its own queue

↓

generate()

↓

data: def

req-e90

its own queue

↓

generate()

↓

data: Sure

3 tasks here — one per in-flight request, each awaiting only its own queue

↑abort_requests_asyncback across the same boundary

The only traffic that runs the other way, and it exists for one reason: stop strings are detected here, in text that only exists on this side.

Seven hops back, against the four that carried the request in — and one genuinely surprising back-channel. All of it is below.

The seven hops back

Hop 1 runs in the engine-core process, on its output IO thread. Hops 2 through 7 all run in the API-server process.

#SymbolDoes
1EngineCoreProc.process_output_socketsPops from output_queue, msgpack-encodes, sends on a PUSH socket
2AsyncMPClient.get_output_asyncAwaits the next decoded EngineCoreOutputs, which its own socket task reads off a PULL socket
3AsyncLLM._run_output_handlerThe background loop that ties 2 to 4, in chunks
4OutputProcessor.process_outputsPer request: stats, detokenize, logprobs, build output, push to that request's queue
5BaseIncrementalDetokenizer.updateAppends the new token ids to text and checks stop strings
6RequestState.make_request_outputBuilds the RequestOutput the caller will see, honouring the stream interval
7AsyncLLM.generate, then
OpenAIServingChat.chat_completion_stream_generator
Yields each output to the serving layer, which formats SSE chunks and finally data: [DONE]

The background loop

# distilled — real names, reduced bodyclass AsyncLLM(EngineClient):  def _run_output_handler(self):1      if self.output_handler is not None:1          return       engine_core = self.engine_core      output_processor = self.output_processor      chunk_size = envs.VLLM_V1_OUTPUT_PROC_CHUNK_SIZE       async def output_handler():          while True:2              outputs = await engine_core.get_output_async()2              num_outputs = len(outputs.outputs)2              iteration_stats = IterationStats() if num_outputs else None 3              for start in range(0, num_outputs, chunk_size):3                  end = start + chunk_size3                  processed_outputs = output_processor.process_outputs(3                      outputs.outputs[start:end], outputs.timestamp, iteration_stats3                  )3                  if end < num_outputs:3                      await asyncio.sleep(0)4                  if processed_outputs.reqs_to_abort:4                      await engine_core.abort_requests_async(4                          processed_outputs.reqs_to_abort4                      )               output_processor.update_scheduler_stats(outputs.scheduler_stats)       self.output_handler = asyncio.create_task(output_handler())
Distilled from vllm/v1/engine/async_llm.py · AsyncLLM._run_output_handler · vLLM v0.26.0 · real file is 1,095 lines · verified 2026-07-26 · open the real file

Four things to notice, each marked on the lines it is about:

Note 1

One loop for the whole server, not one per request. This guard is the reason. AsyncLLM.add_request calls _run_output_handler on every request, and every call after the first returns right here. The one asyncio task is created on the last line of the method and then drains the socket for everything in flight. A request's own generate coroutine never touches that socket — it waits on its private queue, which this loop fills.

Note 2

One await, one engine step, every request. get_output_async hands back a single EngineCoreOutputs carrying one engine step's outputs — for all the requests that step produced tokens for, which can be hundreds. That is also why iteration_stats is built here, once per pull rather than once per request: this loop's unit of work is a step, the same unit the engine loop counts in.

Note 3

The chunking is a fairness device. Detokenizing hundreds of requests in one synchronous pass would block the event loop — and the event loop is also the thing serving HTTP. So the batch is sliced, and await asyncio.sleep(0) between slices hands control back to the scheduler so FastAPI's own tasks can run. That sleep(0) is not a delay; it is a deliberate yield.

Note 4

This is the back-channel. reqs_to_abort arrives from detokenization, and these lines are where the API server tells the engine core to stop working on a request — the one place in the whole path where anything travels that direction. The next section is why it has to exist.

The surprising back-channel: stop strings

The engine core knows about stop token ids. It does not know about stop strings — because a stop string like "\n\nUser:" may not correspond to any single token, and detecting it requires the detokenized text, which only exists on the API-server side.

So the check happens late, and its result has to travel backwards:

  1. BaseIncrementalDetokenizer.update appends the new token ids to output_text and calls the stop-string check against the newly added characters.
  2. If a stop string matched, OutputProcessor.process_outputs treats the request as finished even though the engine did not say so — and adds its id to reqs_to_abort.
  3. The output handler sees that list and sends an abort back over the boundary, which reaches the scheduler and frees the request's KV blocks.

This is the one place in the request path where the API server tells the engine core to stop doing something, and it exists because detokenization and generation live in different processes. It also means a stop string cannot stop the engine as promptly as a stop token id can: the abort is asynchronous, so the engine may already have sampled further tokens by the time it arrives. A stop token id is checked inside the engine and has no such lag.

Why "incremental" detokenizer

A tokenizer does not decode token-by-token cleanly. A single multi-byte character can be split across tokens, and some tokenizers only produce the correct spacing once they can see the following token. Decoding the whole sequence from scratch on every step would be correct but quadratic.

So BaseIncrementalDetokenizer keeps state — the token ids so far, the text produced so far, an offset — and each update call only decodes the newly arrived ids and appends. The stop-string check then runs against a window of the newly added characters rather than the whole output. Incremental, in this file, means the detokenizer is stateful and per-request, which is exactly why the RequestState created back in phase 2 had to exist before the first token could arrive.

The last hop

RequestState.make_request_output decides whether this output is ready to be handed to the caller — it respects a stream interval, so a client asking for streaming does not necessarily get one SSE chunk per token — and puts the result on the request's queue. AsyncLLM.generate, still sitting in its while not finished loop from phase 2, picks it up and yields it. OpenAIServingChat.chat_completion_stream_generator turns each yielded RequestOutput into data: {...} and, at the end, data: [DONE].

That is the whole path. Eleven phases, two processes, and about a dozen files. Count the socket crossings and the loop shows up again: the request crosses inbound exactly once, but hops 1 and 2 above repeat for every engine step that produced output for you — so a 500-token answer crosses outbound many times, batched together with every other request's tokens. Plus a ready handshake at startup, and an abort travelling backwards whenever a stop string fires.

You can now name the file for every hop

The point of this module was never the eleven phases — you could have got those from a blog post. It was the fourth column of every table: the file. Test yourself before the knowledge check.

If you want to change…Open…
how the chat template is appliedvllm/renderers/base.py
which requests run this stepvllm/v1/core/sched/scheduler.py
how KV blocks are handed outvllm/v1/core/kv_cache_manager.py, vllm/v1/core/block_pool.py
what the batch looks like on the GPUvllm/v1/worker/gpu_model_runner.py
how a token is chosenvllm/v1/sample/sampler.py
what crosses the process boundaryvllm/v1/engine/core.py, vllm/v1/engine/core_client.py
how text is streamed outvllm/v1/engine/output_processor.py, vllm/v1/engine/detokenizer.py
LAVLAV

Learn how AI systems actually work — from the GPU up to running an agent fleet.

support@learnaivisually.com
Learn
AI Systems TracksvLLM TracksHandbook
AI Latest
AI ExplainedInterview, ExplainedAI Knowledge MapAtom
© 2026 Learn AI Visually · learnaivisually.comAll tracks free forever