How a Request Flows Through vLLM — Visual Source Walkthrough

The Whole Path in One Picture

What is the vLLM request path?

The vLLM request path is the eleven-phase journey one HTTP request takes from POST /v1/chat/completions to the last streamed token — crossing two operating-system processes on the way. An API server process renders and tokenizes your prompt, hands it over a socket to a separate engine-core process that owns the GPU, and then reads generated token ids back over a second socket to detokenize and stream them out.

That is the whole story. The rest of this module is coordinates: for each of the eleven phases, which process runs it and which file to open.

Why coordinates and not just concepts?

You can already describe what an inference server does. What you probably cannot do — and what stops most people from ever contributing to vLLM — is answer the question "where would I put a print statement to see that happen?"

So every claim in this track lands on a real file. Every code block is distilled from a pinned release (v0.26.0) and carries a footer with its upstream path, its symbol, and the true size of the file it came from. A 30-line excerpt from a 7,846-line file is a reading aid, not a summary — the footer keeps that honest.

The eleven phases

Read the Process column and a shape appears: blue, blue, blue → green, green, green → violet, violet → green → grey → blue. Out and back. The two socket crossings are phases 3 and 10, and they are the only two places in the path where data leaves one process's memory for another's.

Three things about this path that surprise people

It is not one pass — the middle is a loop

Phases 1 through 4 happen exactly once per request. Phases 5 through 9 happen once per forward pass, and for a request that is already generating, each pass normally advances it by one token. (Not always one: a long prompt can be chunk-prefilled across several passes before it produces anything, and speculative decoding can land several tokens in one pass.) A 500-token answer means those five phases ran on the order of 500 times for your request, interleaved with every other request in flight.

The scheduler is therefore not deciding "which request runs next." It is deciding, before every single forward pass, which set of requests moves one step. That is continuous batching, and it is the reason a phase-5-to-9 loop exists at all.

Phases 7 and 8 are two separate calls into the worker, not one

The natural way to narrate a forward pass is "run the model, get a token." The code does not do that: running the model and choosing the token are two separate calls, and the engine does useful work in between.

You can see the seam in the right-hand panel — the connector between chips 07 and 08 is marked ⟲ 2 calls rather than a plain arrow. Step 5 puts the fifteen lines of real code in front of you and explains what the gap buys.

Nothing runs on the GPU until phase 7

Phases 1 through 6 are all CPU work: parsing JSON, applying a Jinja chat template, running a tokenizer, serializing a message, walking a queue, and bookkeeping block indices. On a short prompt that is microseconds; on a long one with images it is not. It is also why vLLM puts phases 1-3 in a different process from phases 4-9 — the topic of the next step.

Say it out loud

You should now be able to narrate the whole path without notes. Try it against the strip above the panes:

1. HTTP request arrives at the API server; the chat template renders and the prompt is tokenized.
2. The frontend registers the request's output-side state first, then submits it.
3. The request is msgpack-encoded and pushed over a socket.
4. The engine-core process receives it, decodes it, and enqueues it for the scheduler.
5. The scheduler admits some subset of waiting and running requests under a token budget.
6. KV blocks are allocated for whatever it admitted, reusing any blocks that already hold the same prefix.
7. The worker assembles the batch, runs the forward pass, and parks the logits.
8. A second worker call applies logits processors and samples the token ids.
9. The scheduler folds those tokens back into its own state and builds an engine output.
10. The engine output crosses the socket back to the API server.
11. Token ids are detokenized into text and streamed to the client.

If any of those eleven sentences feels like a black box, the remaining steps open it. If you also want the file, keep reading — that is the part you cannot get from a diagram.

What vllm serve Actually Spawns

What does vllm serve actually start?

vllm serve is one command that becomes more than one operating-system process. On a default single-GPU launch it becomes two: the process you typed the command into, which runs the HTTP server, and a child process named EngineCore, which owns the GPU and runs the scheduler-and-forward-pass loop. Raise the tensor-parallel size and each extra GPU rank gets its own process on top of those two.

Knowing the count matters for a practical reason: a stack trace, a py-spy dump, or an OOM kill only makes sense once you know which process it came from.

The default launch: two processes

vllm serve meta-llama/Llama-3.1-8B-Instruct

vLLM stamps that bracketed prefix onto every log line, and sets the OS-visible process title (via setproctitle) to VLLM:: plus the same name. So ps and your logs agree, and both tell you which side of the boundary a message came from.

Two details in that table are easy to misread:

At tensor-parallel size 1 — with pipeline- and data-parallel size also at 1 — there is no separate worker process. The model runs inside the EngineCore process. vLLM picks its executor from the world size: world size 1 selects the single-process (uni) executor, so the forward pass is a direct function call, not an IPC round-trip. Chips 07 and 08 in the right-hand panel are still labelled worker because that is the role — but on one GPU the worker and the engine core share a PID.

The API server is a child only when you ask for more than one. With the default --api-server-count 1, the HTTP server runs in the launching process. Pass --api-server-count 4 and you get four child processes named ApiServer_0 through ApiServer_3, all talking to the same engine core.

Tensor parallelism adds one process per rank

vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

World size is now 4, so vLLM defaults to the multiprocessing (mp) executor and spawns one worker process per rank. Six processes total:

APIServer                 the process you launched: HTTP, rendering, streaming
VLLM::EngineCore          scheduler, KV cache manager, executor front-end
 ├── VLLM::Worker_TP0     rank 0: its shard of the weights, its slice of the KV cache
 ├── VLLM::Worker_TP1     rank 1
 ├── VLLM::Worker_TP2     rank 2
 └── VLLM::Worker_TP3     rank 3

The workers are children of the engine core, not of the API server. Each worker process holds one shard of the model weights and one slice of the KV cache. The rank suffix in the title is built from the parallelism axes, so a run using several axes at once produces titles like Worker_DP1_TP2. When an OOM kill lands on VLLM::Worker_TP2 and not on the others, that asymmetry is information.

Inside the engine core: one loop, two IO threads

The engine-core process is not a single thread of control. It runs three:

The reason for the split is written into the source as a comment on the thread creation: the IO threads exist so that ZMQ socket work — which releases Python's global interpreter lock — can overlap with the GPU, and so that serialization and deserialization can overlap with the forward pass. The busy loop never touches a socket; it only touches queues.

The annotated command

Here is the same launch with the flags that matter, each one pointing at the part of the engine it configures — and at the module of this track that opens that part up.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --block-size 16 \
  --enable-prefix-caching \
  --api-server-count 2

Those modules are still being written, so treat the right-hand column as a table of contents rather than a set of links. What you can do today is the thing the rest of this module teaches: take any flag above, and name the phase and the process it lands in.

In: Rendering and Tokenization

What is rendering in vLLM?

Rendering is the step that turns an OpenAI-shaped JSON body — a list of messages, maybe some tools — into the flat list of integer token ids the model will actually consume. It is two jobs stacked: apply the model's chat template to flatten the conversation into one string, then Tokenization that string.

vLLM gives rendering its own top-level package, vllm/renderers/, because it is where a surprising amount of "why did the model see something different from what I sent?" lives.

The inbound chain

Four hops get a request from the socket FastAPI is listening on to a tokenized prompt. The first two live in vllm/entrypoints/openai/chat_completion/, the last two in vllm/renderers/:

Note that the route file is per-endpoint. There is no single monolithic api_server module holding every handler — chat completions live in their own package alongside their own protocol definitions and their own serving class.

Render, tokenize, prepare

# distilled — real names, reduced bodyclass BaseRenderer:  async def render_chat_async(      self,      conversations: Sequence[list[ChatCompletionMessageParam]],      chat_params: ChatParams,      tok_params: TokenizeParams | None = None,  ):      arrival_time = time.time()      if tok_params is None:          tok_params = self.default_chat_tok_params       rendered = [          self.render_messages_async(conversation, chat_params)          for conversation in conversations      ]       out_conversations = []      dict_prompts = []      for conv, prompt in await asyncio.gather(*rendered):          out_conversations.append(conv)          dict_prompts.append(prompt)       tok_prompts = await self.tokenize_prompts_async(dict_prompts, tok_params)       eng_prompts = await asyncio.gather(          *(self.process_for_engine_async(p, arrival_time) for p in tok_prompts)      )      return out_conversations, eng_prompts

Three things worth reading off that body:

arrival_time is stamped here, before tokenization. It rides along on the request all the way through, and the time-to-first-token and end-to-end latency vLLM reports for this request are both measured from it — not from the moment the scheduler admitted you. Queue time is therefore inside the measurement, which is what you want, and also why a saturated server's reported latency climbs even when the forward pass itself did not get slower.

Everything is a list. The signature takes conversations, plural, and the body asyncio.gathers over them. One HTTP request can legitimately carry several prompts, so the frontend's data structures are batched from the very first hop.

Rendering and tokenization are separate awaits. render_messages_async produces a dict prompt; tokenize_prompts_async turns it into ids; process_for_engine_async does the last preparation (including multimodal work) before the engine sees it. If your token count is not what you expected, the second of those three is the one to instrument.

Registration happens before submission

Once there are token ids, the API-server process registers the request with itself before it tells the engine anything. The order is the interesting part:

RequestState is where the request's output-side machinery lives: its incremental detokenizer, its logprobs processor, and the queue the streaming response will read from. None of that has anything to do with running the model. It exists purely so that when token ids come back, there is somewhere to put them.

The strip above these panes is on phase 2 now. Phase 3, the submit line itself, is the next step — and it is where the request leaves this process for good.

Across the Boundary

What is the process boundary in vLLM?

The process boundary is the pair of ZeroMQ sockets that separate the API-server process from the engine-core process. Requests cross it going in, engine outputs cross it coming back, and everything on it is msgpack-serialized — because the two sides do not share memory.

Everything before this point was ordinary Python objects being passed by reference. From here on, the request is bytes on a socket.

Why split the processes at all?

Both halves want the CPU, and Python will only give it to one of them at a time. Rendering, tokenizing, detokenizing, and formatting SSE chunks are all CPU-bound Python. So is the scheduler's per-step bookkeeping. Put them in one process and the global interpreter lock forces them to take turns — meaning a burst of HTTP traffic can stall the loop that is feeding the GPU.

Splitting them means the engine-core process does one thing: drain a queue, step the engine, push outputs. It never parses JSON and never touches a tokenizer.

The everyday version: this is a kitchen with a pass-through window. Front-of-house takes orders and plates food; the line cook only ever looks at tickets coming through the window. Neither is waiting on the other to finish talking.

The four hops

The socket pair is ROUTER on the client side and DEALER on the engine side, which is ZeroMQ's shape for "one party talking to many identified peers." That is not over-engineering for the two-process case: under data parallelism there are several engine-core processes behind that one client socket, and the identity frame is how a request reaches the right one.

The receiving side

# distilled — real names, reduced bodyclass EngineCoreProc(EngineCore):  def process_input_sockets(      self, input_addresses, coord_input_address, identity, ready_event  ):      add_request_decoder = MsgpackDecoder(EngineCoreRequest, ...)      generic_decoder = MsgpackDecoder(...)       with ExitStack() as stack, zmq.Context() as ctx:          input_sockets = [              stack.enter_context(                  make_zmq_socket(ctx, addr, zmq.DEALER, identity=identity, bind=False)              )              for addr in input_addresses          ]           poller = zmq.Poller()          ready_payload = msgspec.msgpack.encode(EngineCoreReadyResponse(...))          for input_socket in input_sockets:              input_socket.send(ready_payload)              poller.register(input_socket, zmq.POLLIN)          ready_event.set()           while True:              for input_socket, _ in poller.poll():                  type_frame, *data_frames = input_socket.recv_multipart(copy=False)                  request_type = EngineCoreRequestType(bytes(type_frame.buffer))                   if request_type == EngineCoreRequestType.ADD:                      req = add_request_decoder.decode(data_frames)                      request = self.preprocess_add_request(req)                  else:                      request = generic_decoder.decode(data_frames)                   self.input_queue.put_nowait((request_type, request))

This is the thread from the previous step's table, and its whole job fits in the last four lines: decode, and put on a queue. It never schedules anything.

Two details:

The engine sends first. Before the loop starts, each socket sends a ready_payload and only then does ready_event.set() fire. That handshake is required by the ROUTER/DEALER pairing — the client's ROUTER socket cannot address a peer it has not heard from — and it doubles as the channel that reports the engine's real configuration back to the frontend: the number of GPU blocks it actually allocated, the block size, the dtype, the vLLM version. If startup hangs waiting on ready_event, this is the line it is hanging on.

The message type is its own frame. recv_multipart splits into a type frame and data frames, and only the ADD type gets the typed EngineCoreRequest decoder. Aborts, utility calls, and wake-ups ride the same socket with a different first byte.

Where the request finally lands

Back on the busy-loop thread, _handle_client_request unpacks the ADD tuple and calls EngineCore.add_request, which validates the request id's type, warns if it asked for a KV transfer that no connector can serve, and then hands off with a single line: self.scheduler.add_request(request). Inside Scheduler.add_request the request is appended to the waiting queue and filed in the scheduler's requests dict.

That is the end of phase 4, and it is worth being precise about the state of the world at this moment:

The Engine Loop

What is the vLLM engine loop?

The engine loop is the while loop in the engine-core process that runs forever, and on each pass does five things: schedule, allocate, execute, sample, update. For a request that is already generating, one pass normally advances it by one token. Your 500-token answer is on the order of 500 passes of this loop, sharing every pass with whatever else is in flight.

Phases 5 through 9 of the request path are that one pass. This step shows you the loop body and names the file that owns each hop. It does not teach the hops — each one is a module of its own, and cramming them in here would replace understanding with vocabulary.

The loop body

# distilled — real names, reduced bodyclass EngineCore:  def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:      if not self.scheduler.has_requests():          return {}, False       scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())      future = self.model_executor.execute_model(scheduler_output, non_block=True)      grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)       model_output = future.result()      if model_output is None:          model_output = self.model_executor.sample_tokens(grammar_output)       self._process_aborts_queue()      engine_core_outputs = self.scheduler.update_from_output(          scheduler_output, model_output      )      return engine_core_outputs, scheduler_output.total_num_scheduled_tokens > 0

Fifteen lines, and the whole serving loop is in them. The engine-core process's busy loop calls this repeatedly — drain the input queue, step, drain, step — and pushes whatever comes back onto the outbound queue. There is no other layer of orchestration hiding underneath.

Reading the five hops off those lines

Map those five rows onto the body above and they are only four lines: phases 5 and 9 are the two self.scheduler.… calls, phases 7 and 8 are the two self.model_executor.… calls. Two structural facts fall out of that arithmetic.

Phase 6 has no line of its own. KV allocation is not a separate step the loop calls; it happens inside schedule(). The scheduler cannot decide to run a request without also deciding where its keys and values will live, so admission and allocation are one atomic decision. If allocate_slots cannot find blocks, the request is not admitted — and an already-running request may be preempted to free some, which is what Scheduler._preempt_request is for.

Phases 5 and 9 are the same file, at opposite ends. Scheduler.schedule opens the pass by deciding what runs; Scheduler.update_from_output closes it by folding the sampled tokens back into request state, marking finished requests, and building the EngineCoreOutputs that will cross the boundary. The scheduler is not a queue — it is a state machine that is stepped once per forward pass. That is why the module about it is called "Both Halves."

The part that contradicts the usual narration

Look again at these four lines:

future = self.model_executor.execute_model(scheduler_output, non_block=True)
model_output = future.result()
if model_output is None:
    model_output = self.model_executor.sample_tokens(grammar_output)

For a generative model, execute_model runs the forward pass — and returns None. It does not sample. Down in GPUModelRunner.execute_model, the last thing the method does before returning is stash its results, including the Logits, on the runner as ExecuteModelState. That is exactly what the if model_output is None guard is testing: a None means "the forward pass ran, the logits are parked, come back for them." Then sample_tokens — a second, separate call into the worker — unpacks that state, runs the logits processors, samples, and returns the token ids.

So phases 7 and 8 are two calls, not one combined "run the model and get a token" step. The right-hand panel marks this: the connector between chips 07 and 08 is ⟲ 2 calls, not an arrow.

Three qualifications, because the guard is a real branch and not decoration. execute_model returns a value directly — and the second call is skipped — when there is nothing to sample: a pass that scheduled no tokens returns an empty output, an embedding or other pooling model returns its pooled output, and under pipeline parallelism a non-final rank returns hidden states for the next rank instead. Module 4 walks all three. And whether "call into the worker" means an actual inter-process round-trip depends on the executor: with the default single-process executor at tensor-parallel size 1 these are two function calls inside the engine-core process, while with the multiprocessing executor they are two genuine round-trips to the worker processes.

Why split it? Because the gap between the two calls is useful. get_grammar_bitmask is computed on the CPU while the GPU is busy with the forward pass — the non_block=True on execute_model is what makes that overlap possible — and its result is passed into sample_tokens, not into execute_model. Structured output needs to mask the logits, and the mask can be built concurrently with the very forward pass that produces them.

What "one step" costs

Nothing in the loop body is per-request. schedule() returns one SchedulerOutput describing the whole batch; execute_model runs one forward pass for all of it; update_from_output walks the batch once. That is why the return value's second element is scheduler_output.total_num_scheduled_tokens > 0 — the loop's unit of work is tokens per step, not requests.

That framing is the bridge back to the LLM Serving track: the token budget you set with --max-num-batched-tokens is a ceiling on that number, and the loop's cost per pass is roughly fixed by the weights it has to read regardless of how full the batch is. Which is exactly why filling the batch is the whole game.

Out: Detokenization and Streaming

How does vLLM stream tokens back to the client?

Token ids leave the engine-core process the same way requests came in — msgpack-encoded on a ZeroMQ socket — and are turned back into text on the API-server side. A background task in the API server pulls engine outputs off that socket in a loop, runs each request's incremental detokenizer, and drops finished text into a per-request queue. Your HTTP handler is a separate task reading that queue and emitting data: lines.

The return leg has one more hop than the inbound leg, and one genuinely surprising back-channel. Both are below.

The seven hops back

Hop 1 runs in the engine-core process, on its output IO thread. Hops 2 through 7 all run in the API-server process.

The background loop

# distilled — real names, reduced bodyclass AsyncLLM(EngineClient):  def _run_output_handler(self):      if self.output_handler is not None:          return       engine_core = self.engine_core      output_processor = self.output_processor      chunk_size = envs.VLLM_V1_OUTPUT_PROC_CHUNK_SIZE       async def output_handler():          while True:              # 1) Pull EngineCoreOutputs from the EngineCore.              outputs = await engine_core.get_output_async()              num_outputs = len(outputs.outputs)              iteration_stats = IterationStats() if num_outputs else None               for start in range(0, num_outputs, chunk_size):                  end = start + chunk_size                  # 2) Process EngineCoreOutputs.                  processed_outputs = output_processor.process_outputs(                      outputs.outputs[start:end], outputs.timestamp, iteration_stats                  )                  # Allow other asyncio tasks to run between chunks.                  if end < num_outputs:                      await asyncio.sleep(0)                  # 3) Abort any reqs that finished due to stop strings.                  if processed_outputs.reqs_to_abort:                      await engine_core.abort_requests_async(                          processed_outputs.reqs_to_abort                      )               output_processor.update_scheduler_stats(outputs.scheduler_stats)       self.output_handler = asyncio.create_task(output_handler())

Three things to notice.

It is one loop for the whole server, not one per request. A single asyncio task drains the socket for every in-flight request. AsyncLLM.add_request starts it lazily on the first request and the if self.output_handler is not None: return guard makes sure there is never a second one. Each request's own generate coroutine never touches the socket — it waits on its private queue, which this loop fills.

The chunking is a fairness device. One EngineCoreOutputs can carry outputs for hundreds of requests. Detokenizing all of them in one synchronous pass would block the event loop — and the event loop is also the thing serving HTTP. So the loop slices the batch and yields with await asyncio.sleep(0) between slices, letting FastAPI's own tasks run. That sleep(0) is not a delay; it is a deliberate hand-back to the scheduler.

Step 3 sends data the other way. More on that next.

The surprising back-channel: stop strings

The engine core knows about stop token ids. It does not know about stop strings — because a stop string like "\n\nUser:" may not correspond to any single token, and detecting it requires the detokenized text, which only exists on the API-server side.

So the check happens late, and its result has to travel backwards:

1. BaseIncrementalDetokenizer.update appends the new token ids to output_text and calls the stop-string check against the newly added characters.
2. If a stop string matched, OutputProcessor.process_outputs treats the request as finished even though the engine did not say so — and adds its id to reqs_to_abort.
3. The output handler sees that list and sends an abort back over the boundary, which reaches the scheduler and frees the request's KV blocks.

Why "incremental" detokenizer

A tokenizer does not decode token-by-token cleanly. A single multi-byte character can be split across tokens, and some tokenizers only produce the correct spacing once they can see the following token. Decoding the whole sequence from scratch on every step would be correct but quadratic.

So BaseIncrementalDetokenizer keeps state — the token ids so far, the text produced so far, an offset — and each update call only decodes the newly arrived ids and appends. The stop-string check then runs against a window of the newly added characters rather than the whole output. Incremental, in this file, means the detokenizer is stateful and per-request, which is exactly why the RequestState created back in phase 2 had to exist before the first token could arrive.

The last hop

RequestState.make_request_output decides whether this output is ready to be handed to the caller — it respects a stream interval, so a client asking for streaming does not necessarily get one SSE chunk per token — and puts the result on the request's queue. AsyncLLM.generate, still sitting in its while not finished loop from phase 2, picks it up and yields it. OpenAIServingChat.chat_completion_stream_generator turns each yielded RequestOutput into data: {...} and, at the end, data: [DONE].

That is the whole path. Eleven phases, two processes, and about a dozen files. Count the socket crossings and the loop shows up again: the request crosses inbound exactly once, but hops 1 and 2 above repeat for every engine step that produced output for you — so a 500-token answer crosses outbound many times, batched together with every other request's tokens. Plus a ready handshake at startup, and an abort travelling backwards whenever a stop string fires.

You can now name the file for every hop

The point of this module was never the eleven phases — you could have got those from a blog post. It was the fourth column of every table: the file. Test yourself before the knowledge check.