vLLM's Model Runner — the Persistent Batch, in Source

Who Calls the GPU

What is vLLM's model runner?

The model runner is the object that owns everything on the GPU side of one forward pass: the batch, the input tensors, the attention metadata, and the call into the model itself. In vLLM it is GPUModelRunner, it lives in vllm/v1/worker/gpu_model_runner.py, and one instance exists per device. Every claim in this module lands somewhere inside it or in the layer that reaches it.

The engine loop reaches it in one line. That line is the subject of this step, because between the line and the runner there are two objects most people never think about — and one of them is the reason the same vLLM code behaves differently on one GPU versus eight.

Four objects, one line of the loop

Layers 1 and 4 are the ones with names you will recognise from a stack trace. Layers 2 and 3 are pass-through: neither one decides anything about what the model computes. That sets up the module's shape — almost everything interesting in this phase happens in layer 4, and layer 2 exists only to answer the question "does this call cross a process boundary or not?"

The line that leaves the engine

# distilled — real names, reduced bodyclass EngineCore:  def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:      if not self.scheduler.has_requests():          return {}, False       scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())      future = self.model_executor.execute_model(scheduler_output, non_block=True)      grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)       model_output = future.result()      if model_output is None:          model_output = self.model_executor.sample_tokens(grammar_output)       self._process_aborts_queue()      engine_core_outputs = self.scheduler.update_from_output(          scheduler_output, model_output      )      return engine_core_outputs, scheduler_output.total_num_scheduled_tokens > 0

You have seen this body before, in module 1, read as the loop. Read it again as a call site and two different things stand out.

Exactly two of these statements leave layer 1, and both go through self.model_executor. Everything else — schedule, get_grammar_bitmask, update_from_output — is the scheduler, in this process, on this thread. So the whole of this module hangs off two expressions, and the rest of the engine loop belongs to modules you have already read.

Nothing in this body names the model runner. There is no self.model_runner anywhere in the engine core, because the runner lives in a different object graph — possibly in a different OS process. What crosses from layer 1 to layer 2 is a SchedulerOutput and a method name; what comes back is a ModelRunnerOutput or a None. That is the entire contract, and it is why swapping one executor for another is a configuration change rather than a code change.

What "the batch" means from here down

The scheduler's job, once per pass, is to decide which requests move. You already know that. The thing that is genuinely surprising below this line is what the runner does with that decision.

The natural assumption — the one every diagram of continuous batching encourages — is that each pass builds a batch out of whatever the scheduler admitted. If a pass has requests R1, R3 and R4 in it, something constructs a batch of three and hands it to the model.

That is not what the code does. GPUModelRunner holds one batch object that outlives the pass, and the first thing it does each step is diff that object against the scheduler's decision: remove what left, add what arrived, leave the rest alone. A request that was in the batch last pass and is in it again this pass is not re-created — it is the same entry, with its token buffer and block table already in place. (Its position in the batch is a different question, and step 2 answers it.)

That single design choice is why this file is hard to read cold. A variable that looks freshly computed is often the same object from the previous call with two entries changed, and you cannot tell which from the line in front of you. The next step is that diff.

What this module covers, and what it does not

Six steps, and the two that matter most are the two that contradict a common description of vLLM: the batch is mutated rather than rebuilt, and the forward pass hands off its results without sampling them.

The Batch That Persists

What is vLLM's persistent batch?

The persistent batch is a single mutable object, held by the model runner as self.input_batch, that survives from one forward pass to the next. It is not rebuilt each step. Each step, GPUModelRunner._update_states diffs it against the scheduler's decision: it removes the entries that should no longer be there, adds the ones that just arrived, updates the ones that changed, and leaves everything else exactly where it was.

The source says so out loud. The comment above the removal loop calls it "the persistent batch optimization", and it warns what happens if the assumption behind the name breaks: the optimization "assumes that consecutive batches contain mostly the same requests. If batches have low request overlap (e.g., alternating between two distinct sets of requests), this optimization becomes very inefficient."

Here is the gap between the concept the previous step recapped and the code in front of you. Continuous batching says the membership of the batch is re-decided every pass, which is true. It is easy to read that as "the batch is re-created every pass", which is false. Membership churns; the container does not.

Why mutating rather than rebuilding is worth the trouble

An entry in this batch is not just a request id. It carries a row in a token-id buffer wide enough for the model's full context, a block table listing the request's physical KV blocks, a computed-token counter, the request's sampling parameters, and a slot in the GPU-side metadata that sampling will read.

Rebuilding that per pass would mean re-populating every running request's row from scratch on every single token — re-writing its token history into the buffer, rebuilding its block table, re-deriving its metadata — thousands of times per request, for data that has not changed. Mutating means a decoding request costs one appended token and, occasionally, one appended block id. (The saving is on the CPU side, in these buffers. What crosses to the GPU each pass is only this pass's scheduled tokens either way — step 3 is that copy.)

The trade is stated in the code's own warning above. Mutation is cheap when consecutive batches overlap. A workload that alternates between two disjoint sets of requests pays the removal and insertion cost every pass and gets nothing back, which is an unusual but real way to make this file slow.

Two removals, and they mean different things

# distilled — real names, reduced body: the two removals onlyclass GPUModelRunner:  def _update_states(self, scheduler_output: "SchedulerOutput") -> Callable | None:      for req_id in scheduler_output.finished_req_ids:          self.requests.pop(req_id, None)      for req_id in scheduler_output.finished_req_ids:          self.input_batch.remove_request(req_id)       scheduled_req_ids = scheduler_output.num_scheduled_tokens.keys()      cached_req_ids = self.input_batch.req_id_to_index.keys()      resumed_req_ids = scheduler_output.scheduled_cached_reqs.resumed_req_ids      unscheduled_req_ids = cached_req_ids - (scheduled_req_ids - resumed_req_ids)      for req_id in unscheduled_req_ids:          self.input_batch.remove_request(req_id)

Two of those loops call the same remove_request, and the difference between them is the whole state model of this file.

The first removal is permanent. A finished request loses its entry in self.input_batch and its entry in self.requests — the runner's dictionary of per-request state. Both are gone; if that id ever comes back it comes back as a brand-new request. Note that self.requests and self.input_batch are two different collections, and only the finished path clears both.

The second removal is temporary, and it keeps the cached state. unscheduled_req_ids is set arithmetic on three key sets: everything the batch currently holds, minus everything the scheduler scheduled this pass, with resumed requests added back into the subtracted set. Whatever is left is a request the batch holds that this pass will not run — a preempted request, or a running request the scheduler chose not to schedule. Those lose their batch entry and keep their self.requests entry, because the source expects them back: they "will be scheduled again sometime in the future."

The - resumed_req_ids term is worth a sentence, because it looks like it does more than it does. A resumed request is normally not in the batch already, so subtracting it changes nothing and the expression reduces to "everything held minus everything scheduled". The source says as much, and names the one case where the term earns its keep: a forced preemption during a prefix-cache reset, where a request is both still in the batch and about to be re-scheduled, and so has to be cleared out before the resumed path re-adds it.

So self.requests is the long-lived record and self.input_batch is the per-pass working set. A request can be absent from the batch for many passes and still be fully known to the runner. That distinction is the first thing to check when a request seems to have vanished.

Adding, then compacting

# distilled — real names, reduced body: same method, continued — the additions      reqs_to_add: list[CachedRequestState] = []      for new_req_data in scheduler_output.scheduled_new_reqs:          req_state = CachedRequestState(req_id=new_req_data.req_id, ...)          self.requests[new_req_data.req_id] = req_state          reqs_to_add.append(req_state)       req_data = scheduler_output.scheduled_cached_reqs      for i, req_id in enumerate(req_data.req_ids):          req_index = self.input_batch.req_id_to_index.get(req_id)          if req_index is None:              reqs_to_add.append(self.requests[req_id])              continue           self.input_batch.num_computed_tokens_cpu[req_index] = (              req_data.num_computed_tokens[i]          )          if req_data.new_block_ids[i] is not None:              self.input_batch.block_table.append_row(                  req_data.new_block_ids[i], req_index              )       for request in reqs_to_add:          self.input_batch.add_request(request)       self.input_batch.condense()      self._may_reorder_batch(scheduler_output)      self.input_batch.refresh_metadata()

Three things in that body are easy to walk past.

The already-running requests get an update, not an insert — and the req_index is None test is what decides. For a request the batch already holds, the loop looks up the position it occupies and writes into that row: the new computed-token count, and an appended block-table row if the scheduler handed out new blocks. That is two assignments, and it is the path a plain decoding request takes every pass. A request the batch does not hold — preempted earlier, or simply not scheduled last pass — has no index to write into, so it is queued into reqs_to_add and inserted below instead. One loop, two outcomes, chosen by whether a key is in a dictionary.

append_row is where KV allocation becomes visible in the worker. The previous module ended with a set of block ids per KV cache group, carried on the scheduler's output. This is where they land: appended onto the request's existing block table, in place, at its existing index. The one exception is a request resumed from preemption, whose block ids are replaced rather than extended — it lost its blocks when it was evicted, so the ids arriving now are its whole allocation, not an addition to it.

condense() is why an index is not an identity. Removals leave gaps in the batch's rows. condense closes those gaps by moving later entries down, which means a request's req_index can change between passes even though the request itself never left. Then _may_reorder_batch lets the attention backend reorder the batch again — some backends want prefill and decode rows grouped — and refresh_metadata pushes the settled arrangement to the GPU.

So the consequence, which burns people reading this code: req_id is stable across passes, req_index is not. Any print statement that keys off an index and expects it to mean the same request next pass will lie to you. The mapping is self.input_batch.req_id_to_index, and it is rebuilt as a side effect of compaction.

Where to put the print statement

If a request is not being computed and you want to know why, the order of these removals tells you where to look.

The batch is now settled for this pass: a fixed set of rows, in a fixed order, with correct counters and block tables. It contains no tensors the model can consume. Building those is a different method, and it is the next step.

Building This Step's Inputs

What does the model runner build before the forward pass?

GPUModelRunner._prepare_inputs turns the settled batch into the flat tensors one kernel launch can consume: a single 1-D array of token ids for the whole pass, the position of each of those tokens in its own request, the physical KV slot each one writes to, and a list of which output rows will be sampled from. It reads self.input_batch and never changes who is in it.

That division of labour is the thing to hold on to. The previous step's method decides membership; this one decides layout. They are two methods for a reason: membership has to be settled and compacted before any index arithmetic can be correct.

The batch is not a rectangle

Here is the shape problem this method exists to solve. The source's own comments walk a three-request pass with [2, 5, 3] scheduled tokens, so this step uses exactly that pass and fills in plausible request state around it:

Three requests, ten tokens, and no two rows the same length. Nothing here is padded into a rectangle — vLLM concatenates the rows end to end into one 10-token sequence and tells the attention kernel where the boundaries are. So this method builds arrays on two different scales: per-token ones with 10 entries in that order, and per-request ones with 3. Keeping track of which is which is most of what makes the body readable.

None of these three is decoding, which is deliberate: it keeps the arithmetic below on one shape. A decoding request needs no special treatment in any of it — it simply contributes exactly one token to the same flat arrays, which is why prefill and decode share this code path rather than having one each.

Flattening the batch into one token axis

# distilled — real names, reduced body: flattening the batch into one token axisclass GPUModelRunner:  def _prepare_inputs(      self,      scheduler_output: "SchedulerOutput",      num_scheduled_tokens: np.ndarray,  ) -> tuple[torch.Tensor, SpecDecodeMetadata | None]:      total_num_scheduled_tokens = scheduler_output.total_num_scheduled_tokens      num_reqs = self.input_batch.num_reqs      self.input_batch.block_table.commit_block_table(num_reqs)       req_indices = np.repeat(self.arange_np[:num_reqs], num_scheduled_tokens)      cu_num_tokens = self._get_cumsum_and_arange(          num_scheduled_tokens, self.query_pos.np      )      positions_np = (          self.input_batch.num_computed_tokens_cpu[req_indices]          + self.query_pos.np[: cu_num_tokens[-1]]      )      token_indices = (          positions_np + req_indices * self.input_batch.token_ids_cpu.shape[1]      )      torch.index_select(          self.input_batch.token_ids_cpu_tensor.flatten(),          0,          torch.from_numpy(token_indices),          out=self.input_ids.cpu[:total_num_scheduled_tokens],      )

Run the table above through those five expressions and the whole method stops being cryptic. num_scheduled_tokens is [2, 5, 3].

Two things follow.

All of this is NumPy on the CPU, and the big buffers are written in place rather than created. self.arange_np, self.query_pos.np, self.input_ids.cpu, self.query_start_loc — every one of those was sized once at startup for the largest batch the engine will ever run, and each pass overwrites a prefix of it. Only the small per-pass index arrays (req_indices, token_indices) are fresh allocations. That is the same trick as the persistent batch, one level down: reuse the container, overwrite the contents, and note that index_select is given an explicit out= so even the gather writes into a buffer that already exists.

token_indices is the only reason the token buffer is 2-D. The batch keeps one fixed-width row per request, so a request's token at position p lives at flat offset req_index * M + p. The single index_select then gathers all ten of this pass's token ids out of that buffer in one call, straight into the pinned CPU staging tensor that will be copied to the GPU. One gather, no Python loop over requests.

The two things this method returns

# distilled — real names, reduced body: same method, continued — the two returns      self.query_start_loc.np[0] = 0      self.query_start_loc.np[1 : num_reqs + 1] = cu_num_tokens      self.query_start_loc.np[num_reqs + 1 :].fill(cu_num_tokens[-1])      self.query_start_loc.copy_to_gpu()      query_start_loc = self.query_start_loc.gpu[: num_reqs + 1]       num_tokens = [self.requests[r].num_tokens for r in self.input_batch.req_ids]      self.discard_request_mask.np[:num_reqs] = (          self.optimistic_seq_lens_cpu[:num_reqs].numpy()          < np.array(num_tokens, dtype=np.int32)      )      self.discard_request_mask.copy_to_gpu(num_reqs)       self.input_batch.block_table.compute_slot_mapping(          num_reqs,          self.query_start_loc.gpu[: num_reqs + 1],          self.positions[:total_num_scheduled_tokens],      )       use_spec_decode = len(scheduler_output.scheduled_spec_decode_tokens) > 0      if not use_spec_decode:          logits_indices = query_start_loc[1:] - 1          spec_decode_metadata = None      else:          spec_decode_metadata = self._calc_spec_decode_metadata(              num_draft_tokens, cu_num_tokens          )          logits_indices = spec_decode_metadata.logits_indices      return logits_indices, spec_decode_metadata

query_start_loc comes out as [0, 2, 7, 10] — a per-request array, the slice boundaries with a leading zero. It is padded out to the end of its buffer with the final value, because some attention kernels require the array to be non-decreasing, and a partially-filled buffer with stale numbers in the tail would not be. optimistic_seq_lens_cpu, which the mask compares against, is another per-request array computed a few lines earlier: each row's computed-token count plus its scheduled tokens — how long each sequence will be once this pass finishes. "Optimistic" because with speculative decoding it assumes every draft token is accepted.

logits_indices = query_start_loc[1:] - 1 is one of the highest-value lines in the file. With [0, 2, 7, 10] it evaluates to [1, 6, 9] — the last token of each request's slice. Those three rows of the model's output are the only ones sampling will ever look at, because the last token of a slice is the only position whose prediction could be about a token that does not exist yet. (Could, not is — for a request still mid-prefill it is not, and the last section of this step is about what happens to those.) Seven of the ten rows the forward pass produces are never read. That is not waste: R2's first four tokens were prompt tokens being prefilled, and their whole purpose was to write keys and values into the KV cache.

compute_slot_mapping is where paging becomes an integer. Each of the ten tokens has an absolute position; each row has the block table the previous step's diff appended to. The slot mapping resolves both into the single physical index in the KV cache where that token's key and value will be written — one integer per token, which is all the attention kernel needs in order to write into non-contiguous blocks.

The batch contains rows that must not be sampled

Look again at R2 and R3 in the table. Both are still mid-prefill after this pass — 5 of 600 tokens and 43 of 60. Their last scheduled token predicts a token the prompt already specifies, so sampling from them is meaningless. R1 is the opposite case: 100 + 2 reaches its full 102, so its last row is a genuine prediction and must be sampled.

The code samples all three anyway. The source is explicit about the reason — because the batch may contain partial requests, "while we should not sample any token from these partial requests, we do so for simplicity", and the sampled tokens from those requests are discarded afterwards. discard_request_mask is that record: a boolean per row, true when the request's computed-plus-scheduled count is still short of its total token count, which is exactly the definition of "still prefilling". For this pass it comes out [False, True, True].

Two consequences. Uniform work is cheaper than a branch — sampling three rows and throwing two away costs less than shaping the kernel around which rows deserve it. And if you are instrumenting sampled tokens and see values for a request that has not finished its prompt, that is not a bug; the mask has not been applied yet.

The tensors now exist and are on the GPU. Nothing has run the model yet. The next step is the method that owns both of the ones you have just read, plus the forward pass itself.

The Forward Pass

What does GPUModelRunner.execute_model actually do?

It does all four of the things this module has covered so far, in one method: it runs the diff that updates the persistent batch, builds this step's input tensors, builds the attention metadata, runs the model — and then, for an ordinary generative model, computes the Logits, stores them on the runner, and returns None.

That last clause is not a typo. The method whose name promises to execute the model returns nothing in the common case, and the results are collected by a separate call. Step 5 is about why. This step is about everything up to that point, because the shape of the method is what makes the ending make sense.

One method, four jobs

# distilled — real names, reduced body: the outer shape of one forward passclass GPUModelRunner:  def execute_model(      self,      scheduler_output: "SchedulerOutput",      intermediate_tensors: IntermediateTensors | None = None,  ) -> ModelRunnerOutput | AsyncModelRunnerOutput | IntermediateTensors | None:      if self.execute_model_state is not None:          raise RuntimeError(              "State error: sample_tokens() must be called "              "after execute_model() returns None."          )       num_scheduled_tokens = scheduler_output.total_num_scheduled_tokens      with self.synchronize_input_prep():          deferred_state_corrections_fn = self._update_states(scheduler_output)          if not num_scheduled_tokens:              return EMPTY_MODEL_RUNNER_OUTPUT           logits_indices, spec_decode_metadata = self._prepare_inputs(              scheduler_output, num_scheduled_tokens_np          )          attn_metadata, spec_decode_common_attn_metadata = (              self._build_attention_metadata(...)          )          input_ids, inputs_embeds, positions, ... = self._preprocess(...)       with set_forward_context(attn_metadata, self.vllm_config, ...):          model_output = self._model_forward(              input_ids=input_ids,              positions=positions,              intermediate_tensors=intermediate_tensors,              inputs_embeds=inputs_embeds,          )

Four structural facts, each one something you would only learn by reading this.

The engine never calls the diff or the input build directly. Both are private methods invoked from here, in this order, and that order is load-bearing: the diff has to settle and compact the batch before any of the index arithmetic in the input build can be correct. From layer 1's point of view there is exactly one call, execute_model, and everything in the previous two steps happened inside it.

The no-work early return happens after the diff, not before it. _update_states runs first, then the if not num_scheduled_tokens test returns. That is deliberate: a pass can legitimately schedule zero tokens while still having finished requests to clean out of the batch, so the bookkeeping runs even when the model does not. (There is a second variant of this return for engines with a KV transfer group configured, which returns a connector-only output instead of the empty one — same idea, different payload.)

synchronize_input_prep() is the price of reusing buffers. Step 3's pre-allocated CPU staging tensors are cheap precisely because they are the same tensors every pass — which means the previous pass's asynchronous copy to the GPU might still be reading them. This context manager waits on a CUDA event before the prep runs and records a new one after, so the two passes cannot overlap on the same memory. Reuse buys speed and creates a hazard; this is the fence.

The attention metadata is not an argument to the model. Look at the _model_forward call: ids, positions, intermediate tensors, embeds. No attention metadata, no slot mapping, no block tables. All of that is installed by set_forward_context and read by the attention layers out of that context while the forward pass runs. This is the single most confusing thing about reading a vLLM model file for the first time — the layers appear to receive nothing about paging, because they do not receive it, they fetch it.

Where the logits are computed, and where they go

# distilled — real names, reduced body: same method, continued — the postprocess      hidden_states = model_output      if not self.broadcast_pp_output:          if not get_pp_group().is_last_rank:              return hidden_states           if self.is_pooling_model:              return self._pool(hidden_states, ...)           sample_hidden_states = hidden_states[logits_indices]          logits = self.model.compute_logits(sample_hidden_states)      else:          sample_hidden_states = hidden_states[logits_indices]          if not get_pp_group().is_last_rank:              get_pp_group().send_tensor_dict(hidden_states.tensors, ...)              logits = None          else:              logits = self.model.compute_logits(sample_hidden_states)           model_output_broadcast_data: dict[str, Any] = {}          if logits is not None:              model_output_broadcast_data["logits"] = logits.contiguous()          broadcasted = get_pp_group().broadcast_tensor_dict(              model_output_broadcast_data, src=len(get_pp_group().ranks) - 1          )          logits = broadcasted["logits"]       self.execute_model_state = ExecuteModelState(          scheduler_output,          logits,          spec_decode_metadata,          spec_decode_common_attn_metadata,          hidden_states,          sample_hidden_states,          ...,      )      return None

Logits are computed here, in execute_model. self.model.compute_logits(sample_hidden_states) is the vocabulary projection — the step that turns one hidden vector per sampled position into one score per vocabulary entry. It is not deferred to sampling. And notice hidden_states[logits_indices]: this is where step 3's [1, 6, 9] is spent. Ten rows of hidden states come out of the model; three are selected; only those three are projected. On a vocabulary of the size modern open models ship with — of order 100,000 entries — that selection is the difference between projecting ten rows into vocabulary space and projecting three.

Then the method parks and returns. ExecuteModelState is a NamedTuple whose own docstring calls it "ephemeral cached state transferred between execute_model() and sample_tokens(), after execute_model() returns None." The logits go in it, along with the scheduler output, the hidden states, and the speculative-decode metadata. Nothing is returned to the caller.

Four things this method can return

The else branch is worth reading once for what it is not. self.broadcast_pp_output is set at construction, and it is true only for the external-launcher backend running more than one pipeline-parallel rank. On that path a non-final rank does not return early: it sends its hidden states onward, receives the last rank's logits back through a broadcast, and carries on to park state like everybody else. So "a non-final pipeline rank returns hidden states" is true of the common path and false of that one — which is exactly why the condition is written as a negation around the whole block rather than folded into the rank test.

One more thing the ordering buys

deferred_state_corrections_fn is assigned by the diff and, in the ordinary path, called at the very end of this method — after the forward pass has been launched. When speculative decoding is on with async scheduling, the diff optimistically assumes every draft token from the previous step was accepted, and queues a correction to be applied once the real count is known. Calling that correction after the launch means the CPU waits for the previous step's result while the GPU is already working on this one.

That is the same pattern as the gap between the two worker calls, and it is the next step's subject.

Two Calls, Not One

Why is sampling a separate call from the forward pass?

Because the engine has useful work to do in between. For an ordinary generative model, GPUModelRunner.execute_model runs the forward pass, computes the logits, parks them on the runner as ExecuteModelState, and returns None — and then EngineCore.step makes a second, separate call into the worker, sample_tokens, which unpacks that state, applies the logits processors, samples, and returns the token ids. The gap between the two calls is where the grammar bitmask for structured output gets built, on the CPU, while the GPU is still busy.

Module 1 introduced this from the engine's side. This step is the same fact read from the worker's side, where you can see what enforces it and what the branches are.

The four lines that say it

future = self.model_executor.execute_model(scheduler_output, non_block=True)
grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)

model_output = future.result()
if model_output is None:
    model_output = self.model_executor.sample_tokens(grammar_output)

Read the order rather than the content. execute_model is launched with non_block=True, so what comes back is a future rather than a result. get_grammar_bitmask then runs on this thread — CPU work, overlapping the GPU's forward pass. Only after that does the engine resolve the future with future.result(), and only if the result is None does it call sample_tokens, passing in the mask it built in the meantime.

How much future.result() actually waits is an executor question, and step 6 answers it: on the multiprocessing path it dequeues a reply from a worker process, while on the single-process path the future is already resolved by the time it exists and the real asynchrony is the GPU's, not the executor's. Either way the CPU work in between is off the critical path.

Collapse the two calls into one and the mask loses its free slot. Structured output has to reshape the logits before sampling, and the logits do not exist until the forward pass finishes — so the one window in which a mask can be built at no cost is during the forward pass, and using that window requires the sampling call to be separate from it.

None is a protocol, not a failure

The thing that makes this readable rather than mysterious is that the runner enforces the protocol itself. execute_model opens with:

if self.execute_model_state is not None:
    raise RuntimeError(
        "State error: sample_tokens() must be called "
        "after execute_model() returns None."
    )

So the runner is a two-state machine. Parked state present means "the logits are waiting, come and get them"; parked state absent means "ready for a new pass." sample_tokens unpacks the tuple and immediately sets self.execute_model_state = None, which is what returns the runner to the ready state. Call execute_model twice in a row and you get that error rather than silently overwritten logits.

That error string is also the most useful thing in this step for debugging. If you see it, something in between the two calls swallowed the None — a wrapper that returned early, a retry that re-entered, an exception path that skipped sample_tokens.

Three cases where the second call never happens

The if model_output is None guard is a real branch, not decoration. execute_model returns a value directly — and the second call is skipped entirely — when there is nothing to sample. These are the same three non-None returns step 4's table listed, read from the caller's side:

The first is the if not num_scheduled_tokens return in the previous step's first code block; the other two are the two early return statements above the compute_logits call in its second. (A fourth exists off this track's path: an encoder-cache-transfer producer returns an empty encoder output before the forward pass. It needs a KV/EC connector configured, so no default engine takes it.)

The pipeline-parallel row carries its qualification for a reason. On the external-launcher path — the one where self.broadcast_pp_output is true, which needs that backend and more than one pipeline rank — a non-final rank does not return early at all. It sends its hidden states onward, receives the final rank's logits back through a broadcast, and continues to park state like every other rank. So the correct statement is that a non-final pipeline rank returns hidden states on the common path, and that the broadcast path receives logits and carries on.

"A call into the worker" is not always a round trip

One more qualification, and it is the reason step 6 exists. Whether these two calls are inter-process round trips depends entirely on which executor the engine was built with.

The two-call structure is the same either way — the None, the parked state, the second call. What changes is the cost of the gap. On uni there is no serialization and no IPC latency to hide, and the concurrency win comes purely from the GPU running asynchronously behind the CPU. On mp each call also pays a message queue crossing, which is a second thing the gap is hiding. Step 6 is those two executors side by side.

What to take from this step

Three sentences, in order of how often they get stated wrongly:

1. For a generative model the forward pass and sampling are two calls into the worker, and execute_model returns None in between rather than returning tokens.
2. The second call is skipped when there is nothing to sample — a pooling model, or a non-final pipeline rank on the common path.
3. Whether either call crosses a process boundary is a property of the executor, not of this code.

The first is the fact people get wrong when describing vLLM from memory. The second is what stops the first from being an over-claim. The third is next.

Where the Executor Varies

What does vLLM's executor actually do?

It answers one question: how does a method call reach the workers? The executor holds no model, owns no batch, and decides nothing about what gets computed. It takes a method name, some arguments, and a policy for who replies — and it either calls the method on a worker object in this process, or serializes the name and sends it to worker processes over a message queue.

Executor.get_class picks which of those you get, once, at startup. After that the engine loop's self.model_executor.execute_model(...) is the same line of code on one GPU and on eight.

This is the one step the right-hand panel does not cover. PersistentBatchViz walks the persistent batch and stops at the forward-pass handoff; the executor layer is the seam above all of that, and it is carried here in code and prose. The Who Calls chip marks where it sits — the boundary EngineCore.step crosses to reach the runner — but the panel deliberately does not model the crossing itself.

The selection happens once, at startup

# distilled — real names, reduced body: imports inlined, env-gated Ray variant elidedclass Executor(ABC):  @staticmethod  def get_class(vllm_config: VllmConfig) -> type["Executor"]:      executor_class: type[Executor]      parallel_config = vllm_config.parallel_config      distributed_executor_backend = parallel_config.distributed_executor_backend       if isinstance(distributed_executor_backend, type):          if not issubclass(distributed_executor_backend, Executor):              raise TypeError("distributed_executor_backend must be a subclass ...")          executor_class = distributed_executor_backend      elif distributed_executor_backend == "ray":          executor_class = RayDistributedExecutor      elif distributed_executor_backend == "mp":          executor_class = MultiprocExecutor      elif distributed_executor_backend == "uni":          executor_class = UniProcExecutor      elif distributed_executor_backend == "external_launcher":          executor_class = ExecutorWithExternalLauncher      elif isinstance(distributed_executor_backend, str):          executor_class = resolve_obj_by_qualname(distributed_executor_backend)      else:          raise ValueError(              f"Unknown distributed executor backend: {distributed_executor_backend}"          )      return executor_class

A plain chain of string comparisons — no registry, no plugin discovery, no subclass scanning. Three things about it are worth noticing.

It runs at construction, not per step. The engine's setup calls Executor.get_class(vllm_config) and passes the resulting class down; the engine core then does self.model_executor = executor_class(vllm_config). Nothing in the loop re-dispatches. Whatever this function returned at startup is what every subsequent pass goes through, which is why an executor question is always a deployment question and never a per-request one.

The default is decided elsewhere. There is no else branch here for "the user did not choose" — the source has a comment saying distributed_executor_backend "must be set in VllmConfig.__post_init__", and that is where the default lands: uni when the world size is 1, mp when there is more than one rank and no other backend has claimed the slot. So on a single GPU with no flags you get UniProcExecutor without ever having typed the word, and the else at the bottom of this function is for a genuinely unrecognised value.

Two of the branches accept your own class. Passing a type that subclasses Executor uses it directly; passing any other string resolves it as a fully-qualified name, and the real body runs the same issubclass check on the result. That is the extension point, and those two checks are the only thing standing between a typo and a much more confusing failure later.

The two you will actually run

The comparison worth making is not between the two executors' execute_model methods. Both are three-line wrappers that forward the string "execute_model" and their arguments straight into collective_rpc, and side by side they would show you nothing. All of the variance lives one level down, in collective_rpc itself.

uni — a function call wearing an RPC's clothes

# distilled — real names, reduced body: async-scheduling branches elidedclass UniProcExecutor(Executor):  def collective_rpc(      self,      method: str | Callable,      timeout: float | None = None,      args: tuple = (),      kwargs: dict | None = None,      non_block: bool = False,      single_value: bool = False,  ) -> Any:      if kwargs is None:          kwargs = {}       if not non_block:          result = run_method(self.driver_worker, method, args, kwargs)          return result if single_value else [result]       try:          result = run_method(self.driver_worker, method, args, kwargs)          future = Future[Any]()          future.set_result(result if single_value else [result])      except Exception as e:          future = Future[Any]()          future.set_exception(e)      return future

run_method(self.driver_worker, method, args, kwargs) is the whole implementation. self.driver_worker is a worker object in this process; method is the string "execute_model"; the helper resolves it against that object and calls it. Two consequences:

non_block=True is a promise, not a mechanism. The engine loop passes it so it can do CPU work while the GPU runs. Here, the work is already done by the time the future exists — the method is called synchronously and the future is handed back pre-resolved. The concurrency the engine gets on this path comes from CUDA's own asynchrony, not from the executor: the forward pass has been queued on the GPU and the Python call returned before it finished.

The try/except is there so both executors fail the same way. A synchronous exception is caught and stuffed into the future, so future.result() in the engine loop raises at the same point it would if a worker process had failed. Uniform failure shape across executors is why the engine loop needs no branch of its own.

mp — the same string, over a message queue

# distilled — real names, reduced body: aggregator branch and deadline handling elidedclass MultiprocExecutor(Executor):  def collective_rpc(      self,      method: str | Callable,      timeout: float | None = None,      args: tuple = (),      kwargs: dict | None = None,      non_block: bool = False,      unique_reply_rank: int | None = None,      kv_output_aggregator: KVOutputAggregator | None = None,  ) -> Any:      assert self.rpc_broadcast_mq is not None      if self.is_failed:          raise RuntimeError("Executor failed.")       output_rank = unique_reply_rank      if isinstance(method, str):          send_method = method      else:          send_method = cloudpickle.dumps(method, protocol=pickle.HIGHEST_PROTOCOL)      self.rpc_broadcast_mq.enqueue((send_method, args, kwargs, output_rank))

Same signature shape, entirely different body. The call becomes a four-tuple — method, args, kwargs, and which rank should reply — enqueued onto self.rpc_broadcast_mq, which every worker process is reading. A string is sent as-is; a callable is cloudpickle-serialized first, which is how vLLM can ship an arbitrary function to every worker without the workers having imported it.

On the other side of that queue, in the same file, a worker-side loop dequeues the tuple and resolves the string against its own worker object:

func = getattr(self.worker, method)
output = func(*args, **kwargs)

Which is the payoff for reading both bodies. The target is identical. uni resolves the string in-process; mp puts the string in a queue and a worker process resolves it with getattr. Neither executor knows or cares that "execute_model" is a forward pass.

And the target itself is layer 3, not layer 4. What "execute_model" resolves to is the worker's own method of that name; that method does a little pipeline-parallel send/receive bookkeeping and then calls execute_model on the GPUModelRunner it holds. So the full trace of one forward pass, on either executor, is four hops: EngineCore.step → the executor's collective_rpc → the worker's method → GPUModelRunner.execute_model. Only the second hop differs between the two, which is the entire content of this step.

Who is allowed to answer

# distilled — real names, reduced body: same method, continued — the reply side      response_mqs: Sequence[MessageQueue] = self.response_mqs      if output_rank is not None:          response_mqs = (response_mqs[output_rank],)       def get_response():          responses = []          for mq in response_mqs:              status, result = mq.dequeue(timeout=dequeue_timeout)              if status != WorkerProc.ResponseStatus.SUCCESS:                  raise RuntimeError(                      f"Worker failed with error '{result}', please check the"                      " stack trace above for the root cause"                  )              responses.append(result)          return responses[0] if output_rank is not None else responses       future = FutureWrapper(          self.futures_queue, get_response=get_response, aggregate=aggregate      )      return future if non_block else future.result()

Every worker runs the method. Only output_rank's reply is read — the three lines at the top narrow a sequence of response queues down to one. The multiprocessing executor's execute_model passes unique_reply_rank=self.output_rank, and the source describes that rank precisely: it "only returns ModelRunnerOutput from TP rank=0 and PP rank=-1 (the first TP worker of the last PP stage)." So the engine sees one ModelRunnerOutput — or one None — rather than a list of them, even though every worker process just did work.

Note also where a worker's failure becomes an exception. It is not at the call site: it is inside get_response, which the future wrapper runs when the future is resolved. On a non-blocking call that is future.result() in the engine loop, several statements later.

That also explains something otherwise puzzling in GPUModelRunner.sample_tokens: it opens with a branch for the case where there is no parked state. Under pipeline parallelism the second call is broadcast to every worker, so ranks that returned hidden states from execute_model — and therefore parked nothing — still receive it, and take that branch.

What actually changes, side by side

Notice what is not in that table: anything about the batch, the tensors, the model, or the two-call structure. All of that is identical, because all of it lives in GPUModelRunner, and the runner has no idea which executor called it.

Where to put the print statement