vLLM's Distributed Execution — Rank Layout, Groups and collective_rpc, in Source

Three Axes, One Point Each

What is a rank in vLLM's distributed execution?

A rank is one participant in a distributed vLLM launch — one slice of the model, one slice of the compute, addressed by a single integer id. Every rank's position comes from three independent dials, set as launch flags, and each one is a count of GPUs or a count of copies — they don't all mean the same kind of "size":

TP and PP are two ways of cutting up one model: every TP rank and every PP rank has to cooperate — exchange activations, exchange hidden states — to produce a single forward pass. DP is different in kind. It doesn't cut anything up; it copies the whole engine N times. Each DP replica runs its own scheduler, admits its own requests, and owns its own KV cache. For a dense model, two requests landing in different DP replicas never interact — Step 3 covers the one real exception, a mixture-of-experts model under DP, where the replicas do coordinate.

Sharding a Checkpoint already walked TP from the weight-loading side — how one checkpoint tensor becomes N slices, one per rank. This module picks up from the process side: given TP, PP and DP as three numbers, how many processes does vLLM actually start, and how are they wired together.

One rank is one coordinate, not three separate numbers

A rank is a point (tp, pp, dp) in a three-dimensional grid — the TP index it holds, the PP stage it's at, and which DP replica it belongs to. vLLM turns that triple into a single global rank id with one formula, DP the slowest-varying axis and TP the fastest:

rank = dp × (pp_size × tp_size) + pp × tp_size + tp

Turn the tp/pp/dp dials in the panel on the right (stage Grid) and watch this play out. At tp=2, pp=2, dp=1 there are four ranks:

Set dp=2 instead of dp=1 and the grid doesn't grow those four ranks into something bigger — it duplicates all four, once per DP replica, because DP is a copy of the whole engine, not a further slice of it. Ranks 0-3 belong to replica 0, ranks 4-7 to an entirely separate replica 1, each running its own scheduler.

The arithmetic this track has to get exactly right

vLLM's own configuration code answers "how many GPU processes does this launch need?" with a specific field, ParallelConfig.world_size, computed once at startup:

# vllm/config/parallel.py — ParallelConfig
world_size: int = Field(init=False)
"""world_size is TPxPP, it affects the number of workers we create."""

# __post_init__
self.world_size = (
    self.pipeline_parallel_size
    * self.tensor_parallel_size
    * self.prefill_context_parallel_size
)

Prefill-context parallelism (PCP) is a real fourth axis, fixed at 1 throughout this module so it doesn't complicate the picture — with PCP=1, world_size = TP × PP. Notice what's missing from that product: data_parallel_size never appears. world_size is a per-DP-replica count, not a total.

The total process count across every DP replica has its own name, and it's a different field:

# vllm/config/parallel.py — ParallelConfig
@property
def world_size_across_dp(self) -> int:
    """Process world size across TP, PCP, PP, and DP."""
    return self.world_size * self.data_parallel_size

Walk the same four-rank example through both fields:

That last row is worth sitting with. world_size reports 1 — by the letter of vLLM's own field, this is a single-rank configuration. world_size_across_dp reports 2. Whether that difference matters — whether it changes how many operating-system processes this launch spawns — is exactly what the next step answers, and the answer is not what a TP-only reading of "world size 1" would predict.

Say it out loud

Before moving on, you should be able to state, without looking back: what TP splits, what PP splits, what DP copies: the rank-id formula, and why world_size and world_size_across_dp are two different numbers with two different jobs. Every remaining step in this module builds on exactly this grid.

How Do TP, PP and DP Become Processes?

How do TP, PP and DP become operating-system processes?

What vllm serve Actually Spawns already showed you the shape at TP=1: one API server process, one child EngineCore process, and — because world size was 1 — no separate worker at all. Raise TP and that module showed you a worker process per rank, children of the engine core. What it didn't yet have the vocabulary for is DP, and DP changes the picture at a level above the worker: it changes how many engine cores you get, not just how many workers.

One EngineCore process per DP rank

Each DP replica is a complete, independent engine — its own scheduler, its own KV cache — so each one gets its own EngineCore process. The process-naming code makes this literal:

# vllm/v1/engine/core.py — EngineCoreProc.run_engine_core
data_parallel = parallel_config.data_parallel_size > 1 or dp_rank > 0
if data_parallel:
    parallel_config.data_parallel_rank_local = local_dp_rank
    process_title = f"EngineCore_DP{dp_rank}"
else:
    process_title = "EngineCore"
set_process_title(process_title)

At --data-parallel-size 1, one EngineCore. At --data-parallel-size 4, four sibling EngineCore_DP0 .. EngineCore_DP3 processes, launched by run_engine_core once per dp_rank — each with its own copy of everything downstream: its own executor, its own set of workers. Under the internal or hybrid load-balancing modes vLLM also starts one more process, a DP coordinator, to publish per-replica queue stats so requests can be routed to the least-loaded replica; it isn't part of any single replica's own topology, and this module's simulation flags it as present whenever DP>1 under that default routing setup.

Inside one EngineCore: which executor, and how many workers

Which executor class every EngineCore process gets is decided once, but not by the EngineCore process itself. Executor.get_class runs once in the launching process — inside AsyncLLM.from_vllm_config/from_engine_args, before any EngineCore process exists — and the resulting class is handed down as a constructor argument to every DP replica's EngineCore, which then simply does self.model_executor = executor_class(vllm_config). EngineCore.__init__ never calls get_class itself; it receives the answer, the same answer, for every replica in the launch.

# distilled — real names, reduced body: the custom-class passthrough, the# Ray variant, and custom-qualname resolution all elidedclass Executor(ABC):  @staticmethod  def get_class(vllm_config: VllmConfig) -> type["Executor"]:      executor_class: type[Executor]      parallel_config = vllm_config.parallel_config      distributed_executor_backend = parallel_config.distributed_executor_backend      # distributed_executor_backend must be set in VllmConfig.__post_init__      if distributed_executor_backend == "mp":          from vllm.v1.executor.multiproc_executor import MultiprocExecutor          executor_class = MultiprocExecutor      elif distributed_executor_backend == "uni":          from vllm.v1.executor.uniproc_executor import UniProcExecutor          executor_class = UniProcExecutor      elif distributed_executor_backend == "external_launcher":          executor_class = ExecutorWithExternalLauncher      else:          raise ValueError(              f"Unknown distributed executor backend: {distributed_executor_backend}"          )      return executor_class

Mostly a plain string comparison — distributed_executor_backend's real type is str | DistributedExecutorBackend | type[Executor] | None, so the excerpt above elides a branch (isinstance(distributed_executor_backend, type)) that accepts your own Executor subclass directly, the documented extension point the Model Runner module already named. But for the two values this module cares about, "mp" and "uni", it's the string branches, called once — in the launching process, as the previous paragraph traced — and never re-dispatched per step. By the time get_class reads that string, __post_init__ has already resolved it from your flags:

# vllm/config/parallel.py — ParallelConfig.__post_init__
if self.distributed_executor_backend is None and self.world_size_across_dp > 1:
    # ... platform/ray checks elided; the ordinary CUDA, single-node,
    # no-Ray path this module depicts lands here:
    backend = "mp"
    self.distributed_executor_backend = backend

if self.distributed_executor_backend is None and self.world_size == 1:
    self.distributed_executor_backend = "uni"

Read the two conditions in order and the precise boundary falls out on its own, for the ordinary CUDA, single-node, no-Ray, no-explicit-override deployment this module depicts. The first branch fires whenever world_size_across_dp — TP × PP × PCP × DP, the number Step 1 built — is greater than 1, and it fires before the second branch ever gets a chance to run. Only when that first branch does not fire — meaning world_size_across_dp is exactly 1 — does the second branch's world_size == 1 check even matter, and at that point it's trivially true. So under this default CUDA resolution, uni is reachable only when world_size_across_dp == 1, which since every axis is at least 1 means TP = PP = DP = 1, together. DP=2 at TP=1 makes world_size_across_dp equal 2, the first branch fires, and the backend is mp — no separate worker process claim survives that launch.

Two real exceptions live outside that default path, and this module leaves both alone: the world_size_across_dp > 1 branch itself carries a TPU-and-SPMD condition that selects uni anyway, and a deployment that passes --distributed-executor-backend uni explicitly skips this whole resolution — get_class reads the string you gave it, full stop. "Only when TP=PP=DP=1" describes what your flags produce by default, not a law the string comparison enforces.

The DP-scoped executor's own worker count

Once mp is selected, MultiprocExecutor._init_executor runs — once per EngineCore process, so once per DP replica — and spawns that replica's own workers:

# distilled — real names, reduced body: the multi-node nnodes assertion,# message-queue leader/follower branching, and per-worker CPU/OMP-affinity# setup all elidedclass MultiprocExecutor(Executor):  def _get_parallel_sizes(self) -> tuple[int, int, int]:      self.world_size = self.parallel_config.world_size      self.local_world_size = self.parallel_config.local_world_size      tp_size = self.parallel_config.tensor_parallel_size      pp_size = self.parallel_config.pipeline_parallel_size      pcp_size = self.parallel_config.prefill_context_parallel_size      return tp_size, pp_size, pcp_size   def _init_executor(self) -> None:      self.is_failed = False      tp_size, pp_size, pcp_size = self._get_parallel_sizes()      assert self.world_size == tp_size * pp_size * pcp_size, (          f"world_size ({self.world_size}) must be equal to the "          f"tensor_parallel_size ({tp_size}) x pipeline"          f"_parallel_size ({pp_size}) x prefill_context"          f"_parallel_size ({pcp_size}). "      )       for local_rank in range(self.local_world_size):          global_rank = global_start_rank + local_rank          unready_worker_handle = WorkerProc.make_worker_process(              vllm_config=self.vllm_config,              local_rank=local_rank,              rank=global_rank,              distributed_init_method=distributed_init_method,          )          unready_workers.append(unready_worker_handle)

Read that assert closely — it's the same arithmetic Step 1 walked through in prose, now enforced as a runtime invariant on this executor instance's own world_size: TP × PP × PCP, never DP. data_parallel_size doesn't appear anywhere in this function, because it doesn't need to: each DP replica's MultiprocExecutor only ever has to account for its own TP×PP workers. The spawn loop iterates self.local_world_size — world_size // nnodes_within_dp, this replica's worker count on this node — which is a single-node deployment's whole world_size, but not the same field once a replica spans more than one physical node; this module's grid stays single-node throughout, so the two coincide everywhere it draws from. Cross a DP boundary and you've crossed into a different EngineCore process running a wholly separate MultiprocExecutor instance with its own count, its own assertion, its own workers.

Say it out loud

--tensor-parallel-size, --pipeline-parallel-size and --data-parallel-size now map onto real processes: one EngineCore per DP rank, and inside each one, an executor chosen once by Executor.get_class from a backend string that world_size_across_dp — not TP alone — decided. uni means one process, full stop, for the whole launch; anything else means mp, and MultiprocExecutor._init_executor spawns one worker per TP×PP rank in that replica. The next step is what those workers are actually allowed to talk to.

How Does vLLM Decide Which Ranks Exchange Data?

How does vLLM decide which ranks exchange data with which?

Step 1 gave every rank a coordinate; Step 2 gave every rank a process. Neither one says anything about wiring — which ranks actually exchange tensors with which. That's a separate construction step, and it happens once, at startup, right after the workers exist: vLLM builds one communication group per rank per axis it cares about, and a rank only ever exchanges data with the other members of its own group on a given axis.

This module tracks three of those axes — TP, PP, DP — because they're the three you set as launch flags. initialize_model_parallel actually builds five real groups every time (_TP, _DCP, _PCP, _PP, _DP, plus _EP/_EPLB for mixture-of-experts models): decode-context parallelism (DCP) and prefill-context parallelism (PCP) are real fourth and fifth axes, fixed at 1 throughout this module, so their groups exist but degenerate to one rank each — not skipped, just trivial, the same way Step 5 shows the whole framework degenerating at TP=PP=DP=1.

Each axis needs a fundamentally different exchange, which is why there's one group per axis instead of one shared channel. A TP group all-reduces partial results from the same slice of work. A PP group hands hidden states downstream, one stage to the next. For a dense model, a DP group's members barely coordinate at all — mostly they just need to know who else exists. A mixture-of-experts (MoE) model under DP is the exception: its DP group all-reduces to synchronize "wave" progress across replicas, and vLLM's own coordinator-need check (needs_dp_coordinator) specifically carves out MoE-under-DP as needing that coordination even when nothing else about the deployment would.

One rank, three groups, built by holding the other two axes fixed

The construction rule is the same for every axis: fix the other two coordinates, and every rank sharing your remaining coordinate is in your group. Take an 8-rank grid at TP=2, PP=2, DP=2 — every axis doing real work, not a trivial 1:

Worth deriving the DP row by hand once, because it's the row easiest to get backwards. Rank 1 is (tp=1, pp=0, dp=0). Holding tp and pp fixed and varying dp means its DP-mate is whichever rank has (tp=1, pp=0, dp=1) — reading the rank table back, that's rank 5, not rank 6 (rank 6 is (tp=0, pp=1, dp=1), a different tp and a different pp). The real construction confirms it: initialize_model_parallel builds a 5D tensor in axis order [ExternalDP, DP, PP, PCP, TP] and gets the DP groups via all_ranks.transpose(1, 4).reshape(-1, dp_size).unbind(0) — swap the DP and TP axes, then slice off dp_size-sized chunks. That transpose is what "holding tp and pp fixed, varying dp" means in tensor terms, and it's what produces [0,4], [1,5], [2,6], [3,7] — never a pair that only shares one of the two held coordinates.

Every axis produces 4 groups of size 2 here — 4 × 2 = 8, the world size, on every single axis, because every rank belongs to exactly one group per axis by construction. Rank 3's world (on these three axes) is exactly three groups: its TP pair [2,3], its PP pair [1,3], its DP pair [3,7] — and nothing outside those three pairs. Reach past that boundary and there simply is no group to carry it; a rank exchanges data with its own TP peers on the TP axis, its own PP peers on the PP axis, its own DP peers on the DP axis, and never directly beyond any of the three.

The construction, in code

Every group — regardless of axis — is built the same way: a list of rank lists is handed to init_model_parallel_group, which is a thin wrapper:

# distilled — real names, unabridged: the function is this shortdef init_model_parallel_group(  group_ranks: list[list[int]],  local_rank: int,  backend: str,  use_message_queue_broadcaster: bool = False,  group_name: str | None = None,  use_device_communicator: bool = True,) -> GroupCoordinator:  return GroupCoordinator(      group_ranks=group_ranks,      local_rank=local_rank,      torch_distributed_backend=backend,      use_device_communicator=use_device_communicator,      use_message_queue_broadcaster=use_message_queue_broadcaster,      group_name=group_name,  )

group_ranks is the whole table for one axis at once — every group on that axis, not just the caller's own. The real work, and the answer to "how does a rank know which of those groups is its group," is in the constructor it hands off to:

# distilled — real names, reduced body: group naming/registration, timeout# handling, the split_group codepath, and post-construction device# assignment all elidedclass GroupCoordinator:  def __init__(      self,      group_ranks: list[list[int]],      local_rank: int,      torch_distributed_backend: str,      use_device_communicator: bool,      use_message_queue_broadcaster: bool = False,      group_name: str | None = None,  ):      self.rank = torch.distributed.get_rank()       for ranks in group_ranks:          device_group = torch.distributed.new_group(ranks, backend=torch_distributed_backend)          cpu_group = torch.distributed.new_group(ranks, backend="gloo")          if self.rank in ranks:              self.ranks = ranks              self.world_size = len(ranks)              self.rank_in_group = ranks.index(self.rank)              self_device_group = device_group              self_cpu_group = cpu_group       self.group_ranks = group_ranks      self.device_group = self_device_group      self.cpu_group = self_cpu_group

Read the loop literally: every rank in the process runs this same constructor, over the same full group_ranks list, and creates a real torch.distributed process group for every entry — including groups it isn't in. What makes each rank end up with its own GroupCoordinator is the if self.rank in ranks guard: only the one entry containing this process's own global rank gets kept as self.device_group. Every other entry in the list was necessary to construct — torch.distributed.new_group is a collective call every participating rank must join — but only one of them survives as this rank's own group. Do this once per axis and every rank ends up holding one GroupCoordinator per axis vLLM builds — _TP, _DCP, _PCP, _PP, _DP unconditionally, _EP/_EPLB for MoE models — three of which matter to this module: its TP group, its PP group, its DP group.

Which device a rank ends up on

One more piece of startup wiring belongs here: given a rank's coordinate, which physical GPU does it actually claim? Worker.init_device answers it with one line of arithmetic, run once per worker process:

# distilled — real names, reduced body: Ray/external-launcher and# multi-node branches elided (single-node CUDA path only)class Worker:  def init_device(self):      if self.device_config.device_type == "cuda":          dp_local_rank = self.parallel_config.data_parallel_rank_local          tp_pp_world_size = (              self.parallel_config.pipeline_parallel_size              * self.parallel_config.tensor_parallel_size          )          # DP_LOCAL_RANK * TP_PP_WORLD_SIZE + TP_LOCAL_RANK          self.local_rank += dp_local_rank * tp_pp_world_size           visible_device_index = (              current_platform.logical_device_id_to_visible_device_id(self.local_rank)          )          self.device = torch.device(f"cuda:{visible_device_index}")          torch.accelerator.set_device_index(self.device)

Each DP replica's own worker started counting local ranks from 0 — that's what Step 2's spawn loop does inside MultiprocExecutor._init_executor. This offset is what turns "local rank 0 within replica 1" into a distinct physical device from "local rank 0 within replica 0": it shifts by a whole tp_pp_world_size block per DP replica, so two replicas sharing a node claim disjoint GPU ranges instead of colliding on the same device index. Device assignment, like the groups above it, is computed per rank from its coordinate — nothing here is decided once for the whole launch.

What Actually Goes Out Over the Wire?

What actually goes out over the wire, every single forward pass?

EngineCore.step calls self.model_executor.execute_model(scheduler_output, non_block=True) once per pass, and — as the Model Runner module already established — that line never changes depending on which executor is running. What changes underneath it, on the mp executor Step 2 selected whenever more than one rank is in play, is a real fan-out: one Python call becomes one message reaching every worker this executor owns, and one worker's answer becomes the return value.

Note the scope carefully, because it's easy to over-read: this executor owns one DP replica's workers, and no others. MultiprocExecutor is a per-EngineCore-process object — Step 2 built one of these per DP rank — so a call into it fans out across that replica's own TP×PP ranks and stops there. A launch with DP=2 runs two of these fan-outs, on two separate MultiprocExecutor instances, in two separate EngineCore processes, and neither instance's call ever reaches the other replica's workers.

The call, and who it reaches

# distilled — real names, reduced body: kv_output_aggregator branch,# deadline/timeout bookkeeping, and the non_block/future-wrapping return# (Model Runner's territory) all elidedclass MultiprocExecutor(Executor):  def collective_rpc(      self,      method: str | Callable,      args: tuple = (),      kwargs: dict | None = None,      unique_reply_rank: int | None = None,  ) -> Any:      assert self.rpc_broadcast_mq is not None, (          "collective_rpc should not be called on follower node"      )      if self.is_failed:          raise RuntimeError("Executor failed.")       output_rank = unique_reply_rank      if isinstance(method, str):          send_method = method      else:          send_method = cloudpickle.dumps(method, protocol=pickle.HIGHEST_PROTOCOL)      self.rpc_broadcast_mq.enqueue((send_method, args, kwargs, output_rank))       response_mqs = self.response_mqs      if output_rank is not None:          response_mqs = (response_mqs[output_rank],)      # ... get_response reads response_mqs and returns the result(s) —      # the message-queue mechanics themselves are Model Runner's territory

self.rpc_broadcast_mq is a single message queue this executor's own workers all read from — exactly the workers MultiprocExecutor._init_executor spawned in Step 2, no others. enqueue puts one (method, args, kwargs, output_rank) tuple on it, and every one of those workers picks it up and calls the named method on itself. One Python call in the engine core becomes N calls, one per worker in this replica — not N calls scattered across every rank in the launch.

One call in, one reply read

output_rank is the part worth sitting with, because it decides how many of those N replies the engine core actually looks at. Every worker still runs the method — there's no way to fan a call out to fewer workers than own the batch — but response_mqs[output_rank] narrows the read side down to a single queue before anything is dequeued. For execute_model and sample_tokens, that single reader is computed once per executor and never changes:

# vllm/v1/executor/multiproc_executor.py — MultiprocExecutor._get_output_rank
# Only returns ModelRunnerOutput from TP rank=0 and PP rank=-1
# (the first TP worker of the last PP stage).
return (
    self.world_size
    - self.parallel_config.tensor_parallel_size
    * self.parallel_config.prefill_context_parallel_size
)

Take the TP=2, PP=2, DP=2 grid from Step 3 and pick DP replica 0 — local ranks 0, 1, 2, 3, world_size (per-replica) is 4. _get_output_rank returns 4 - 2×1 = 2: local rank 2, which is (tp=0, pp=1) — the first TP worker of the last PP stage, same as the Model Runner module already named it. Every one of the four workers in this replica runs execute_model; only rank 2's ModelRunnerOutput ever reaches EngineCore.step. The other three did real GPU work — their slice of the forward pass — and their return value is simply never dequeued into the engine's result.

And on the common path this happens up to twice every pass, not always exactly once: the request-path module already established that execute_model and sample_tokens are two separate calls, with real CPU work for the engine in between, and EngineCore.step only makes the second call when the first one's single read reply is None. Module 1's 5-engine-loop.mdx carries the three qualifications where that guard goes the other way — a pass that scheduled no tokens returns an empty output, a pooling model returns its pooled output directly, and a non-final pipeline-parallel rank returns hidden states instead — and the second call is skipped in exactly those cases. Every call that does happen, though, is its own collective_rpc — its own fan-out to the same replica, its own single reply read — not one round trip doing double duty.

Simplified Stand-In, or the Real Thing?

Is the single-process executor a simplified stand-in, or the real thing at a smaller size?

The Model Runner module already put UniProcExecutor and MultiprocExecutor side by side and traced their collective_rpc bodies line by line — run_method in-process versus a message queue, a pre-resolved future versus a dequeued one. That's the implementation difference, and it's already covered; this step isn't repeating it.

What that comparison didn't ask is a topology question: is uni a special case bolted onto the side of the rank-grid-and-groups machinery Steps 1-4 just built, or is it what that same machinery produces when you turn every dial down to its minimum? Trace it through the same four steps and the answer is the second one.

The same four steps, at the smallest possible grid

Set TP=1, PP=1, DP=1 — world_size_across_dp collapses to 1, the boundary Step 2 derived precisely. Walk that layout through everything built so far:

Nothing in that table is a shortcut. uni isn't a separate code path that skips the rank-grid framework when there's nothing interesting to distribute — it's a regular point in the same space Steps 1-3 built, evaluated where every axis bottoms out at 1. That's the sense in which it's not a toy: the abstraction doesn't change shape at the boundary, only its size does.

The call still has the same shape — it just doesn't need to choose a reply

collective_rpc's contract is a method name, some arguments, and a promise about who answers. MultiprocExecutor needs unique_reply_rank to narrow N replies down to one, because Step 4 showed it really does have N workers to hear from. UniProcExecutor never needs that parameter at all — not because the contract is different, but because there's exactly one worker and it is trivially both the caller and the only possible reply:

# distilled — real names, reduced body: the non_block/future-wrapping path# and the async-scheduling output unwrap are Model Runner's territory; this# is the synchronous call onlyclass UniProcExecutor(Executor):  def collective_rpc(      self,      method: str | Callable,      args: tuple = (),      kwargs: dict | None = None,      single_value: bool = False,  ) -> Any:      result = run_method(self.driver_worker, method, args, kwargs)      return result if single_value else [result]

self.driver_worker is the one worker object this executor owns — not chosen from a set of candidates by an output_rank calculation, because there was never a set to choose from. Compare Step 4's MultiprocExecutor, where unique_reply_rank had to be computed from world_size and the parallel sizes precisely because the reply could have come from any of several workers. Here there's no computation because there's no ambiguity to resolve — the single worker is the reply, by construction, not by a narrowing rule that happens to degenerate to "pick the only option."

Say it out loud

uni and mp aren't two different designs for "how a request reaches the GPU" — they're the same design, collective_rpc fanning out to whatever workers an executor owns, evaluated at two different points on the same TP×PP×DP grid this module has built one step at a time. The grid, the groups, and the fan-out don't change shape between them; only how many workers show up on the other end of the call does.

What Happens When One Worker Process Dies?

What happens when one worker process dies during live serving?

The specimen: a --tensor-parallel-size 2 deployment, mp executor, two worker processes. One of them gets OOM-killed by the kernel — or segfaults, or is kill -9'd by an operator who targeted the wrong PID. The process is simply gone. Nothing about the request that was in flight changed; the operating system just removed one of the two processes this replica's executor depends on.

The reasonable-sounding guess is that the next call into that worker raises some kind of connection error, and the other worker keeps going fine since it never crashed. Neither half of that guess survives a trace of the source.

Tracing what actually happens

A daemon thread, started once at construction, is the only thing watching for this:

# vllm/v1/executor/multiproc_executor.py — MultiprocExecutor.start_worker_monitor
def monitor_workers():
    sentinels = [h.proc.sentinel for h in workers]
    died = multiprocessing.connection.wait(sentinels)
    _self = self_ref()
    if not _self or getattr(_self, "shutting_down", False):
        return
    _self.is_failed = True
    logger.error("Worker proc %s died unexpectedly ..., shutting down executor.", ...)
    _self.shutdown()
    callback = _self.failure_callback
    if callback is not None:
        _self.failure_callback = None
        callback()

multiprocessing.connection.wait(sentinels) blocks until any one of this executor's worker processes exits — for any reason, crash or otherwise. The instant that happens, is_failed flips to True on this executor instance — the one belonging to the DP replica the dead worker was part of — and shutdown() runs immediately, unconditionally, on the same thread:

# vllm/v1/executor/multiproc_executor.py — MultiprocExecutor.shutdown
def shutdown(self):
    if not getattr(self, "shutting_down", False):
        self.shutting_down = True
        if workers := getattr(self, "workers", None):
            self._ensure_worker_termination([w.proc for w in workers])
            for w in workers:
                if w.worker_response_mq is not None:
                    w.worker_response_mq.shutdown()
    if rpc_broadcast_mq := getattr(self, "rpc_broadcast_mq", None):
        rpc_broadcast_mq.shutdown()
    if response_mqs := getattr(self, "response_mqs", None):
        for mq in response_mqs:
            mq.shutdown()

Two things in that body matter and are easy to miss reading quickly. First, _ensure_worker_termination runs on workers, the whole list — not just the one that already died. This replica's surviving worker gets torn down deliberately, as part of the same shutdown, not because it also crashed. Second, every message queue this executor owns — the broadcast queue every worker reads from, and every worker's own response queue — gets .shutdown() called on it, and each one of those just flips a flag:

# vllm/distributed/device_communicators/shm_broadcast.py — MessageQueue.shutdown
def shutdown(self):
    self.shutting_down = True
    if self._spin_condition is not None:
        self._spin_condition.cancel()

Two different symptoms, depending on timing — and why they're different

That one flag produces two distinct exceptions depending on exactly when a caller touches this executor, and this is the part worth being precise about.

A collective_rpc call already in flight, blocked reading a reply, is sitting inside get_response's mq.dequeue(...), which is MessageQueue.acquire_read's wait loop:

# vllm/distributed/device_communicators/shm_broadcast.py — MessageQueue.acquire_read
while True:
    ...
    self._spin_condition.wait(timeout_ms=read_timeout.timeout_ms())
    if self.shutting_down:
        raise RuntimeError("cancelled")

The moment shutdown() flips shutting_down and cancels the spin condition, that wait wakes up, the condition is true, and the blocked call raises RuntimeError("cancelled") — a generic message with no worker id, no rank, nothing that names the dead process, because from this call's point of view it never got as far as reading a reply at all.

A new collective_rpc call, issued after is_failed is already True, never gets that far. It hits the guard at the very top of the method, before any message is even enqueued:

# vllm/v1/executor/multiproc_executor.py — MultiprocExecutor.collective_rpc
if self.is_failed:
    raise RuntimeError("Executor failed.")

Neither of those two messages is "Worker failed with error '...'". That third message belongs to a different failure mode entirely — an exception raised inside a worker's own forward pass, on a worker process that keeps running: the Model Runner module already traced it, where worker_busy_loop catches the exception, reports a failure status over the worker's own response queue, and get_response's status check is what raises that text. That path requires the process to survive long enough to report its own failure. This specimen is the opposite case — the process is gone, there is no status to report, and the two messages above ("cancelled" and "Executor failed.") are what a dead process produces instead.

What the reader actually sees

Either exception propagates out of collective_rpc the same way any exception from that call would — up through the executor's own execute_model/sample_tokens, the methods that called collective_rpc in the first place, then up through EngineCore.step, and eventually into whatever request was relying on that pass, exactly like the ordinary error path the request-path module already traced. The affected set is every rank sharing the dead worker's DP index — this replica's other TP and PP ranks go down with it, because shutdown() terminates the whole workers list, not just the one that already exited. A different DP replica, running its own separate MultiprocExecutor instance with its own is_failed flag, never sees any of this; its monitor_workers thread is watching a completely different set of sentinels.

None of this applies under uni. There is no monitor thread, no message queue, no second process to lose — the call is a plain function call inside the EngineCore process itself, and an exception there surfaces synchronously, at the call site, the same way any Python exception would. "One rank dying" isn't a failure mode uni has, because uni's one rank is the process asking the question.