vLLM's Compilation Config and CUDA Graph Capture — the Startup Ladder, in Source

What You Know

What do you already know about CUDA graph capture?

If you've done the CUDA Graphs module of the LLM Serving track, you already have the mechanism this module builds on — capture once, replay for free, tied to a fixed shape and fixed buffer addresses. This module doesn't re-explain any of that. If it's unfamiliar, go read Steps 3 and 4 of CUDA Graphs first — everything below assumes it.

What that module didn't show you is vLLM's own code: the object that decides which shapes get captured, the mechanism that decides which parts of the model even can be captured, and what a real request's batch size does when it isn't one of the captured sizes. That's this module's whole job, end to end.

The delta, in one sentence

You already know that capture-then-replay eliminates per-kernel launch overhead. What you don't yet know is that vLLM doesn't try to capture the whole model as one graph, and doesn't try to capture every possible batch size — it makes both of those decisions once, in a typed config object, before a single request arrives, and every request afterward just gets routed against that decision.

Four objects, one config

Everything in this module traces back to a single dataclass, CompilationConfig (vllm/config/compilation.py), and four things it controls:

One more thing worth naming before Step 2 opens the config up: none of this is a per-request decision. Module 1's annotated vllm serve command flagged --compilation-config as landing on "startup, then 7" — startup is where the config is fixed, the graph gets split, and every rung on the ladder gets captured; phase 7 (the forward pass, in this track's request-path numbering) is where an already-decided graph gets replayed, or doesn't. This module lives almost entirely in that first half. Step 6 is the one place it has to talk about what happens during phase 7, and even then, what happens there was decided at startup — nothing gets reconsidered per request.

One Config, Five Fields

What is CompilationConfig?

CompilationConfig is the dataclass (vllm/config/compilation.py) that owns every decision this module traces: whether vLLM compiles the model at all, how it captures CUDA graphs around it, and which batch sizes get a graph. It's one of vLLM's per-domain config objects — the same family as SchedulerConfig and CacheConfig — and it's what --compilation-config on the command line actually populates.

# distilled — real names, reduced body: the five fields this module traces@configclass CompilationConfig:  mode: CompilationMode = None  # None resolves to VLLM_COMPILE (3) at the default optimization  # level; a lower level (-O0) resolves it to NONE instead.  # 0 NONE — fully eager, no torch.compile at all.  # 1 STOCK_TORCH_COMPILE — the standard torch.compile pipeline.  # 2 DYNAMO_TRACE_ONCE — one Dynamo trace, no recompilation.  # 3 VLLM_COMPILE — vLLM's own Inductor backend: caching,  #   piecewise compilation, shape specialization, custom passes.   cudagraph_mode: CUDAGraphMode = None  # NONE, PIECEWISE, FULL, FULL_DECODE_ONLY, FULL_AND_PIECEWISE.  # FULL_AND_PIECEWISE (v1 default): full cudagraph for decode  # batches, piecewise cudagraph for prefill and mixed batches.   splitting_ops: list[str] | None = None  # Ops excluded from cudagraphs -- where the FX graph gets split.  # If None, defaults to attention ops for piecewise cudagraphs.   cudagraph_capture_sizes: list[int] = None  # The sizes vLLM actually captures a graph for.  # None (default): inferred automatically. list[int]: as given.   max_cudagraph_capture_size: int = None  # The top of the ladder -- nothing above this size is captured.

Five fields, five jobs:

mode and cudagraph_mode are independent knobs that happen to combine in the default configuration. Piecewise compilation (the FX-level split Step 3 traces) requires mode = VLLM_COMPILE. Full cudagraphs work with or without it — and even when both are on, the FX graph can still keep its piecewise structure underneath; what "full" actually means is that the captured graph wraps the whole model in one shot, not that the split disappears. The v1 default runs both at once: VLLM_COMPILE plus FULL_AND_PIECEWISE, which is exactly why Step 3 has something to explain — a piecewise split and a full-graph capture path, coexisting.

Why Split the Graph

Why does vLLM split the graph instead of compiling one blob?

Some operations are cudagraph-incompatible often enough that vLLM's default is to keep them out of the piecewise-captured region entirely. Attention is the main one: recall the Attention Backends module — every forward pass, the backend needs a fresh CommonAttentionMetadata, rebuilt from that step's own block table and sequence lengths, and a mixed prefill-decode batch can vary that metadata's shape (not just the values inside a fixed shape) from step to step in ways a single static graph can't safely absorb. That's a default for general-purpose flexibility, not a hard law — the config's own cudagraph_mode documents a FULL_DECODE_ONLY variant that does capture attention whole, for the simpler case of a uniform, pure-decode batch. CompilationConfig.splitting_ops is the list of operations excluded from the piecewise region by default — attention variants plus the ops that write into the KV cache — and split_graph is what actually cuts the traced graph at each one, so the ordinary matmul-and-norm stretches between them can still be captured.

Where the split happens

VllmBackend.__call__ is the entry point torch.compile calls with the model's traced FX graph — torch.fx's representation of the model as a graph of operations, the same one split_graph walks node by node below. Most of __call__'s body is cache-key bookkeeping; the part this module cares about is five lines — decide whether to pre-split at all, then call split_graph:

# distilled — real names, reduced body: where the split happensclass VllmBackend:  def __call__(self, graph: fx.GraphModule, example_inputs):      # ... hashing and cache-directory setup omitted ...       if self.compilation_config.use_inductor_graph_partition:          # Inductor partitions the graph itself downstream;          # skip the FX-level pre-split entirely.          fx_split_ops: list[str] = []      else:          fx_split_ops = self.compilation_config.splitting_ops or []       self.split_gm, self.piecewise_graphs = split_graph(graph, fx_split_ops)       submod_names_to_compile = [          item.submod_name          for item in self.piecewise_graphs          if not item.is_splitting_graph      ]       # Compile every non-split submodule with symbolic shapes, up      # front, so compilation is finished before this callable returns.      PiecewiseCompileInterpreter(          self.split_gm, submod_names_to_compile, self.vllm_config, self      ).run(*fake_args)       return VllmSerializableFunction(...)

splitting_ops — the config field Step 2 named — is handed straight to split_graph. Everything past that call is bookkeeping: naming and compiling the pieces that came out.

How split_graph decides where to cut

# distilled — real names, reduced body: the node-by-node walkdef split_graph(graph: fx.GraphModule, splitting_ops: list[str]):  subgraph_id = 0  node_to_subgraph_id: dict[fx.Node, int] = {}  split_op_graphs: list[int] = []   for node in graph.graph.nodes:      if node.op in ("output", "placeholder"):          continue       if should_split(node, splitting_ops):          subgraph_id += 1          node_to_subgraph_id[node] = subgraph_id          split_op_graphs.append(subgraph_id)           # keep consecutive splitting ops together          if should_split(node.next, splitting_ops):              subgraph_id -= 1          else:              subgraph_id += 1      else:          node_to_subgraph_id[node] = subgraph_id   # torch.fx's split_module() cuts the graph along  # node_to_subgraph_id and returns the pieces  return split_gm, outputs

Walk it by hand for one real, adjacent pair rather than a hypothetical: Attention.forward (vllm/model_executor/layers/attention/attention.py) calls unified_kv_cache_update immediately before unified_attention_with_output — both real entries in the default splitting_ops list, back to back, no other op between them. Ordinary nodes before the pair keep subgraph_id = 0. The KV-cache-update node hits should_split, increments to 1, and claims that id — then, because its very next node (the attention op) is also a split op, the decrement immediately undoes the increment: subgraph_id drops right back to 1 before the attention node claims that same id. Only once both are behind does the next ordinary node open id 2. Two adjacent split-op nodes, one shared id — that's "keep consecutive splitting ops together," in the code's own words. A single, isolated split-op node with no adjacent partner behaves the same way arithmetically (claim an id, then move the next ordinary node on to a fresh one); it just never needs the merge.

Why the count scales with layers, not with the config list

Generalize the hand trace: for N split-op occurrences that don't sit next to each other in the graph — where one "occurrence" can itself be a merged pair, like the KV-cache-update-then-attention pair above — the walk produces 2N + 1 subgraphs, not N + 1. Every decoder layer contributes exactly one such occurrence: Attention.forward emits its update-then-attention pair once per layer, the merge collapses that pair to a single id, and no two layers' occurrences ever land adjacent to each other — there's a full block of non-split compute (MLP, norm, residual) between them. So the number of subgraphs a real model produces scales with how many decoder layers it has, not with how many entries happen to be listed in splitting_ops, and not with how many individual split-op nodes each layer emits either — the merge is exactly what keeps a two-op pair from counting as two occurrences. A 3-layer toy model and a 96-layer production model can have the identical splitting_ops config and produce wildly different subgraph counts.

The real default splitting_ops list is longer than you might expect: 15 attention-variant ops (CompilationConfig._attention_ops — covering standard attention, MLA, and several state-space-model variants for hybrid architectures), plus 2 KV-cache-update ops appended by set_splitting_ops_for_v1 — 17 op types in total, not 2. The simulation's own fixture only shows 2 of those 17, labeled explicitly as an illustrative subset — worth knowing going in, so a short list in the panel doesn't read as the real default.

The Shape Ladder

Which batch shapes actually get a captured CUDA graph?

cudagraph_capture_sizes is a fixed, ascending list of batch sizes, decided before any request arrives. max_cudagraph_capture_size is its last entry — post_init_cudagraph_sizes asserts the two agree. Left unset, vLLM generates the list itself, following a pattern straight out of the config's own docstring:

[1, 2, 4] + list(range(8, 256, 8)) + list(range(256, max_cudagraph_capture_size + 1, 16))

Dense at small sizes, sparser at large ones — steps of 8 up to 256, then steps of 16. If that shape looks familiar, it should: it's the same production bucket list the shipped CUDA Graphs module showed you as vLLM's real default. This module is where that list actually comes from. Left unspecified, max_cudagraph_capture_size itself defaults to min(max_num_seqs * 2, 512) — small enough to avoid OOM in tight-memory setups, capped at 512 so startup doesn't spend forever capturing graphs nobody will use.

What happens to a shape that isn't itself a rung

This is the fact worth getting exactly right, because the wrong version of it is easy to guess and sounds plausible: a batch size that lands strictly between two rungs is not a miss. It's padded up to the next rung and still runs under a captured graph. Only a batch size above the top rung has nothing to pad to and falls back to running without one.

The mechanism lives in CudagraphDispatcher (vllm/v1/cudagraph_dispatcher.py) — outside this module's eight cited symbols, but worth tracing precisely because the two branches look similar and aren't. It precomputes a lookup table sized exactly max_cudagraph_capture_size + 1 — every index in range gets a padded destination, and for any batch size strictly between two captured sizes, that destination is the next higher one. There is no separate "no rung matches" outcome inside that range; every size from 0 up to the top rung maps to some captured size. The table simply doesn't have an entry past the top rung, because a size that large was never going to fit anywhere on the ladder in the first place — that's Step 6's territory, not this one.

CUDAGraphWrapper — the object that actually captures or replays — never computes any of this itself. Its own docstring says so plainly: at runtime it "receives a runtime_mode and a batch_descriptor(key) from the forward context and blindly trust[s] them for cudagraph dispatching." The dispatcher decides the padded size upstream and hands it down; the wrapper's whole job, once it trusts that key, is capture-the-first-time / replay-every-time-after:

# distilled — real names, reduced body: the capture-and-replay path;# the early-return fallback branch is Step 6's excerptclass CUDAGraphWrapper:  def __call__(self, *args, **kwargs):      forward_context = get_forward_context()      batch_descriptor = forward_context.batch_descriptor      cudagraph_runtime_mode = forward_context.cudagraph_runtime_mode       if (cudagraph_runtime_mode == CUDAGraphMode.NONE              or cudagraph_runtime_mode != self.runtime_mode):          return self.runnable(*args, **kwargs)  # Step 6's branch       if batch_descriptor not in self.concrete_cudagraph_entries:          self.concrete_cudagraph_entries[batch_descriptor] = CUDAGraphEntry(              batch_descriptor=batch_descriptor)      entry = self.concrete_cudagraph_entries[batch_descriptor]       if entry.cudagraph is None:          # first call for this batch_descriptor: run it for real,          # inside torch.cuda.graph(), which records every op as it runs          cudagraph = torch.cuda.CUDAGraph()          with torch.cuda.graph(cudagraph, pool=self.graph_pool):              output = self.runnable(*args, **kwargs)          entry.output, entry.cudagraph = output, cudagraph          return output       # a later call, same batch_descriptor: replay what was captured —      # no Python-level forward pass runs at all      entry.cudagraph.replay()      return entry.output

Notice what "captured" means operationally, and notice when the entry.cudagraph is None branch actually fires. It is not a live request that pays for the first capture — Step 5 traces the real trigger: Worker.compile_or_warm_up_model runs a dummy forward pass for every rung on the ladder, deliberately, before the engine opens for traffic, and that dummy pass is what walks into this exact branch and populates entry.cudagraph for every padded size the deployment will ever see. By the time a real request arrives, its padded batch size should already have an entry with entry.cudagraph set — so an ordinary request, on its very first appearance, takes the replay branch: no Python-level forward pass runs at all, just a driver call. An exact rung match and a padded, between-rungs match go through exactly the same code from this point on — the wrapper has no idea, and no need to know, which one it was.

What Startup Buys You

What does vLLM's startup wait actually buy you?

Everything Steps 2 through 4 described — fixing the config, splitting the graph, capturing every rung on the ladder — has to actually run somewhere, in some order, before the engine opens for traffic. This step is that timeline.

What already happened, one slot earlier

By the time this module's work starts, the KV cache has already been sized. EngineCore._initialize_kv_caches runs three calls in exactly this order, and the order is load-bearing, not incidental: Executor.get_kv_cache_specs first, to learn what KV cache each worker's layers actually need — and, for an attention-free model, to learn that no memory-profiling call is even necessary; Executor.determine_available_memory second (skipped entirely when the first call found nothing to size), independently profiling peak activation memory to see how much GPU memory is left over; get_kv_cache_configs third, the one call that actually needs both prior results together, turning specs plus that memory figure into a concrete block count. All three finish before compiling starts, because compiling and capturing consume GPU memory of their own — the source even compares the CUDA graph memory it actually used, after capture, against an estimate it made during that earlier profiling pass, specifically so the two don't silently drift apart. Get the order backwards and graph capture doesn't know how much room the KV cache already claimed.

That earlier work is startup slot 3 — this track's KV Cache Manager territory, not re-taught here. The moment it hands off matters, though: Executor.initialize_from_config allocates the KV cache the sizing step decided on, and the very next call in the same sequence is Worker.compile_or_warm_up_model — this module's slot 4.

Four things one call buys

# distilled — real names, reduced bodydef compile_or_warm_up_model(self) -> CompilationTimes:  warmup_sizes: list[int] = []   if self.vllm_config.compilation_config.mode == CompilationMode.VLLM_COMPILE:      # sizes the user still wants compiled that ARE NOT on the      # capture ladder — e.g. a chunked-prefill token budget      compile_sizes = self.vllm_config.compilation_config.compile_sizes      warmup_sizes = compile_sizes.copy() if compile_sizes else []      cg_sizes = self.vllm_config.compilation_config.cudagraph_capture_sizes or []      warmup_sizes = [x for x in warmup_sizes if x not in cg_sizes]   for size in sorted(warmup_sizes, reverse=True):      self.model_runner._dummy_run(size, skip_eplb=True, remove_lora=False)   kernel_warmup(self)  # tune the real kernels before capture begins   cuda_graph_memory_bytes = 0  if not self.model_config.enforce_eager:      cuda_graph_memory_bytes = self.model_runner.capture_model()   if get_pp_group().is_last_rank:      # one more dummy forward, deliberately WITHOUT a cudagraph, to      # preallocate the sampler's own buffers before serving starts      max_num_reqs = min(self.scheduler_config.max_num_seqs,                          self.scheduler_config.max_num_batched_tokens)      hidden, last_hidden = self.model_runner._dummy_run(          num_tokens=max_num_reqs, skip_eplb=True,          cudagraph_runtime_mode=CUDAGraphMode.NONE)      self.model_runner._dummy_sampler_run(hidden_states=last_hidden)

Four stages, four different things paid for:

Nothing in this function's own source states a millisecond or microsecond figure for any of this — no line here claims a specific latency, and this module won't invent one. What the source does say, in a log line emitted for every size on the compile-and-warm-up list, is exactly which size is being compiled and when — "Compile and warming up model for size %d" — which is the honest way to answer "how long does this take": read the log, on your own hardware, with your own model.

How It Fails

What happens when a batch is too big for any captured graph?

Here is a specimen with numbers you've already seen elsewhere in this track. Module 1's annotated vllm serve command ran with --max-num-seqs 256 and --max-num-batched-tokens 8192. Leave max_cudagraph_capture_size at its auto-generated default — min(max_num_seqs * 2, 512) — and that config lands on a ceiling of 512. Nothing exotic has to happen for a single forward pass to schedule more than 512 tokens: a chunked-prefill step running anywhere near that 8192-token budget clears the ceiling by more than 15×, on a perfectly ordinary launch command this track already showed you.

Tracing what actually happens, not what sounds plausible

The reasonable-sounding guess is that a shape like this "misses the ladder" the same way a shape between two rungs does — some fallback logic kicks in, maybe a warning fires. That's not what the source does, and Step 4 already showed why: everything up to and including the top rung pads to something. The real branch is upstream of that padding logic entirely.

CudagraphDispatcher.dispatch(num_tokens=...) — module 1's "tokens per step, not requests" framing applies exactly here, since num_tokens is the whole step's token count, not a per-request one — checks one condition before it looks at any individual rung:

if (
    not self.keys_initialized
    or self.cudagraph_mode == CUDAGraphMode.NONE
    or max_size is None
    or num_tokens > max_size
    or allowed_modes <= {CUDAGraphMode.NONE}
):
    return CUDAGraphMode.NONE, BatchDescriptor(num_tokens)

Four of those five clauses are lifecycle and configuration guards this deployment isn't hitting: cudagraphs disabled outright, the dispatcher not yet initialized, no ceiling configured, no eligible mode left after exclusions. Holding all four of those constant, num_tokens > max_size is checked unconditionally, before the padded-size lookup table (Step 4's mechanism) is ever consulted — and it's what actually trips for this specimen. An 8192-token step against a 512-token ceiling trips it immediately, and the dispatcher hands back CUDAGraphMode.NONE — not a specific rung, not "closest available," nothing to pad to.

That five-clause guard is not the only place dispatch can return CUDAGraphMode.NONE, and this module's ceiling framing is specifically about the config it has assumed since Step 2. Past the guard, dispatch builds the padded descriptor and checks it against whichever keys were actually captured (Step 5's territory); if neither FULL nor PIECEWISE has a matching key, the function falls through to the identical return CUDAGraphMode.NONE, BatchDescriptor(num_tokens) at the very end. Under FULL_AND_PIECEWISE — the v1 default, and every example in this module — a relaxed PIECEWISE key gets created for every size in cudagraph_capture_sizes at capture time, so any in-range batch should always find one; the terminal return is effectively unreachable for the config this module teaches. It is reachable under other modes: FULL_DECODE_ONLY never populates PIECEWISE keys at all, so a mixed prefill-decode batch — even one comfortably under the ceiling — falls through that same terminal line for a different reason entirely: no captured key matches its shape, not because it was too big. This step's specimen and its "only the ceiling" framing are scoped to the default config; a deployment running a different cudagraph_mode has its own separate miss path this module doesn't trace.

That NONE travels down through the forward context to exactly the branch Step 4's excerpt named but didn't expand:

# distilled — real names, reduced body: the fallback branch only;# the capture-and-replay path is Step 4's excerptclass CUDAGraphWrapper:  def __call__(self, *args, **kwargs):      if not is_forward_context_available():          # outside the normal inference path (e.g. a vision          # encoder pass): run the function, no cudagraphs at all          return self.runnable(*args, **kwargs)       forward_context = get_forward_context()      cudagraph_runtime_mode = forward_context.cudagraph_runtime_mode       if (cudagraph_runtime_mode == CUDAGraphMode.NONE              or cudagraph_runtime_mode != self.runtime_mode):          # NONE covers a profiling run, a warmup run, or — this          # step's case — a shape the dispatcher just rejected          return self.runnable(*args, **kwargs)       # capture-or-replay continues here — see Step 4

self.runnable(*args, **kwargs) is the already-compiled function, called exactly like any other Python call — no torch.cuda.graph() context, no capture attempt, no replay. Ordinary eager execution of a piece that Step 3 already made cudagraph-capable; it just never got a graph for this particular shape.

The part worth naming explicitly: nothing here raises

Read the dispatcher's ceiling check and the wrapper's fallback branch themselves — both are bare return statements, with no exception and no log call sitting next to either one. No exception aborts the request, no warning marks the moment. The request still completes — correctly — just without the launch-overhead savings a captured graph would have bought it. It happens again on every subsequent step that lands above the ceiling, for as long as the deployment keeps scheduling steps that size.

That's silent by default, not silent by design. cuda_graph.py defines a real observability path for exactly this: CUDAGraphStat records runtime_mode (a NONE fallback included) for every dispatched forward pass, CUDAGraphLogging aggregates those into a table with a Runtime Mode column, and gpu_model_runner.py builds one of these stats per pass when ObservabilityConfig.cudagraph_metrics is set — which it isn't unless a deployment turns it on. So the honest version of this step's diagnostic advice is: with that flag off (the default), nothing in the log tells you a graph didn't fire, and a latency regression is the only symptom. With it on, the aggregated table's Runtime Mode column names the fallback directly.

Switch to the simulation's Off ladder stage and look at its one card, "48 in flight" against a ladder topping out at 32. The panel marks it not-captured and names the eager fallback in its own text — but nothing about the panel's amber styling exists in the real vLLM logs. That contrast is the point: the simulation makes visible exactly what the source leaves silent.