vLLM's Attention Backend Selection — the Platform Layer, in Source

Three Machines, One Question

What decides which attention backend a request uses?

Nothing about the request does. This module's simulation puts three machines side by side — an H100 on CUDA, an A100 on CUDA, an MI300X on ROCm — and each one settles on its attention kernel before any of them has served a single token. Backend selection is configuration plus hardware compatibility, resolved once as attention layers are constructed, and validated at startup. Whatever a machine lands on becomes fixed — all the traffic that machine serves afterward runs through that same backend.

That is the whole claim of this module, and it is worth sitting with, because the natural mental model — "the engine picks a kernel for this request" — is wrong in a way that matters. If backend choice were decided fresh for each incoming request, every forward pass would need a branch: check the request, check the hardware, pick a kernel, then run it. Instead vLLM pays that cost exactly once, while the process is still starting up, and the request path never sees it again.

Three machines, three deployments

The simulation to the right sets up the running example for all six steps: an H100 (compute capability 9.0), an A100 (compute capability 8.0), and an MI300X (ROCm, no CUDA compute capability at all). Each is running its own model, which is why their attention head dimensions differ too — this isn't one model cloned across three boxes, it's three independent deployments that happen to share a question: which attention backend do I use? Stage 1 shows exactly that — three machines, nothing decided yet.

The rest of this module answers the question in the order the source code actually asks it: what hardware am I on (Step 2), what class does that resolve to (Step 3), what does that class need from me on every forward pass (Step 4), what would it take to add a fourth backend (Step 5), and what happens if a machine is asked for something it cannot run (Step 6).

Where "startup" actually begins

"At startup" is doing real work in that claim, so it deserves a real anchor. EngineCore.__init__ is where the engine-core process's whole existence begins — before a scheduler exists, before a model is loaded, before any of the machinery this module walks through has run.

class EngineCore:  """Inner loop of vLLM's Engine."""   def __init__(      self,      vllm_config: VllmConfig,      executor_class: type[Executor],      log_stats: bool,      executor_fail_callback: Callable | None = None,      include_finished_set: bool = False,  ):      # plugins need to be loaded at the engine/scheduler level too      from vllm.plugins import load_general_plugins       load_general_plugins()       self.vllm_config = vllm_config      self.log_stats = log_stats       # Setup Model.      self.model_executor = executor_class(vllm_config)       # Setup KV Caches and update CacheConfig after profiling.      kv_cache_config = self._initialize_kv_caches(vllm_config)

Two things worth noticing before the constructor does anything else: it loads plugins, and it holds onto vllm_config as self.vllm_config — the same configuration object every eligibility check in this module ends up reading from. self.model_executor = executor_class(vllm_config) is the line that, several layers down, ends up constructing the model's attention layers — which is the actual moment a backend gets picked. Everything from here through Step 3 is answering what happens on the way to that line.

Plugins run first, and they can register a backend

load_general_plugins() is the first substantive statement in the constructor above. Here is its body, next to the plugin-discovery helper it calls:

# distilled — real names, reduced body: per-group logging and the# allowlist-vs-load-everything branch elidedDEFAULT_PLUGINS_GROUP = "vllm.general_plugins" # make sure one process only loads plugins onceplugins_loaded = False def load_plugins_by_group(group: str) -> dict[str, Callable[[], Any]]:  allowed_plugins = envs.VLLM_PLUGINS  discovered_plugins = entry_points(group=group)  plugins = {}  for plugin in discovered_plugins:      if allowed_plugins is None or plugin.name in allowed_plugins:          plugins[plugin.name] = plugin.load()  return plugins def load_general_plugins():  global plugins_loaded  if plugins_loaded:      return  plugins_loaded = True   plugins = load_plugins_by_group(group=DEFAULT_PLUGINS_GROUP)  for func in plugins.values():      func()

This is Python's own plugin mechanism — entry_points, the same standard-library machinery any installed package can hook into — scanning for anything registered under the vllm.general_plugins group and calling it. The plugins_loaded guard exists because this function runs in more than one OS process (the engine core and, when the multiprocessing executor is in play, every worker), and each process should only do this once.

Nothing here is specific to attention. But Step 5 comes back to this exact mechanism: a third-party backend registers itself through a general plugin, which means it has to exist before self.model_executor = executor_class(vllm_config) constructs anything — which is exactly what this ordering guarantees.

The Platform Layer

What is vLLM's platform layer?

vllm/platforms is vLLM's answer to "what hardware am I actually running on." Rather than scattering if is_cuda: ... elif is_rocm: ... checks through the engine, vLLM defines one Platform interface and a concrete subclass per accelerator family — CudaPlatformBase for NVIDIA GPUs, a ROCm equivalent for AMD's, and others for TPU and CPU. The rest of the codebase asks the current platform object a question and gets a hardware-appropriate answer back, the same way an OS driver interface lets an application call write() without knowing whether the destination is a disk or a network socket.

Attention backend selection is one of those questions. CudaPlatformBase.get_attn_backend_cls is what the H100 and the A100 in this module's running example are each asked; the MI300X, running ROCm, is asked the same conceptual question through ROCm's own version of this method — a separate implementation, not a shared one, because eligibility rules differ enough per accelerator family that vLLM gives each platform its own answer function rather than one that branches internally.

What the platform is asked

# distilled — real names, reduced body: the priority-sort branch's# block-size-precluded-a-higher-priority-backend warning is elidedclass CudaPlatformBase(Platform):  @classmethod  def get_attn_backend_cls(      cls,      selected_backend: AttentionBackendEnum | None,      attn_selector_config: AttentionSelectorConfig,      num_heads: int | None = None,  ) -> str:      device_capability = cls.get_device_capability()       # First try checking just the selected backend, if there is one.      if selected_backend is not None:          try:              backend_class = _get_attn_backend_class(selected_backend)              invalid_reasons = backend_class.validate_configuration(                  device_capability=device_capability,                  **attn_selector_config._asdict(),              )          except ImportError:              invalid_reasons = ["ImportError"]          if invalid_reasons:              raise ValueError(                  f"Selected backend {selected_backend} is not valid for "                  f"this configuration. Reason: {invalid_reasons}"              )          return _backend_cls_path(backend_class)       # No selected backend, so find a valid one.      valid_backends_priorities, all_invalid_reasons = cls.get_valid_backends(          device_capability=device_capability,          attn_selector_config=attn_selector_config,          num_heads=num_heads,      )      if len(valid_backends_priorities) == 0:          raise ValueError(f"No valid attention backend for {cls.device_name}.")       selected_candidate = min(          valid_backends_priorities, key=lambda candidate: candidate.priority      )      return _backend_cls_path(selected_candidate.backend_class)

Three inputs that matter, one output. selected_backend — did the deployment explicitly ask for one, or leave it unset? device_capability — the GPU's compute capability, a major.minor pair like 9.0 for an H100 or 8.0 for an A100, read straight off the device rather than passed in. And attn_selector_config, an AttentionSelectorConfig bundling the model-shape and launch-flag facts that matter for eligibility: head size, dtype, KV cache dtype, block size, whether the model uses MLA (multi-head latent attention — an architecture, popularized by DeepSeek's models, that compresses the KV cache into a smaller latent vector), whether sliding window is in play, and a handful of others (a fourth parameter, num_heads, only matters for a narrower MLA-specific priority case this module doesn't cover). The output is a single class path string naming the backend to use.

Read the branch structure and the design falls out: if the deployment explicitly requested a backend (--attention-backend FLASH_ATTN, say), this method tries only that one and raises if it fails — an explicit request is a hard requirement, not a preference. Otherwise it calls get_valid_backends, which runs every candidate through the same validation and returns whichever backends passed, each carrying a priority number. min(..., key=lambda candidate: candidate.priority) picks the best of what's valid. Either way, nothing here is a request-shaped input — no request id, no prompt, no batch. The only inputs are things that are fixed for the life of the deployment: the chip, and the launch configuration.

Why this belongs to the platform, not the model

Head size and dtype come from the model. Compute capability and the specific kernels available come from the chip. get_attn_backend_cls is where those two independent facts meet — which is exactly why it lives on the platform object rather than on the model or on the attention layer itself. The model doesn't know what hardware it will run on, and the hardware doesn't know what model will run on it; the platform layer is the one place both facts are visible at once.

The Model Runner module's own knowledge check makes a related point from the other side: GPUModelRunner's own configuration decides the attention backend, not the worker. Both framings agree that it's a configuration decision, not a per-layer or per-worker one — "the platform layer, asked once as attention layers are constructed" is simply naming the specific mechanism behind that configuration decision.

The Backend Registry

What is the attention backend registry?

AttentionBackendEnum is a name-to-class directory: more than 30 members, each one a string import path to a real backend class. FLASH_ATTN maps to "vllm.v1.attention.backends.flash_attn.FlashAttentionBackend", TRITON_ATTN maps to "vllm.v1.attention.backends.triton_attn.TritonAttentionBackend", and so on through MLA variants, sparse variants, and platform-specific ones for ROCm and XPU. Step 2's platform layer decides which name is eligible; the registry's job is turning a chosen name into the actual class.

# distilled — real names, reduced body: enum member list trimmed to the# backends this module's simulation covers; get_path's unused# include_classname parameter and its branch are elided togetherclass AttentionBackendEnum(Enum, metaclass=_AttentionBackendEnumMeta):  """Enumeration of all supported attention backends.   The enum value is the default class path, but this can be overridden  at runtime using register_backend().  """   FLASH_ATTN = "vllm.v1.attention.backends.flash_attn.FlashAttentionBackend"  TRITON_ATTN = "vllm.v1.attention.backends.triton_attn.TritonAttentionBackend"  ROCM_ATTN = "vllm.v1.attention.backends.rocm_attn.RocmAttentionBackend"  # Placeholder for third-party/custom backends - must be registered before use  CUSTOM = None   def get_path(self) -> str:      path = _ATTN_OVERRIDES.get(self, self.value)      if not path:          raise ValueError(              f"Backend {self.name} must be registered before use. "              f"Use register_backend(Backend.{self.name}, 'your.module.YourClass')"          )      return path   def get_class(self) -> "type[AttentionBackend]":      return resolve_obj_by_qualname(self.get_path())

get_class() is one line: resolve_obj_by_qualname(self.get_path()) — import the module named by that path string and pull the attribute out by name. That import is not a formality. It can fail, and when it does, it fails before anything else in this module's story gets a chance to run.

Import failure is a compatibility failure too

Look back at Step 2's excerpt: backend_class = _get_attn_backend_class(selected_backend), wrapped in a try/except ImportError. That call is backend.get_class() under a different name — the same method shown above. If the class the registry names can't be imported at all — its module requires an optional package that was never installed, say — get_class() raises ImportError before validate_configuration ever runs, so no compute-capability or head-size check happens either; the platform layer catches that exception and folds it into the same invalid_reasons list a failed validate_configuration check would produce. Step 6 walks a real instance of exactly this: a deployment that explicitly asks for a backend whose module can't be imported at all.

So the registry entry does two jobs at once: it names the class, and — by being an actual Python import rather than a string comparison — it is what lets "this backend can't even be loaded here" surface as a failure the platform layer can catch and fold into its own rejection list, the same list a genuinely incompatible-but-importable backend would land in.

The override path

AttentionBackendEnum's docstring mentions register_backend() for a reason: every entry above can be overridden at runtime, and CUSTOM exists specifically for backends that were never built into vLLM at all. Step 5 covers what registering one actually requires.

What a Backend Demands

What does an attention backend need from the model runner every step?

Picking a backend class is a one-time decision, but running it is not a one-time call. Every forward pass, the selected backend needs a fresh description of exactly what this pass's batch looks like: how many requests, how long each one's sequence is so far, and which physical KV-cache blocks belong to each — the same per-step facts the KV Cache Manager module handed off as block ids. That description is called CommonAttentionMetadata, and a metadata builder is the object that assembles it.

The block table and sequence lengths are different on essentially every pass — a token gets appended, a request finishes and frees its blocks, admission changes who's even in the batch — so this object cannot be computed once at startup and reused. It has to be rebuilt from the current step's numbers, every step.

The real call chain

Here is the exact path from the worker's per-step entry point down to a backend's own builder: GPUModelRunner.execute_model calls GPUModelRunner._build_attention_metadata, which calls the selected backend's builder — .build(...). That is the whole chain. It is worth being precise about it, because the two methods sit next to each other in the source in a way that invites a wrong guess: _build_attention_metadata is defined immediately after a much larger method, _prepare_inputs, ends — but _prepare_inputs never calls it. execute_model does, later in its own body, after _prepare_inputs has already returned.

# distilled — real names, reduced body: cudagraph-capture path, the# within-step update_block_table cache for hybrid KV-cache groups, the# per-group copy of common_attn_metadata, and speculative-decode/DCP# handling elided — the double loop below is simplified accordinglydef _build_attention_metadata(  self,  num_tokens: int,  num_reqs: int,  max_query_len: int,) -> tuple[PerLayerAttnMetadata, CommonAttentionMetadata | None]:  # Attention metadata is not needed for attention free models  if len(self.kv_cache_config.kv_cache_groups) == 0:      return {}, None   kv_cache_groups = self.kv_cache_config.kv_cache_groups  attn_metadata: PerLayerAttnMetadata = {}  cascade_attn_prefix_len = 0  # non-zero only when cascade attention applies  max_seq_len = self.optimistic_seq_lens_cpu.numpy()[:num_reqs].max().item()   cm_base = CommonAttentionMetadata(      query_start_loc=self.query_start_loc.gpu[: num_reqs + 1],      seq_lens=self.seq_lens[:num_reqs],      num_reqs=num_reqs,      num_actual_tokens=num_tokens,      max_query_len=max_query_len,      max_seq_len=max_seq_len,      block_table_tensor=block_table_gid_0,      slot_mapping=slot_mapping_gid_0,      causal=True,  )   def _build_attn_group_metadata(kv_cache_gid: int, attn_gid: int) -> None:      attn_group = self.attn_groups[kv_cache_gid][attn_gid]      builder = attn_group.get_metadata_builder(0)       attn_metadata_i = builder.build(          common_prefix_len=cascade_attn_prefix_len,          common_attn_metadata=cm_base,      )      for layer_name in attn_group.layer_names:          attn_metadata[layer_name] = attn_metadata_i   for kv_cache_gid, _ in enumerate(kv_cache_groups):      for attn_gid, _ in enumerate(self.attn_groups[kv_cache_gid]):          _build_attn_group_metadata(kv_cache_gid, attn_gid)   spec_decode_common_attn_metadata = None  # set under speculative decoding  return attn_metadata, spec_decode_common_attn_metadata

execute_model calls this once per forward pass, and it builds cm_base from this step's own tensors every time it runs — query_start_loc, seq_lens, block_table_tensor, slot_mapping are read fresh off the model runner's live state, not carried over from the previous call. Then, for each attention group, it calls the backend's own builder.build(...) with that fresh metadata.

What the concrete builder does with it

FlashAttentionMetadataBuilder.build is one concrete backend's implementation of that contract:

# distilled — real names, reduced body: AOT-schedule setup, decode-context-# parallel splitting, cascade-attention prefix/suffix bookkeeping, and# R-SWA/multimodal buffer copies elideddef build(  self,  common_prefix_len: int,  common_attn_metadata: CommonAttentionMetadata,  fast_build: bool = False,) -> FlashAttentionMetadata:  num_actual_tokens = common_attn_metadata.num_actual_tokens  max_query_len = common_attn_metadata.max_query_len  max_seq_len = common_attn_metadata.max_seq_len  query_start_loc = common_attn_metadata.query_start_loc  seq_lens = common_attn_metadata.seq_lens  block_table_tensor = common_attn_metadata.block_table_tensor  slot_mapping = common_attn_metadata.slot_mapping  causal = common_attn_metadata.causal   use_cascade = common_prefix_len > 0   attn_metadata = FlashAttentionMetadata(      num_actual_tokens=num_actual_tokens,      max_query_len=max_query_len,      query_start_loc=query_start_loc,      max_seq_len=max_seq_len,      seq_lens=seq_lens,      block_table=block_table_tensor,      slot_mapping=slot_mapping,      use_cascade=use_cascade,      common_prefix_len=common_prefix_len,      causal=causal,  )  return attn_metadata

Every field on the way in is read straight off common_attn_metadata — the object _build_attention_metadata just built from this step's live state — and every field on the way out is a FlashAttentionMetadata built from those same values plus one derived flag, use_cascade. Nothing here is computed from anything older than this step.

One nuance worth naming honestly: within a single step, if a model has more than one KV-cache group that shares the same builder type and spec — some hybrid architectures do — only the first group's call actually runs build(); the rest reuse that result through a separate method, update_block_table, that just swaps in a new block table. That reuse is scoped to this one step's multiple groups, not across steps: the dictionary it reuses from is created fresh at the top of every _build_attention_metadata call. So the fact that matters for this module still holds — nothing from one step's metadata survives into the next.

Adding a Backend

What does it take to add a new attention backend to vLLM?

Three things, and all three map directly onto the previous steps: a name in the registry, an implementation of the platform's eligibility questions, and a metadata builder. Nothing about "adding a backend" is request-shaped wiring — it is a one-time registration plus a startup-time contract, which is exactly what you'd expect once you know selection itself is a startup-time decision.

1. A name

Every built-in backend is a member of AttentionBackendEnum — FLASH_ATTN, TRITON_ATTN, ROCM_ATTN, and roughly thirty others. Out-of-tree code doesn't get to add a member to that enum, so vLLM reserves one, CUSTOM, specifically for it, paired with a register_backend() function:

@register_backend(AttentionBackendEnum.CUSTOM)
class MyCustomBackend:
    ...

The decorator stores the class's own module path under that enum member, in the same override table AttentionBackendEnum.get_path() checks before falling back to the built-in value. After registration, AttentionBackendEnum.CUSTOM.get_class() — Step 3's method, unmodified — resolves to the new class exactly the way it resolves FLASH_ATTN to FlashAttentionBackend. This is also why Step 1's plugin-loading order matters: a plugin that calls register_backend() has to run before anything tries to construct an attention layer, and load_general_plugins() runs first in EngineCore.__init__ specifically so that ordering holds.

2. The eligibility contract

CudaPlatformBase.get_attn_backend_cls (Step 2) doesn't know anything about any specific backend's kernels — it only knows how to ask a handful of questions and read the answers. A new AttentionBackend subclass has to answer them:

None of these are called individually by the platform layer at selection time. They're aggregated by one method, validate_configuration, which calls each predicate in turn and collects every failure into a list of reasons — the same invalid_reasons list Step 2 raises ValueError with, and the same list an ImportError on an unbuildable backend gets folded into. A new backend gets exactly the rejection behavior every existing one gets, for free, just by implementing these methods honestly.

3. A metadata builder

Step 4 already showed the shape: a subclass of AttentionMetadataBuilder implementing build(common_prefix_len, common_attn_metadata) -> M, returning whatever metadata type that backend's kernel expects. get_builder_cls() from the contract above is what determines which builder class gets constructed for this backend when its attention group is set up — FlashAttentionBackend.get_builder_cls() returns FlashAttentionMetadataBuilder, the exact class Step 4 walked, and it's that already-constructed instance _build_attention_metadata retrieves and calls every step.

How It Fails

What happens when you request a backend that was never installed?

The simulation's fifth stage sets this up concretely: the H100 is explicitly asked to run FLASHINFER — a real, CUDA-eligible backend — in a deployment where the optional flashinfer package was never installed. Nothing about that request waits for a real inference call to go wrong. The engine rejects it before it finishes starting, and the other two machines — which never had an explicit override to fail, and simply landed on whatever their own platform and registry resolved to — keep serving normally.

Why an explicit request fails differently than no request at all

Re-read the branch structure from Step 2: when a deployment does not specify --attention-backend, get_attn_backend_cls tries every candidate and returns the best one that passes. But when a deployment does specify one, there is no fallback — that single choice either passes or the whole engine fails to start.

# distilled — real names, reduced body: the explicit-selection path only;# the no-selection priority scan is Step 2's excerptclass CudaPlatformBase(Platform):  @classmethod  def get_attn_backend_cls(      cls,      selected_backend: AttentionBackendEnum | None,      attn_selector_config: AttentionSelectorConfig,      num_heads: int | None = None,  ) -> str:      device_capability = cls.get_device_capability()       if selected_backend is not None:          try:              backend_class = _get_attn_backend_class(selected_backend)              invalid_reasons = backend_class.validate_configuration(                  device_capability=device_capability,                  **attn_selector_config._asdict(),              )          except ImportError:              invalid_reasons = ["ImportError"]          if invalid_reasons:              raise ValueError(                  f"Selected backend {selected_backend} is not valid for "                  f"this configuration. Reason: {invalid_reasons}"              )          return _backend_cls_path(backend_class)

This is the exact same CudaPlatformBase method Step 2 introduced — the H100 is a CUDA machine, so this is precisely what runs for it here, no hedging required. Read the branch: try to resolve the requested backend, catch an ImportError if the class can't even be imported, and raise before returning anything.

The mechanism, traced

AttentionBackendEnum.FLASHINFER.get_class() — Step 3's method — tries to import vllm.v1.attention.backends.flashinfer.FlashInferBackend. That module's own source does from flashinfer import (...) as an ordinary, unconditional top-level import — no try/except, no availability check — because FlashInfer is a separate, optional package vLLM doesn't bundle. If it was never installed on this machine, that import fails the moment Python tries to load the module: before validate_configuration runs, before compute capability or head size are ever checked. The except ImportError branch above catches exactly that and turns it into invalid_reasons = ["ImportError"]. Since selected_backend was explicit, there is no fallback candidate to try next — invalid_reasons is non-empty, so the method raises ValueError and the engine never finishes constructing.

Compare that to what a failure discovered mid-request would look like: a request arrives, the scheduler admits it, KV blocks get allocated, and only then does something discover the kernel can't run — after work has already been spent, with a client waiting on a connection that has to be torn down. Failing at startup instead means the operator who typed the bad flag sees the error immediately, on their own terminal — the process that raised never gets far enough to construct a scheduler, let alone admit a request.