vLLM's KV Connector Interface — Scheduler-Side, Worker-Side and the Factory, in Source

What You Know

What do you already know about moving KV cache from prefill to decode?

If you've done Step 4 of Prefill/Decode Disaggregation in the LLM Serving track, you already have the concept this module builds on: once prefill and decode run on separate GPU pools, the KV cache one of them computed has to physically move to the other before decode can use it, that move costs real time, and the cost depends on the interconnect — PCIe, InfiniBand, or NVLink — plus mitigations like chunked transfer and placement-aware scheduling. This module doesn't re-teach any of that. If it's unfamiliar, go read that step first — everything below assumes it.

What that module didn't show you is vLLM's own code: the interface that turns "the KV cache has to move" from a concept into something a scheduler can call and a worker can implement, without either of them caring whether the bytes travel over a network, a local disk, or a vendor's own transfer library.

The delta, in one sentence

You already know KV cache moves from a prefill instance to a decode instance, and that the move costs time. What you don't yet know is that vLLM doesn't hardcode how — it defines one abstract base class, KVConnectorBase_V1, splits its methods into a scheduler-side half and a worker-side half, and lets a config string pick which concrete implementation runs underneath both halves. Every request afterward just calls that interface; the scheduler never has to know whether get_num_new_matched_tokens is checking a local folder (this module's example) or querying a remote KV store over RDMA.

Why this is the module to read most skeptically

Every other module in this track cites code that has been stable inference-engine plumbing for a while — a scheduler loop, a block pool, a sampler. Connectors are the opposite: KV transfer between disaggregated instances is one of the fastest-moving surfaces in vLLM right now, with new connector implementations landing and existing ones being reshaped release to release. So this module draws a hard line between two kinds of claim:

• The interface's shape — one ABC, split into a scheduler-side half and a worker-side half, selected by name through a factory — is the durable thing. It is what every step below teaches as a fact you can rely on.
• Any specific method's signature, or ExampleConnector's specific behavior, is an illustration at the pinned v0.26.0 ref, not a contract. Expect it to keep moving upstream, and treat every code excerpt below as "this is what it looked like when we checked," not "this is what it will always look like."

What the next five steps cover

Steps 3 and 5 happen once, at startup, before any request arrives — a connector is chosen and constructed from config, not re-decided per request. Steps 2, 4 and 6 are about the already-constructed connector being called on the request path.

Two Sides

What are the two halves of the KVConnector interface?

One abstract base class, KVConnectorBase_V1, and its methods split into two groups by who calls them: a scheduler-side group, called from the scheduler role while it's deciding what to run this step, and a worker-side group, called from the worker role while the forward pass is being prepared and executed. The source marks the split literally — the class body has a comment reading # Scheduler-side methods above one set of methods and # Worker-side methods above the other. This step reads one method from each side, plus the concrete implementation that fills each one in. (Step 3 covers who constructs each side's object and where — that's a separate question from which methods belong to which side, and this step is only the second one.)

The file that looks like the real one, but isn't

Before any of that: if you go looking for KVConnectorBase_V1 yourself, there are two files that could plausibly hold it, and only one does. vllm/distributed/kv_transfer/kv_connector/base.py — no v1 in the path — looks like the natural place. Below the license header, this is the whole file:

# vllm/distributed/kv_transfer/kv_connector/base.py — everything
# below the SPDX license header
"""Defines the base type for KV cache connectors."""

from vllm.distributed.kv_transfer.kv_connector.v1 import KVConnectorBase_V1

KVConnectorBase = KVConnectorBase_V1
KVConnectorBaseType = KVConnectorBase_V1

__all__ = ["KVConnectorBase", "KVConnectorBaseType"]

A module docstring, a two-line re-export, and an __all__ — no class body. Every method this step cites lives one directory down, at vllm/distributed/kv_transfer/kv_connector/v1/base.py. And a path that would seem even more natural for a "v1 core" component — vllm/v1/core/kv_connector/ — does not exist at all, even though every other v1 subsystem this track has opened (the scheduler, the KV cache manager) does live under vllm/v1/core/. Connectors are namespaced under vllm/distributed/kv_transfer/, not vllm/v1/.

Scheduler-side: does the external cache have anything?

# distilled — real names, docstring condensed to the return contractclass KVConnectorBase_V1(ABC):  # ==============================  # Scheduler-side methods  # ==============================   @abstractmethod  def get_num_new_matched_tokens(      self,      request: "Request",      num_computed_tokens: int,  ) -> tuple[int | None, bool]:      """      Returns:          - An optional count of additional tokens loadable from the            external KV cache beyond num_computed_tokens. If None, the            connector needs more time; the scheduler should ask again            on a later step.          - True if those tokens will load asynchronously, between            scheduler steps. Must be False if the first element is 0.      """      pass

# distilled — real names, NOTE-comment noise removedclass ExampleConnector(KVConnectorBase_V1):  def get_num_new_matched_tokens(      self,      request: "Request",      num_computed_tokens: int,  ) -> tuple[int | None, bool]:      if not self._found_match_for_request(request):          return 0, False       logger.info("External Cache Hit!")       token_ids = request.prompt_token_ids or []      num_tokens_to_check = align_to_block_size(          len(token_ids) - 1, self._block_size      )      return num_tokens_to_check - num_computed_tokens, False

Two things worth being exact about. First, the return type is tuple[int | None, bool] — the second element is not a hit/miss flag. It's is_async: whether the matched tokens will be loaded asynchronously, between scheduler steps, rather than being ready immediately. A connector can report a real match (first element > 0) and still set the second element False, meaning "the load is fast enough that the scheduler doesn't need to treat this specially." ExampleConnector — this disk-backed debug implementation — always returns False here, on both its hit and its miss branch; it never reports an async load. That's a property of this one implementation, not of the interface: a connector moving bytes over a network is exactly the kind that would legitimately return True.

Second, the comment opening ExampleConnector's class body — the first thing written inside the class, though the file itself opens with the usual license header and imports first — describes it plainly: "This is Simple debug implementation of the KV connector. It save / load the KV cache to / from the disk." It illustrates the interface's shape with a local, disk-backed stand-in — a real production connector (registered in Step 3's factory alongside this one) moves KV between two live GPU instances over a network, not to and from /tmp.

Worker-side: load what the lookup found

# distilled — real names, docstring condensed to the timing contractclass KVConnectorBase_V1(ABC):  # ==============================  # Worker-side methods  # ==============================   @abstractmethod  def start_load_kv(      self, forward_context: "ForwardContext", **kwargs: Any  ) -> None:      """      Start loading the KV cache from the connector into vLLM's paged      KV buffer. Called from the forward context before the forward      pass, to enable async loading during model execution.      """      pass

# distilled — real names, reduced body: the nested inject_kv_into_layer# helper's own body (both its MLA and plain block/offset branches) is# omitted entirely — only its call site remainsclass ExampleConnector(KVConnectorBase_V1):  def start_load_kv(      self, forward_context: "ForwardContext", **kwargs: Any  ) -> None:      metadata = self._get_connector_metadata()      attn_metadata = forward_context.attn_metadata      if attn_metadata is None:          return       for request in metadata.requests:          if request.is_store:              continue          for layer_name in forward_context.no_compile_layers:              layer = forward_context.no_compile_layers[layer_name]              kv_cache_layer = getattr(layer, "kv_cache", None)              if kv_cache_layer is None:                  continue              filename = self._generate_filename_debug(                  layer_name, request.token_ids, request.mm_hashes              )              kv_cache = safetensors.torch.load_file(                  filename, device=str(kv_cache_layer.device)              )["kv_cache"]              if isinstance(attn_metadata, dict):                  # writes kv_cache into kv_cache_layer at                  # request.slot_mapping — the paged-buffer injection                  inject_kv_into_layer(kv_cache_layer, kv_cache,                      request.slot_mapping, attn_metadata[layer_name])

start_load_kv's own docstring says exactly when it runs: "called from the forward context before the forward pass to enable async loading during model execution." That's worker-role, GPU-adjacent timing — the opposite end of the pipeline from get_num_new_matched_tokens, which runs under the scheduler role, on the CPU, before any block has moved.

Whether "the scheduler role" and "the worker role" land in two different operating-system processes or the same one is a separate question — Step 3 answers it, and the answer depends on the executor, not on the connector.

The Factory

How does a connector name in your config become a running object?

--kv-transfer-config sets a string field, kv_connector, on KVTransferConfig. At startup, that string is looked up in a registry that's populated once, at import time, by calls to KVConnectorFactory.register_connector sitting at the bottom of factory.py — one call per shipped connector, each naming a module path and a class name to lazily import. No registry file, no decorator scanning: the registration calls are just ordinary Python statements that run when factory.py is imported.

# distilled — real names, reduced body: the HMA-support check and the# external-module-path branch (an escape hatch for out-of-tree# connectors) elidedclass KVConnectorFactory:  _registry: dict[str, Callable[[], type[KVConnectorBase]]] = {}   @classmethod  def register_connector(cls, name: str, module_path: str, class_name: str) -> None:      def loader() -> type[KVConnectorBase]:          module = importlib.import_module(module_path)          return getattr(module, class_name)      cls._registry[name] = loader   @classmethod  def create_connector(      cls,      config: "VllmConfig",      role: KVConnectorRole,      kv_cache_config: "KVCacheConfig",  ) -> KVConnectorBase:      kv_transfer_config = config.kv_transfer_config      connector_cls = cls.get_connector_class(kv_transfer_config)      # NOTE(Kuntai): v1 connector is explicitly separated into two      # roles. Scheduler connector: co-located with the scheduler      # process. Worker connector: co-located with the worker      # process. We build separately to enforce strict separation      return connector_cls(config, role, kv_cache_config)   @classmethod  def get_connector_class(cls, kv_transfer_config) -> type[KVConnectorBaseType]:      connector_name = kv_transfer_config.kv_connector      if connector_name in cls._registry:          return cls._registry[connector_name]()      raise ValueError(f"Unsupported connector type: {connector_name}") # Registered once, at import time — a name mapped to a module and class:KVConnectorFactory.register_connector(  "ExampleConnector",  "vllm.distributed.kv_transfer.kv_connector.v1.example_connector",  "ExampleConnector",)KVConnectorFactory.register_connector(  "NixlConnector",  "vllm.distributed.kv_transfer.kv_connector.v1.nixl",  "NixlConnector",)

Read past the lookup and the interesting part is the comment sitting directly above the object it returns: two roles, built separately. create_connector takes a role: KVConnectorRole argument — an enum with exactly two values, SCHEDULER and WORKER — and it's called once from inside the scheduler process (role SCHEDULER) and once from inside every worker process (role WORKER), each time with the same kv_connector name from config. A --tensor-parallel-size 4 engine ends up with five instances of the same class: one SCHEDULER-role object and four separate WORKER-role objects, one per worker process — not a pair, a fan-out. Neither instance is "the" connector; the connector, as a running thing, is that whole set.

That's true whenever the scheduler and the workers are genuinely separate operating-system processes — the mp executor, world size greater than one. What vllm serve Actually Spawns already established that under the single-process (uni) executor there is no separate worker process at all: the model runs inside the same process as the scheduler. create_connector still gets called twice in that case — once per role — but both calls land in the same process. The two-roles split is universal; two separate processes is not.

That's also why Step 2's two method groups line up with this so cleanly: the scheduler-side methods only ever run on the instance built with role=SCHEDULER, and the worker-side methods only ever run on the instance built with role=WORKER. A single class implements both halves, but no single object ever executes both — which one runs is decided entirely by which role it was constructed with, never by which process it happens to share.

This whole exchange — read the config, resolve a class, build it — happens once per role, at startup, before this engine has accepted a single request. It's the same startup lane as the executor and attention-backend selection earlier in this track: a config-plus-hardware decision, made once, never reconsidered per request.

One more field worth naming here without a code excerpt: kv_connector_module_path. When set, get_connector_class imports a connector from that path instead of the built-in registry — the escape hatch that lets an out-of-tree connector plug into this exact interface without a single line changing in vLLM's own factory.py. That's Step 1's "pluggable" claim, concretely: the registry above ships with vLLM, but it isn't the only way in.

Lookup and Save

Where does the scheduler consult a connector, and where does it tell the connector to persist?

Inside the same Scheduler.schedule method the KV Cache Manager module already walked you through the allocation side of. A connector doesn't get its own separate code path — it's two extra calls inline, inside the admission loop you've already partly read, plus one more call after the loop is done for the step. The first of those calls isn't guaranteed to happen only once: get_num_new_matched_tokens's own docstring says it may be called more than once for the same request and must be side-effect-free — which is exactly what makes the None-and-retry branch below safe to loop on.

The lookup

Right after the scheduler asks its own local prefix cache what it already has, and before it asks for any blocks, it asks the connector the same question about the external cache:

# vllm/v1/core/sched/scheduler.py — Scheduler.schedule, the connector lookup
if self.connector is not None:
    ext_tokens, load_kv_async = (
        self.connector.get_num_new_matched_tokens(
            request, num_new_local_computed_tokens
        )
    )
    if ext_tokens is None:
        # the connector needs more time — try this request again later
        request_queue.pop_request()
        step_skipped_waiting.prepend_request(request)
        continue
    num_external_computed_tokens = ext_tokens

num_new_local_computed_tokens is the local prefix-cache hit length — the same number Longest Match computed. The connector is only ever asked about the remainder: tokens the local cache didn't already have. Its answer, num_external_computed_tokens, gets added to that local number to form num_computed_tokens — the total the rest of scheduling reasons about, regardless of which of the two caches actually supplied it.

The None branch is worth a beat of its own: it isn't a miss. It means the connector genuinely doesn't know yet — a real production connector querying a remote store asynchronously might not have an answer this instant — so the scheduler requeues the request and asks again on a later step, rather than treating "no answer yet" as "no match."

Allocation doesn't get a separate rule for connector tokens

num_external_computed_tokens flows straight into KVCacheManager.allocate_slots as one more argument:

# vllm/v1/core/kv_cache_manager.py — KVCacheManager.allocate_slots, the
# connector-relevant arguments only
def allocate_slots(
    self,
    request: Request,
    num_new_tokens: int,
    num_external_computed_tokens: int = 0,  # from the lookup above
    delay_cache_blocks: bool = False,       # True when load_kv_async is True
    reserved_blocks: int = 0,               # connector-only, when load_kv_async
    ...
) -> KVCacheBlocks | None:
    total_computed_tokens = min(
        num_local_computed_tokens + num_external_computed_tokens,
        self.max_model_len,
    )
    ...
    available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks
    if required_blocks > available_blocks:
        return None

That total feeds the same capacity question Will It Fit? already taught: required_blocks computed from the total, compared against free blocks, None returned if it doesn't fit. A connector's promised tokens don't bypass that gate or get a separate one — blocks for externally-matched tokens are counted by the same required_blocks > available_blocks check every other block request goes through. But the check isn't always run with the same numbers. When load_kv_async is true, Scheduler.schedule also computes a connector-only reserved_blocks value — self._inflight_prefill_reserved_blocks() — and passes it through to allocate_slots, which subtracts it from the free count before comparing: available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks. An async connector load is admitted against a stricter bar than an ordinary request, precisely so it can't consume blocks an already-prefilling request is relying on. delay_cache_blocks — the other connector-relevant argument, also set to load_kv_async — is the separate piece that skips caching the newly allocated blocks immediately, because their content hasn't arrived yet; the comment in the source calls this out directly as a P/D concern: skip caching while a remote receive is still in flight.

Told where its blocks are: after allocation, not before

Once allocate_slots returns real blocks, the scheduler calls back into the connector a second time — this is the "save" half of this step's name, in the sense of committing the allocation the connector can now act on:

# vllm/v1/core/sched/scheduler.py — Scheduler.schedule, after allocation
if self.connector is not None:
    self.connector.update_state_after_alloc(
        request,
        self.kv_cache_manager.get_blocks(request_id),
        num_external_computed_tokens,
    )

update_state_after_alloc is a real scheduler-side method on KVConnectorBase_V1 — not one of this module's five cited symbols, so no distilled excerpt of it appears here, but its job is exactly what its name says: hand the connector the block ids the cache manager just committed to this request, so the connector knows where to put the load it promised. Its own docstring is explicit that this can fire twice for one request — once when the buffer for an async load is first allocated, again once the transfer has actually landed — and that the decision to load should be driven by num_external_computed_tokens, not by whether blocks happens to be non-empty.

Then the worker actually moves bytes

Everything above runs under the scheduler role, on the CPU, before a single kernel launches. Once the loop over all requests for this step is done, Scheduler.schedule calls one more connector method — build_connector_meta — which asks the connector to package up whatever it needs (which request, which blocks) into a KVConnectorMetadata object, stored on scheduler_output.kv_connector_metadata. That field is the bridge to the worker role — whether the worker role is running in a separate process or, under a single-process executor, sharing the scheduler's own process — and that's where Step 2's other half picks up: start_load_kv runs from the forward context, right before the forward pass, and writes whatever was matched into the paged KV buffer using the block ids that metadata carried across.

There's a mirror-image method on the producing side that this module doesn't cite either: save_kv_layer, called from inside the attention layer on the prefill instance's worker, pushes newly computed KV out to the connector as each layer finishes — the write half of the same interface start_load_kv reads from. Naming it here completes the picture — lookup, allocate, tell the connector where the blocks are, load on one side, save on the other — even though only the load half was one of this module's cited symbols.

Wiring a P/D Pair

How are a prefill instance and a decode instance wired together, as configuration?

By one config object, KVTransferConfig, populated from --kv-transfer-config and set independently on each vllm serve launch. There's no separate "pairing" step — a prefill instance and a decode instance are just two engines, each with its own KVTransferConfig, that happen to agree on the same connector name and complementary roles. That agreement configures the endpoints; a real deployment also needs something in front of them — a router or proxy that decides which prefill instance and which decode instance a given request goes to, and forwards the transfer parameters between them. That routing layer is a separate vLLM component from anything this module cites, and this step doesn't cover it.

# vllm/config/kv_transfer.py — KVTransferConfig, the properties an engine
# uses to ask what role it's playing
KVProducer = Literal["kv_producer", "kv_both"]
KVConsumer = Literal["kv_consumer", "kv_both"]

@property
def is_kv_producer(self) -> bool:
    return self.kv_connector is not None and self.kv_role in get_args(KVProducer)

@property
def is_kv_consumer(self) -> bool:
    return self.kv_connector is not None and self.kv_role in get_args(KVConsumer)

A prefill instance is launched with kv_role="kv_producer" — it computes KV and pushes it out. A decode instance is launched with kv_role="kv_consumer" — it pulls KV in and only computes locally what wasn't supplied. "kv_both" exists for a connector that does both in the same engine (an offloading connector spilling its own local cache to a second tier is the natural case, not classic P/D).

Two different things both called "role," and they answer different questions

It's easy to conflate this step's kv_role with Step 3's KVConnectorRole, because both are literally named "role" and both show up in the same sentence about connectors. They're independent axes:

So a single decode instance, at this ref, ends up with: one config saying kv_role="kv_consumer", one connector object built with role=SCHEDULER, and one connector object per worker built with role=WORKER — sharing a process with the scheduler role or not, depending on the executor (Step 3). Every one of those pieces agrees on the same kv_connector name, because that name is the one thing every piece read out of the same KVTransferConfig.

How It Fails

What happens when a connector reports nothing to load?

The specimen: a decode instance's connector is asked, via get_num_new_matched_tokens, about a prompt whose KV cache the prefill side never produced — or produced and already evicted. ExampleConnector's own check, _found_match_for_request, looks for a folder on disk keyed by a hash of the prompt tokens; it isn't there. The connector reports zero matched tokens.

The reasonable-sounding guess is that this is an error condition — a failed lookup, something that should raise, log a warning, or route the request down some dedicated fallback path. Tracing the source says otherwise.

Tracing what actually happens

get_num_new_matched_tokens's own contract, from Step 2, requires the second element of its return tuple to be False whenever the first is 0 — an async load can't be "in progress" for zero tokens. ExampleConnector returns exactly that on a miss:

# vllm/distributed/kv_transfer/kv_connector/v1/example_connector.py —
# ExampleConnector.get_num_new_matched_tokens
if not self._found_match_for_request(request):
    return 0, False

Back in Scheduler.schedule, that (0, False) becomes num_external_computed_tokens = 0 and load_kv_async = False. Follow those two variables into the branch that decides how many tokens to schedule this pass:

# vllm/v1/core/sched/scheduler.py — Scheduler.schedule, right after the
# connector lookup (Step 4 traced the two calls before this branch)
if load_kv_async:
    # KVTransfer: loading remote KV, do not allocate for new work.
    assert num_external_computed_tokens > 0
    num_new_tokens = 0
elif defer_prefills and num_computed_tokens < request.num_tokens - 1:
    # defer_prefills is a DP-load-balancing concern, unrelated to
    # this connector — not this specimen's branch either
    break
else:
    # Number of tokens to be scheduled.
    num_new_tokens = request.num_tokens - num_computed_tokens

load_kv_async is False, so the first branch is skipped outright — there's no code path here that even checks whether the miss was "expected." The middle branch, defer_prefills, is an unrelated data-parallel load-balancing concern this specimen doesn't trigger either. Control falls to the plain else: num_new_tokens = request.num_tokens - num_computed_tokens. That line does not know, and does not ask, whether a connector is even configured. It is the exact same arithmetic a connector-less deployment runs on every fresh prompt. The request proceeds through ordinary local prefill — the same allocate_slots call Step 4 traced, the same capacity check the KV Cache Manager module already taught, just with num_external_computed_tokens sitting at 0 instead of some positive number.

The part worth naming explicitly: nothing here raises

Read both snippets again and there is no raise, no logger.warning, no branch labeled "miss" or "fallback." A miss reaches the same scheduling arithmetic a fresh, connector-less request already takes — the same request-level flow this track's own KV Cache Manager module walked through before a KV connector was even part of the picture. There is nothing here for a stack trace to point at, because nothing goes wrong at the level this code operates at: a connector answering "zero" is a legitimate, ordinary answer, handled by the same arithmetic that handles "there is no connector."

That doesn't mean nothing can go wrong with a connector — it means this particular specimen isn't that. A connector can promise tokens (get_num_new_matched_tokens returns a positive first element with is_async=True) and then fail to deliver them before the load completes — a genuine transfer failure, on a different timeline than the one this step traces. That case is governed by a separate config field, kv_load_failure_policy, which Step 5 named and is "fail" by default: the request is immediately failed with an error finish reason, unless an operator has explicitly set "recompute" to reschedule it instead. A zero-token miss never reaches that policy at all — it never made a promise to break.

What you can now say about a miss

A KV connector reporting zero matched tokens is not a special case this codebase treats specially. It's the ordinary "nothing to reuse" branch, reached by a request that happens to have a connector configured, running the identical scheduling arithmetic a request without one would run. A connector's presence changes what's possible — it adds a second source of already-computed tokens alongside the local prefix cache — but a miss on that second source doesn't corrupt any state or trigger recovery logic; the request simply pays the same local compute a no-connector deployment always pays for an uncached prompt. That's still a real cost — recomputing a prompt is exactly the work Prefill/Decode Disaggregation's KV transfer exists to avoid — just not a cost this code path treats as an error.