vLLM's KV Cache Manager — Four Layers, in Source

Four Layers, Not One

What are the four layers of vLLM's KV cache manager?

KVCacheManager, KVCacheCoordinator, SingleTypeKVCacheManager, BlockPool — four classes in four files, and a request's blocks pass through all of them on the way in. Only the last one owns any blocks. The three above it own, in order: the policy that decides whether to ask, the routing that decides which pool client asks, and the arithmetic that turns a token count into a block count.

You already know both of those. This module does not re-explain why paging beats a contiguous reservation, or why chaining the hashes makes a prefix lookup cheap. It shows you where those two ideas are written — which object holds the free list, which method computes the hit length, which if returns the failure — because that is the part you need in order to read a stack trace or change a line.

The one idea to carry in: the concept modules draw one grid of blocks and one free list. The code has four layers, and almost everything surprising about it happens at a seam — the moment a call crosses from one layer to the next and the arguments, or the return type, or the failure representation, change shape.

The four layers, and what each one owns

A KV cache group is the unit layer 2 fans out over: one group per distinct attention type in the model. A plain full-attention model has one group and therefore one layer-3 manager. A model that mixes full attention with sliding-window layers has two, each with its own block size and its own opinion about how many blocks a given token count needs.

Which is why "the block size" is three numbers in this code, and the constructor below names all three. The group's block size is how many tokens one of that group's physical blocks holds. The hash block size is the granularity the prefix-cache hashes were computed at. The scheduler block size is the least common multiple of every group's, so that one scheduling decision lands on a boundary all groups agree on — the constructor asserts it divides evenly by both of the others. On a single-group full-attention engine all three are the same number, and you can read the rest of this module as if there were only one. On a hybrid model you cannot.

One body builds three of the four

# distilled — real names, reduced body: only the construction of the layers belowclass KVCacheCoordinator(ABC):  def __init__(      self,      kv_cache_config: KVCacheConfig,      enable_caching: bool,      scheduler_block_size: int,      hash_block_size: int,  ):      self.kv_cache_config = kv_cache_config      self.enable_caching = enable_caching      assert scheduler_block_size % hash_block_size == 0 and all(          scheduler_block_size % g.kv_cache_spec.block_size == 0          for g in kv_cache_config.kv_cache_groups      )      self.scheduler_block_size = scheduler_block_size       self.block_pool = BlockPool(          num_gpu_blocks=kv_cache_config.num_blocks,          enable_caching=enable_caching,          hash_block_size=hash_block_size,      )       self.single_type_managers = tuple(          get_manager_for_kv_cache_spec(              kv_cache_spec=kv_cache_group.kv_cache_spec,              block_pool=self.block_pool,              enable_caching=enable_caching,              kv_cache_group_id=i,              scheduler_block_size=self.scheduler_block_size,          )          for i, kv_cache_group in enumerate(self.kv_cache_config.kv_cache_groups)      )

Three facts, and each one is a claim about shape rather than about paging.

Layer 4 is built before layer 3, and then passed into it. self.block_pool = BlockPool(...) runs first; every manager in the tuple below receives block_pool=self.block_pool. So the chain is not a chain of ownership. Layer 3 does not contain layer 4 — it holds a reference to the same single pool every other layer-3 manager holds. Which means the free queue is not partitioned per group. Two managers with different block sizes compete for the same blocks.

self.single_type_managers is a tuple, and its length is the number of KV cache groups. That tuple is the fan-out. Every downward call in the rest of this module is either a loop over it or an index into it, and which of those two you are looking at is the fastest way to tell a generic path from a special-cased one.

Layer 1 keeps a shortcut to layer 4. KVCacheManager builds its coordinator, and immediately after, takes self.coordinator.block_pool and files it under its own self.block_pool. That alias is not decoration — it is how layer 1 asks the free-block count directly in the capacity check of step 4, without going through layer 2. So the four layers are a call chain, not a strict hierarchy: one arrow skips a level, and it is the arrow that decides whether an allocation happens at all.

"The coordinator" is three classes

# distilled — real names, reduced body: the three-way selection, arguments elideddef get_kv_cache_coordinator(  kv_cache_config: KVCacheConfig,  enable_caching: bool,) -> KVCacheCoordinator:  if not enable_caching:      return KVCacheCoordinatorNoPrefixCache(kv_cache_config)  if len(kv_cache_config.kv_cache_groups) == 1:      return UnitaryKVCacheCoordinator(kv_cache_config)  return HybridKVCacheCoordinator(kv_cache_config)

KVCacheManager never names a coordinator class. It calls get_kv_cache_coordinator once and stores whatever comes back, and the function is a plain three-way selection on config — no registry, no plugin lookup, no subclass discovery.

Every remaining step follows the middle row — one full-attention KV cache group, caching on, which is what a normal single-GPU model gets. That qualification is doing real work, not hedging: the base class's own find_longest_cache_hit is an @abstractmethod whose body is the single word pass, so there is no such thing as "what the coordinator does" for the cache-hit path. There is only what a particular subclass does, and the hybrid one's version of that method is a different and much longer body. When you go looking in a real checkout, check which class you landed in first.

Why the seams are where the surprises are

Read the four-layer list again and notice that each layer knows something the layer below it does not, and none of that knowledge flows downward automatically. Layer 1 knows about the watermark; layer 4 has never heard of it. Layer 2 knows which cache spec belongs to which group; layer 3's cache-hit method is a @classmethod and holds no instance state at all, so layer 2 has to hand it the spec on every call. Layer 3 knows how many blocks a request already has; layer 1 only knows how many it asked for.

That asymmetry is what makes the next five steps worth reading one at a time. The cache-hit lookup (steps 2 and 3) crosses two seams downward and stops before the pool. The allocation (steps 4 and 5) crosses all three. And the failure (step 6) is the same shortage of blocks reported two different ways at two different depths, because the two layers that notice it know different things.

Asking for a Cache Hit

What happens before vLLM allocates a KV block?

A read. When the scheduler is admitting a request — pulling it off a waiting queue for the first time, or readmitting one it preempted — it calls KVCacheManager.get_computed_blocks before it calls anything that allocates. That method frees nothing, allocates nothing, and mutates nothing in the pool. It asks one question, how much of this request's prefix is already resident, and returns the answer as a number.

The qualifier matters, and in the scheduler it is one condition: the lookup is guarded by if request.num_computed_tokens == 0. A request already in self.running has computed tokens and no new prefix to discover, so the running loop goes straight to allocation and never makes this call at all. A preempted request passes the guard because preemption reset that counter to zero — which is why, as module 2 put it, request.num_computed_tokens = 0 is the line that costs money: it also buys a fresh trip through this method.

The concept module tells you cross-request reuse exists. What it cannot tell you is that reuse and allocation are two calls in a fixed order, that the first one is read-only, and that the number they pass between them is the same number you met one module ago on the right-hand side of a subtraction.

Where this number lands

Module 2's admission loop computed num_new_tokens as request.num_tokens - num_computed_tokens, and the local on the right was described there as "the number of its prompt tokens the KV cache already holds". This method is where that local comes from. A prefix-cache hit does not reach the scheduler as an object, a flag, or an event — it reaches it as one smaller integer.

The read-only half

# distilled — real names, reduced body: the early exits and the one downward callclass KVCacheManager:  def get_computed_blocks(self, request: Request) -> tuple[KVCacheBlocks, int, int]:      if not self.enable_caching or request.skip_reading_prefix_cache:          return self.empty_kv_cache_blocks, 0, 0       max_cache_hit_length = request.num_tokens - 1      computed_blocks, num_new_computed_tokens, num_uncached = (          self.coordinator.find_longest_cache_hit(              request.block_hashes, max_cache_hit_length          )      )       shared_prefix_boundary = (          num_new_computed_tokens + num_uncached if num_uncached else 0      )       if self.log_stats:          self.prefix_cache_stats.record(              num_tokens=request.num_tokens,              num_hits=num_new_computed_tokens,              preempted=request.num_preemptions > 0,          )       blocks = self.create_kv_cache_blocks(computed_blocks)      return blocks, num_new_computed_tokens, shared_prefix_boundary

Four things in that body, in the order the code reaches them.

Two conditions skip the lookup entirely, and they are checked before anything else. self.enable_caching is the engine-wide setting; request.skip_reading_prefix_cache is per-request, and it is set for a request that asked for prompt logprobs or for an all-pooling call to a pooling model. Both need the prompt genuinely recomputed, so reusing a cached block would give the wrong answer rather than a slow one. The early return is therefore a correctness guard on one arm and a configuration check on the other, sharing one if.

The empty result is a pre-built object, not a fresh one. self.empty_kv_cache_blocks is constructed once, up front, as nested tuples so it is immutable and safe to hand out repeatedly. The stated reason is to avoid the garbage-collection cost of minting a new empty result on every miss. That is a small thing, but it tells you how often this path runs: once per request per admission attempt, on a method called from the hot loop of every forward pass.

max_cache_hit_length = request.num_tokens - 1 is a deliberate off-by-one, and it means a 100% prefix hit is impossible. If every token of a prompt were served from cache, the model would have computed no logits for the final position, and there would be nothing to sample from. So the ceiling on the hit is one token short of the prompt, always. The source notes the knock-on effect: because the layer above requires the computed-token count to be block-size aligned, holding back one token can force the recomputation of an entire block, not just that token. On a fully-cached 1,024-token prompt with 16-token blocks, the last 16 tokens get recomputed to produce one position's logits.

The return is a three-tuple, and only the middle element is what the scheduler spends. The first is the block set. The third, shared_prefix_boundary, is a hybrid-model concern — the token position where a group that caches sparsely stops having the prefix — and on the single-group full-attention path this module follows it is 0.

The hashes arrive already computed

self.coordinator.find_longest_cache_hit(request.block_hashes, max_cache_hit_length) passes request.block_hashes straight through. Layer 1 does not compute the chain, and neither does layer 2 or layer 3 — the hashes are a field on the Request, computed elsewhere, and by the time the KV cache manager sees them they are just a list to walk.

That fixes a boundary the concept module leaves fuzzy: hashing is not part of the cache lookup. The lookup is "given a list of hashes, how many are keys in the pool's table, counting from the front". Nothing in the four layers decides what a block hash is made of.

Two arguments, then two seams

Layer 1 calls exactly one method on layer 2 here, with exactly two arguments. Step 3 opens that call and shows what layer 2 adds to it on the way down — which is where the seam becomes visible, because layer 3 needs eight arguments to do the same job.

The Longest Match

How does vLLM find the longest cached prefix?

With a for loop that breaks on the first miss, and a subtraction that rounds the answer down to a block boundary. No tree walk, no binary search, no scan of the whole hash list.

The concept module gets you as far as expecting that. What it cannot give you is where the loop lives, and how much has to be handed to it before it can run.

Layer 2's whole job here is to add arguments

# distilled — real names, full bodyclass UnitaryKVCacheCoordinator(KVCacheCoordinator):  def find_longest_cache_hit(      self,      block_hashes: list[BlockHash],      max_cache_hit_length: int,  ) -> tuple[tuple[list[KVCacheBlock], ...], int, int]:      hit_blocks, hit_length = self.single_type_managers[0].find_longest_cache_hit(          block_hashes=block_hashes,          max_length=max_cache_hit_length,          kv_cache_group_ids=[0],          block_pool=self.block_pool,          kv_cache_spec=self.kv_cache_spec,          drop_eagle_block=0 in self.eagle_group_ids,          alignment_tokens=self.block_size,          dcp_world_size=self.dcp_world_size,          pcp_world_size=self.pcp_world_size,      )      return hit_blocks, hit_length, 0

Two arguments came in. Nine go out. That expansion is the seam, and it is worth naming precisely, because "the coordinator forwards the call" undersells what happens.

Layer 3's method is a @classmethod. It is called on the class, not through instance state, so it cannot reach for the pool or the cache spec on its own — it has no self. Everything it needs to do its job has to arrive in the argument list. Layer 2 is the only object in the chain that holds all of it: the single pool it constructed, the spec of the one group, its own derived block size, and which groups belong to a speculative-decoding draft model. So layer 2 is not a pass-through with an extra frame on the stack. It is the place the context lives.

Three of those nine arguments name features that do not apply to a plain single-GPU full-attention run, and it is quicker to define them once than to trip over them. EAGLE is a speculative-decoding scheme with its own small draft model, whose KV lives in its own cache group. DCP and PCP are decode- and prefill-context parallelism, which shard one block's keys and values across several ranks — so a block covers more tokens than its nominal size, and the block size has to be multiplied out before any hash is compared. alignment_tokens is the boundary a hit is allowed to land on, and layer 2 passes its own block size for it, which is what makes the trim at the bottom of the loop a no-op in the ordinary case.

self.single_type_managers[0] is a hard index, and it is only legal because of an assertion. UnitaryKVCacheCoordinator's constructor asserts len(self.kv_cache_config.kv_cache_groups) == 1 outright. That assertion is what buys the [0]. In the hybrid coordinator the same lookup is a loop over spec groups, and its version of this method is a much longer body — so the moment you see an index rather than a loop, you are on the single-group path and nowhere else.

The third return value is the literal 0. The signature promises a three-tuple; the body computes two things and appends a constant. Its comment says why: with one group, there is no second group to lag behind, so there can be no shared prefix that some other group has not cached yet. A return type that looks general because the base class had to be general, filled in with a constant on the path that cannot need it.

The loop that computes the hit

# distilled — real names, reduced body: phase 1 and the trim, fine-grained phase elidedclass FullAttentionManager(SingleTypeKVCacheManager):  @classmethod  def find_longest_cache_hit(      cls,      block_hashes: BlockHashList,      max_length: int,      kv_cache_group_ids: list[int],      block_pool: BlockPool,      kv_cache_spec: KVCacheSpec,      drop_eagle_block: bool,      alignment_tokens: int,  ) -> tuple[tuple[list[KVCacheBlock], ...], int]:      assert isinstance(          kv_cache_spec, FullAttentionSpec | ChunkedLocalAttentionSpec      )      block_size = kv_cache_spec.block_size      full_block_hashes = resolve_block_hashes(          block_hashes, block_pool.hash_block_size, block_size,          alignment_tokens=alignment_tokens,      )       computed_blocks: tuple[list[KVCacheBlock], ...] = tuple(          [] for _ in range(len(kv_cache_group_ids))      )      for block_hash in itertools.islice(full_block_hashes, max_length // block_size):          cached_block = block_pool.get_cached_block(block_hash, kv_cache_group_ids)          if not cached_block:              break          for computed, cached in zip(computed_blocks, cached_block):              computed.append(cached)      hit_length = len(computed_blocks[0]) * block_size       if drop_eagle_block and hit_length > 0:          hit_length -= min(alignment_tokens, block_size)      hit_length -= hit_length % alignment_tokens      num_blocks = cdiv(hit_length, block_size)      for computed in computed_blocks:          del computed[num_blocks:]      return computed_blocks, hit_length

The break is the algorithm. block_pool.get_cached_block is a dictionary lookup; the loop does one per block from the front, and the first falsy result ends it. The source states the licence for stopping in a comment: a missing block implies every later block misses too, because the hashes are chained. So the cost of a lookup is proportional to the length of the hit, not to the length of the prompt — a request that shares nothing with the cache costs one dictionary probe.

itertools.islice(..., max_length // block_size) is where the off-by-one from step 2 lands. max_length arrived as request.num_tokens - 1, and integer division turns it into a count of whole blocks. That single expression is the reason a fully-cached prompt still recomputes its last block: for a 1,024-token prompt with 16-token blocks, 1023 // 16 is 63, so the loop can never look at the 64th hash.

hit_length is counted in blocks and reported in tokens. len(computed_blocks[0]) * block_size — the hit is however many blocks the loop appended, multiplied out. There is no partial credit for tokens inside the first missing block. The concept module's "you only hit on block boundaries" is this multiplication.

Then the answer is trimmed, and the block list is truncated to match. hit_length -= hit_length % alignment_tokens rounds down to the alignment — a no-op in the ordinary case, where layer 2 passed its own block size and the hit was already a whole number of blocks. del computed[num_blocks:] drops blocks past the tail, so what comes back is not simply "the blocks the loop found", it is that list cut back to agree with the length. The drop_eagle_block arm above them is the one that bites in practice: with a draft model attached, the tokens immediately before the generation point have to be recomputed, so one unit of hit is thrown away before the rounding. Three independent reasons to shorten a hit, applied in sequence, in five lines.

What the real body also does

The distilled version above is phase 1 plus the trim. The real method has a second phase, taken only in fine-grained mode — when the hash granularity is smaller than the group's block size, which lets the lookup probe inside the first non-full block, highest boundary first, and extend the hit past a block edge. That is where the alignment trim stops being a no-op.

It does not change the shape: probe, stop at a miss, round down, truncate. The fine-grained phase is the same loop run at a finer stride over one block's interior.

What layer 1 does with the answer

hit_length comes back up through layer 2 as num_new_computed_tokens, and layer 1 returns it to the scheduler. Nothing has been reserved. The blocks in that list are still, as far as the pool is concerned, whatever they were before the lookup — possibly sitting in the free queue as eviction candidates with a reference count of zero.

Which is a problem, and the next step is where it gets solved: the request cannot be admitted on the strength of a hit until someone has checked that the blocks it still needs can be found. That check is a pure question with a -> int answer, and it runs before a single block changes hands.

Allocating the Rest

How does vLLM decide whether a KV allocation will fit?

By asking a pure function for a number, adding a margin to it, and comparing. KVCacheManager.allocate_slots is the only method the scheduler calls to get blocks, and the first thing it does with the pool is not take from it — it reads the free count and compares it against a prediction. Nothing is handed out until that comparison passes.

The prediction comes from layer 2, it has return type int, and it allocates nothing. Which means the interesting part of an allocation is a question, and the question runs twice per request with two different meanings.

The capacity spine

# distilled — real names, reduced body: the two capacity questions and their comparisonsclass KVCacheManager:  def allocate_slots(      self,      request: Request,      num_new_tokens: int,      num_new_computed_tokens: int = 0,      num_lookahead_tokens: int = 0,      full_sequence_must_fit: bool = False,      reserved_blocks: int = 0,      has_scheduled_reqs: bool = True,  ) -> KVCacheBlocks | None:      num_local_computed_tokens = (          request.num_computed_tokens + num_new_computed_tokens      )      total_computed_tokens = min(          num_local_computed_tokens, self.max_model_len      )       watermark_blocks = 0      if has_scheduled_reqs and request.status in (          RequestStatus.WAITING,          RequestStatus.PREEMPTED,      ):          watermark_blocks = self.watermark_blocks       if full_sequence_must_fit:          full_num_tokens = min(request.num_tokens, self.max_model_len)          num_blocks_to_allocate = self.coordinator.get_num_blocks_to_allocate(              request_id=request.request_id,              num_tokens=full_num_tokens,              num_tokens_main_model=full_num_tokens,              apply_admission_cap=True,          )          required_blocks = num_blocks_to_allocate + watermark_blocks          if required_blocks > self.block_pool.get_num_free_blocks():              return None       num_tokens_main_model = total_computed_tokens + num_new_tokens      num_tokens_need_slot = min(          num_tokens_main_model + num_lookahead_tokens, self.max_model_len      )      self.coordinator.remove_skipped_blocks(          request.request_id,          max(0, total_computed_tokens - request.num_in_flight_tokens),          num_prompt_tokens=request.num_prompt_tokens,      )       num_blocks_to_allocate = self.coordinator.get_num_blocks_to_allocate(          request_id=request.request_id,          num_tokens=num_tokens_need_slot,          num_tokens_main_model=num_tokens_main_model,      )      available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks      required_blocks = num_blocks_to_allocate + watermark_blocks      if required_blocks > available_blocks:          return None       new_blocks = self.coordinator.allocate_new_blocks(          request.request_id, num_tokens_need_slot, num_tokens_main_model      )      return self.create_kv_cache_blocks(new_blocks)

The real method is much longer — it also handles the prefix-hit blocks, external blocks fetched by a KV connector, and the caching of newly-filled blocks on the way out. Strip those and this is what remains: two comparisons and two return Nones, then one downward call.

Two calls to one predictor, two different questions

The same method on layer 2 is called twice, and treating those calls as duplicates is the fastest way to misread this code.

The flag is not cosmetic, and its effect is the opposite of what "cap" suggests. One layer down it selects a single min() against a per-request ceiling the manager was constructed with — so with the cap on, the required-block count is clamped downward.

That is there for recycling attention types: a sliding-window or chunked-local group never holds a long sequence's whole length in blocks at once, because blocks fall out of the window and return to the pool as it advances. Asking "do all 100,000 tokens fit" of such a group would demand blocks the request will never simultaneously own, and the gate would refuse admission forever. The clamp replaces that number with the same recycling-aware ceiling the startup pool sizer used, so the two agree. The source comment names the bug the mismatch caused. For a full-attention group the ceiling is unset and the clamp does nothing.

Which is exactly why the per-step call must leave the flag alone. The docstring says so outright: per-step allocation "must leave it False so the predictor matches allocate_new_blocks". The two calls ask for two different quantities — a ceiling on what one request can ever hold at once, and the exact number of blocks this step will take. Only the second is a prediction of the allocation that follows it.

Why the admission gate exists at all. Chunked prefill only ever checks the chunk in front of it, so without the first call an engine could admit a 100,000-token request on the strength of its first 4,096 tokens fitting, then discover the rest does not fit and start preempting immediately. The gate makes the whole sequence the unit of the admission decision, and leaves the per-step check to do exact arithmetic afterwards.

The margin is different for different requests

watermark_blocks is the other reason this is not one uniform check. It starts at 0 and is only raised to self.watermark_blocks when both conditions hold: some request is already scheduled this pass, and the request in hand is WAITING or PREEMPTED.

So inside one method, on the same pool, a request that is already RUNNING is measured against the bare free count while a request trying to get in is measured against the free count minus a reserve. An engine can therefore be simultaneously "full" for a newcomer and "not full" for the requests it is already serving — which is the intended behaviour, since it is cheaper to keep serving than to admit and then evict. self.watermark_blocks is computed once at construction as int(watermark * kv_cache_config.num_blocks), and the default watermark is 0.0, which collapses the distinction unless the operator asks for it.

reserved_blocks layers a second, separate reserve on top, and it is per-call rather than per-config: it is how a caller says "leave this many blocks for sequences that are already prefilling", so an asynchronous KV-connector load cannot consume blocks an in-flight request is relying on.

The predictor itself

# distilled — real names, reduced body: the fan-out, docstring strippedclass KVCacheCoordinator(ABC):  def get_num_blocks_to_allocate(      self,      request_id: str,      num_tokens: int,      new_computed_blocks: tuple[Sequence[KVCacheBlock], ...],      num_encoder_tokens: int,      total_computed_tokens: int,      num_local_computed_tokens: int,      num_tokens_main_model: int,      apply_admission_cap: bool = False,  ) -> int:      num_blocks_to_allocate = 0      for i, manager in enumerate(self.single_type_managers):          if isinstance(manager, CrossAttentionManager):              num_blocks_to_allocate += manager.get_num_blocks_to_allocate(                  request_id, num_encoder_tokens, [], 0, 0, num_encoder_tokens,                  apply_admission_cap=apply_admission_cap,              )          else:              num_blocks_to_allocate += manager.get_num_blocks_to_allocate(                  request_id, num_tokens, new_computed_blocks[i],                  total_computed_tokens, num_local_computed_tokens,                  num_tokens_main_model,                  apply_admission_cap=apply_admission_cap,              )      return num_blocks_to_allocate

It is a sum over the tuple from step 1. One term per KV cache group. On the single-group path this is a loop that runs once, and the coordinator adds nothing to the number layer 3 returned — which is exactly why a single-group engine can be read as if it had three layers instead of four, and exactly why that reading breaks on a hybrid model.

new_computed_blocks[i] is the reason the argument is a tuple of sequences rather than a list of blocks. The prefix hit from step 3 is per-group, and each manager is only told about its own share. Index i is the group id.

The isinstance arm is a genuinely different question. A CrossAttentionManager holds the encoder-side KV of an encoder–decoder model, which does not grow as the decoder generates. So it is sized by num_encoder_tokens and passed empty prefix blocks and zeroed computed-token counts — one static allocation, asked about with the same method name and completely different arguments. Two branches, one loop, and the branch is chosen by the type of the object rather than by a flag.

Nothing here allocates. The return type is int, the body has no assignment to any pool state, and the only calls are to more -> int methods one layer down. That is what makes the number safe to compare before committing: layer 1 can ask the question as many times as it likes.

What layer 3 counts

The per-group term is not simply "tokens divided by block size". Layer 3's version starts at num_required_blocks = cdiv(num_tokens, self.block_size), then subtracts what the request already holds, and then adds a term the concept module gives you no reason to expect: the number of evictable blocks among the prefix-cache hits. A hit block whose reference count is zero is currently sitting in the free queue as an eviction candidate, so touching it will remove it from that queue — which means it has to be counted against free capacity even though it is not a new block. Layer 3 also reserves one extra block when the hit ends part-way through a block, because a partial hit needs a private copy of the shared tail.

So the number that reaches the comparison is: blocks this request still needs, plus cached blocks it is about to rescue from the free queue, plus one for a copy-on-write tail if there is one.

What has and has not happened

One thing between the two comparisons is not a read. self.coordinator.remove_skipped_blocks(...) frees blocks that the attention window has moved past, and it runs before the second capacity question — deliberately, the comment says, to reduce the number of blocks that have to be evicted, and it is safe to run even for a request that turns out not to be schedulable.

On the single-group full-attention path this module follows, that call is a no-op: the base implementation of the skipped-token count returns 0, so full attention never skips anything. It bites on sliding-window and chunked-local groups, where old blocks genuinely fall out of the window, and on Mamba, where only the last computed token's state is kept. Worth knowing before you conclude that a failed allocation left the world untouched.

The comparison has now passed. Step 5 follows the one downward call that follows it, through layer 2's fan-out and layer 3's arithmetic to the queue the blocks actually come off.

Where Blocks Come From

Where do vLLM's KV blocks actually come from?

The front of a queue. BlockPool holds one self.free_block_queue, and every block any request has ever received came off the left end of it. The whole descent from layer 1 to that queue is three calls: a fan-out, a division, and a pop.

The concept module describes a free list you pop from. Two things about the real one are not derivable from that description, and both live in the same eleven-line method: the pop is where a block stops being a cache entry, and the queue's order is the eviction order because position in it is what decides who goes.

Layer 2: one comprehension, N groups

# distilled — real names, full body, docstring strippedclass KVCacheCoordinator(ABC):  def allocate_new_blocks(      self,      request_id: str,      num_tokens: int,      num_tokens_main_model: int,      num_encoder_tokens: int = 0,  ) -> tuple[list[KVCacheBlock], ...]:      return tuple(          manager.allocate_new_blocks(              request_id,              num_encoder_tokens              if isinstance(manager, CrossAttentionManager)              else num_tokens,              num_tokens_main_model,          )          for manager in self.single_type_managers      )

This is the allocation twin of the predictor in step 4, and the resemblance is the point: same loop over self.single_type_managers, same isinstance special case for cross-attention, same per-group treatment. One returns an int and takes nothing; this one returns blocks and takes them.

The return type is a tuple of lists, one list per group. That shape is what makes the four-layer chain honest about hybrid models: the caller does not get "the blocks", it gets the blocks per group, and a request in a hybrid model genuinely holds several disjoint sets that grow at different rates.

There is no capacity check anywhere in this body. Layer 2 does not re-ask the question layer 1 asked; it does not compare anything; it does not catch anything. It commits. Remember that when you reach step 6, because a generator expression with no error handling in the middle of a four-layer descent is exactly the structure that makes a failure here unrecoverable.

Layer 3: where a token count becomes a block count

# distilled — real names, reduced body, docstring strippedclass SingleTypeKVCacheManager:  def allocate_new_blocks(      self, request_id: str, num_tokens: int, num_tokens_main_model: int  ) -> list[KVCacheBlock]:      cow_blocks: list[KVCacheBlock] = []      if request_id in self._partial_hit_reqs:          block_idx, source_block = self._partial_hit_reqs.pop(request_id)          cow_block = self.block_pool.get_new_blocks(1)[0]          self._apply_cow(request_id, block_idx, source_block, cow_block)          self.new_block_ids.append(cow_block.block_id)          cow_blocks.append(cow_block)       req_blocks = self.req_to_blocks[request_id]      num_required_blocks = cdiv(num_tokens, self.block_size)      num_new_blocks = num_required_blocks - len(req_blocks)      if num_new_blocks <= 0:          return cow_blocks      else:          new_blocks = self.block_pool.get_new_blocks(num_new_blocks)          req_blocks.extend(new_blocks)          if self._record_new_block_ids:              self.new_block_ids.extend(b.block_id for b in new_blocks)          return cow_blocks + new_blocks

cdiv(num_tokens, self.block_size) is the only place in the chain that divides. Layers 1 and 2 pass token counts around; layer 3 turns one into a block count, using its own block size, which is the group's — not the engine's, and not the hash block size. On a hybrid model, two managers given the identical num_tokens compute different num_required_blocks.

Subtracting len(req_blocks) is the whole idea of incremental allocation. self.req_to_blocks[request_id] is what this group has already given this request. num_required_blocks - len(req_blocks) is the delta, and it is recomputed from scratch on every call rather than tracked — so the bookkeeping cannot drift out of sync with the list.

if num_new_blocks <= 0: return cow_blocks — layer 3 can decide to allocate nothing. A decoding request whose current block is not yet full needs zero new blocks, and that is the common case: most forward passes add one token to a partially-filled block and no block leaves the pool. (Read the return value, not the method name: the name is cow_blocks, and the copy-on-write branch at the top of this same method does pull one block with its own get_new_blocks(1) before the subtraction runs. That branch fires on a partial prefix hit at admission, not on a steady decode — which is why the common case still allocates nothing.) The <= rather than == is deliberate: with speculative decoding, blocks may have been allocated for draft tokens that were then rejected, so the required count can come out below what the request already holds. The method treats that as "nothing to do" rather than as an error.

So the fan-out from layer 2 hits the pool somewhere between zero and N times per call. "One allocation per request per step" is not what the code does.

And it can hit the pool twice for one group. The _partial_hit_reqs block at the top is the copy-on-write case: when the prefix hit from step 3 ended part-way through a block, that shared tail block cannot be written into, because another request is still using it. So layer 3 pulls a single private block with its own separate self.block_pool.get_new_blocks(1) call, redirects the request's block table entry to it, and queues a copy for the worker to run. That is the extra block step 4's predictor reserved. Two calls into layer 4, from one call into layer 3 — and it matters again in step 6, where committing that block before the main allocation is part of why a failure below cannot be walked back. (The panel's demo request hits on a clean block boundary, so this branch is not one you will see there.)

Layer 4: the pop, and what the pop does

# distilled — real names, reduced body: the caching arm only, docstring strippedclass BlockPool:  def get_new_blocks(self, num_blocks: int) -> list[KVCacheBlock]:      if num_blocks > self.get_num_free_blocks():          raise ValueError(f"Cannot get {num_blocks} free blocks from the pool")       ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)       if self.enable_caching:          for block in ret:              self._maybe_evict_cached_block(block)              assert block.ref_cnt == 0              block.ref_cnt += 1      return ret

Eleven lines, and three of them change how you should read the concept module.

popleft_n — the front, always. There is no search for a suitable block, no size class, no best fit. Every block is interchangeable, so the pool takes however many it needs off the left end in one operation. The queue is a doubly linked list of every block in the pool, built once at construction from self.blocks, and its documented purpose is to store "the free blocks in eviction order".

self._maybe_evict_cached_block(block) is where eviction under memory pressure happens. There is no eviction pass, no background sweeper, no watermark that triggers a cleanup. A block sitting in the free queue may still be a valid prefix-cache entry — reachable through the block-hash table, ready to be reused by a future request that shares that prefix. It stays that way until somebody needs a block. Then, at the moment it is popped, its hash metadata is cleared and its entries are removed from the table, one block at a time, inside this loop. Eviction under pressure is a side effect of allocation, and the free queue is the eviction order because position at the front of it is what decides who goes.

(BlockPool does expose two deliberate invalidation paths as well — one that drops specific block ids from the cache at a KV connector's request, and a full prefix-cache reset used after a weight update or before a benchmark. Neither is driven by memory pressure, and neither is on the allocation path.)

That also explains why a cache hit has to reach into the queue. When step 3's lookup finds a cached block whose reference count is zero, the pool removes that block from the free queue on the way to handing it over — un-queueing it so that a later allocation cannot pop and evict the block that a request now depends on. Which is why step 4's predictor counted those evictable hits against free capacity: rescuing them from the queue makes them unavailable to everybody else.

assert block.ref_cnt == 0 then block.ref_cnt += 1. The reference count is the entire ownership model. Zero means "in the free queue, evictable"; anything above zero means somebody is using it; the assertion is the invariant that a block cannot be in the queue and owned at the same time. Nothing above layer 4 ever touches a reference count — that is the one thing the bottom layer keeps entirely to itself.

And the first line is a capacity check, four layers below the one in step 4. Same shortage, checked twice, in two places that know different things. That is step 6.

The whole descent, once

Read the last column top to bottom and the four layers stop looking like ceremony. Each one exists because the number it passes down is not the number it received.

How Allocation Fails

Why does a full KV cache preempt instead of crashing?

Because the layer that notices is layer 1, and layer 1 knows enough to have an opinion. The same shortage of blocks is checked twice in this chain — once at the top and once at the bottom, four layers apart — and the two checks are not redundant, because the two layers can see different things. The top one returns a value. The bottom one raises. Only one of them is on the path the scheduler takes.

Module 2 told you what the scheduler does with the sentinel, and that it does two different things depending on which of the two allocate_slots call sites returned it. In the loop over self.running, if new_blocks is not None failing enters a while True that pops a victim off self.running, preempts it, and asks again. In the admission loop over self.waiting, the identical None frees the request's encoder cache and breaks the admission attempt — nothing is evicted on that side. This step is the other side of that boundary — why the value is a None rather than an exception in the first place, and why the exception one floor down cannot be turned into a None even in principle.

The same predicate, four layers apart

# distilled — real names: the two comparisons that produce the sentinel, everything else cutclass KVCacheManager:  def allocate_slots(self, request: Request, ...) -> KVCacheBlocks | None:      if full_sequence_must_fit:          required_blocks = num_blocks_to_allocate + watermark_blocks          if required_blocks > self.block_pool.get_num_free_blocks():              return None       available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks      required_blocks = num_blocks_to_allocate + watermark_blocks      if required_blocks > available_blocks:          # Cannot allocate new blocks          return None

# distilled — real names: the guard onlyclass BlockPool:  def get_new_blocks(self, num_blocks: int) -> list[KVCacheBlock]:      if num_blocks > self.get_num_free_blocks():          raise ValueError(f"Cannot get {num_blocks} free blocks from the pool")

Put them side by side and the shape is nearly identical: a required count, a free count, a >. What differs is the vocabulary each layer has for the word "free".

Layer 1's bar is strictly stricter, and that is what makes layer 4's guard unreachable. required_blocks is the prediction plus the watermark; available_blocks is the free count minus the reserve. Both adjustments push in the direction of failing earlier. So on any path that goes through allocate_slots, the pool is never asked for more blocks than it has — not because anyone catches the ValueError, but because the question was already asked one floor up by somebody who knew more.

Which makes the prediction load-bearing

The only reason layer 1's answer can be trusted to protect layer 4 is that the number it compares is the number layer 4 will actually be asked for. That is precisely the guarantee the apply_admission_cap flag from step 4 exists to preserve.

# distilled — real names: the contract, body reduced to its shapeclass KVCacheCoordinator(ABC):  def get_num_blocks_to_allocate(      self,      request_id: str,      num_tokens: int,      apply_admission_cap: bool = False,  ) -> int:      """apply_admission_cap: Set only by the full-sequence admission gate;      per-step allocation must leave it False so the predictor matches      `allocate_new_blocks`."""      num_blocks_to_allocate = 0      for i, manager in enumerate(self.single_type_managers):          num_blocks_to_allocate += manager.get_num_blocks_to_allocate(              request_id, num_tokens, apply_admission_cap=apply_admission_cap          )      return num_blocks_to_allocate

That comment is the docstring's own words, and it is the whole safety argument in one sentence. With the flag left False, the per-step prediction is exact — the same arithmetic the allocation will do, run ahead of time and returned as an int. Which is why return None can be an ordinary control-flow signal rather than a guess: layer 1 is not estimating whether the allocation will succeed, it is computing it.

Note which of the two calls is doing the protecting. The full-sequence gate is a separate, earlier filter with a different question and a clamped number; it can refuse an admission the per-step check would have allowed. The guarantee that the pool is never over-drawn comes from the second call, the exact one.

Why converting the exception would not help

It is tempting to read the two representations as a style inconsistency someone should clean up — have get_new_blocks return None too and let the caller decide. The pool's own guard is in fact clean enough for that: the comparison runs before popleft_n, so a failing call to get_new_blocks mutates nothing at all. The problem is not that method. It is everything already committed above it.

return tuple(
    manager.allocate_new_blocks(request_id, ..., num_tokens_main_model)
    for manager in self.single_type_managers
)

That is a generator expression over every KV cache group, with no error handling and no transaction. Suppose the pool did run dry mid-descent on a hybrid model. Group 0's manager has already popped its blocks off the free queue, already had their prefix-cache entries evicted, already incremented their reference counts, and already appended them to its req_to_blocks. Group 1's raises. Nothing unwinds any of that.

The same is true inside a single group: the copy-on-write path takes one block from the pool with its own separate call and immediately rewrites a block-table entry to point at it, before the main allocation runs. Fail after that and the request's block table has been mutated to reference a block the request does not fully own.

So a None handed up from layer 4 would be safe locally and useless globally. Whoever received it has no way to undo what earlier iterations of that generator already did, and would report a clean failure to the scheduler while leaving blocks leaked and a block table pointing at a partly-owned block — a slow corruption instead of a fast crash. A sentinel is only a usable answer where the whole operation can still be declined: layer 1's position, before any downward call, and not layer 4's, in the middle of one.

What each representation does after that is module 2's subject, traced in full in The Branch That Preempts — the retry loop, the two victim-selection branches, and the reason one preemption cancels every admission in the pass. The point here is upstream of all of it: the reason there is a value to branch on at all is that the layer holding the policy is also the layer that checks, and it checks before it commits.

What you can now do with a traceback

If a log line reads ValueError: Cannot get N free blocks from the pool, the useful question is not "how do I give the engine more memory". Layer 1's comparison exists to make that line impossible, so its appearance says the pool was reached by something that did not go through allocate_slots — or that one of the two numbers layer 1 compared was wrong. Either way it is a defect, not a capacity signal, and the capacity signals are silent by design: a None, a preemption, and a request that quietly re-prefills.

The four layers are behind you. Layer 1 held the policy, layer 2 held the fan-out, layer 3 held the arithmetic, layer 4 held the blocks — and the two most interesting behaviours in the module, the read-only lookup and the two-faced failure, were both properties of a seam rather than of any one class. What leaves this chain is a set of block ids per group, riding out on the scheduler's output to the worker, which is where the next module picks it up.