vLLM's Scheduler — Both Halves of One Class, in Source

Two Halves, One Class

What are the scheduler's two halves?

Scheduler is a single class in vllm/v1/core/sched/scheduler.py, and the engine loop calls into it twice per forward pass, from opposite ends of the pass. Scheduler.schedule runs first and decides what the GPU will work on. Scheduler.update_from_output runs last: it takes the worker's result, folds the sampled tokens back into per-request state, and builds the EngineCoreOutputs that leave the process. Both halves read and write the same collections — self.running and the waiting queue.

You already know what the first half decides, and why. That is the whole subject of the LLM Serving track's scheduler step, and this module does not repeat it. What that step cannot give you is the part you need in order to change anything: which local variable holds the budget, which line moves a request from one collection to another, and which if picks the preemption branch. That is what the next five steps are.

The one structural fact to take from this step is that the return path lives in the same class as the admission path. update_from_output is not a separate component that reports back to the scheduler — it is the scheduler, mutating the same lists that schedule finished writing earlier in the very same pass.

Three writers, not two

There is a third method, and it is neither half. Scheduler.add_request is the only way a request enters the scheduler's world at all, and it runs before that request has been near either half.

Module 1 left the request at the last line of phase 4: the engine-core busy loop had just called self.scheduler.add_request(request). This is that call.

The entry point

# distilled — real names, reduced bodyclass Scheduler(SchedulerInterface):  def add_request(self, request: Request) -> None:      existing = self.requests.get(request.request_id)      if existing is not None:          update = StreamingUpdate.from_request(request)          if existing.status != RequestStatus.WAITING_FOR_STREAMING_REQ:              existing.streaming_queue.append(update)          elif update is not None:              self._update_request_as_session(existing, update)          else:              self.finish_requests(                  request.request_id, RequestStatus.FINISHED_ABORTED              )      else:          if request.resumable:              request.streaming_queue = deque()          self._enqueue_waiting_request(request)          self.requests[request.request_id] = request          if self.connector is not None:              self.connector.on_new_request(request)          if self.log_stats:              request.record_event(EngineCoreEventType.QUEUED)

Two things in that body are worth reading closely, because both of them are load-bearing later.

The whole method is an if existing is not None fork. A request id that the scheduler has already seen does not create a new request — so "one call per request" is the wrong model. A second call for a live id is treated as the next input chunk of a streaming-input session already in flight: the payload becomes a StreamingUpdate and is either queued on the existing request or applied to it immediately. add_request is two operations sharing one name — create, and continue — and only the else arm grows a queue.

It does not append to self.waiting directly. The else arm calls an enqueue helper, and that helper looks at request.status before it decides which collection to put the request in. There is more than one waiting collection, and step 3 is entirely about that. A request whose id came back from add_request is not necessarily in self.waiting.

Notice also what add_request does not do: no KV blocks are allocated, no budget is consulted, nothing is checked against a limit. It cannot fail for capacity reasons. Every capacity decision in this class happens in schedule.

What "one class" buys, and what it costs

Module 1 established the order: the engine-core busy loop calls EngineCore.step, which calls schedule near the top and update_from_output near the bottom, on the same thread, once each per pass. Keeping both in one class is what lets each of them read the other's state as plain attribute access — no message, no snapshot, no defence against a concurrent mutation, because within a pass there is no concurrency to defend against.

It is also why the file is 2,823 lines and why schedule alone runs to several hundred. There is no smaller unit to read. The next step takes the one thread through it that you can actually hold in your head — the token budget — and ignores everything else.

Inside the Budget Loop

Where does the token budget actually live?

In a local variable. token_budget is initialised at the top of Scheduler.schedule, spent by subtraction as requests are scheduled, and thrown away when the method returns. It is not a field, not a counter that persists across passes, and not something any other component can read. Every call to schedule starts over from self.max_num_scheduled_tokens — or from 0, on the one branch where the engine has been paused outright.

That is the first thing you cannot learn from the concept: there is no budget object. There is an integer on the stack, and two while loops that decrement it.

Two loops, one budget

# distilled — real names, reduced body: the token_budget skeleton onlyclass Scheduler(SchedulerInterface):  def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:      num_scheduled_tokens: dict[str, int] = {}      preempted_reqs: list[Request] = []      token_budget = self.max_num_scheduled_tokens      req_index = 0      while req_index < len(self.running) and token_budget > 0:          request = self.running[req_index]          num_new_tokens = (              request.num_tokens_with_spec              + request.num_output_placeholders              - request.num_computed_tokens          )          num_new_tokens = min(num_new_tokens, token_budget)          num_scheduled_tokens[request.request_id] = num_new_tokens          token_budget -= num_new_tokens          req_index += 1       if not preempted_reqs and self._pause_state == PauseState.UNPAUSED:          while (self.waiting or self.skipped_waiting) and token_budget > 0:              request_queue = self._select_waiting_queue_for_scheduling()              request = request_queue.peek_request()              num_new_tokens = request.num_tokens - num_computed_tokens              if not self.scheduler_config.enable_chunked_prefill and num_new_tokens > token_budget:                  break              num_new_tokens = min(num_new_tokens, token_budget)              self.running.append(request_queue.pop_request())              num_scheduled_tokens[request.request_id] = num_new_tokens              token_budget -= num_new_tokens

Everything else in this method — encoder inputs, speculative-decode padding, mamba block alignment, KV-connector transfers, LoRA slot accounting, the whole preemption branch — hangs off this skeleton. Find token_budget in the real file and you have found the spine.

Five concrete facts fall out of the shape:

The two loops are two literal statements, in that order. The while over self.running is written first; the while over the waiting queues is written second, and it is nested inside an if. There is no comparison between a running request and a waiting one anywhere — no scoring function decides which goes first. Order of execution is the policy.

A request already in self.running never gets rejected by the budget. It gets clamped. The line that matters is num_new_tokens = min(num_new_tokens, token_budget). A request that wants 900 tokens with 300 left in the budget is scheduled for 300, and the remaining 600 wait for the next pass. This is chunked prefill, and it is not a separate code path — it is one min().

A request in the waiting queue can be rejected outright by the budget. The plainest case is the guard shown above: if not self.scheduler_config.enable_chunked_prefill and num_new_tokens > token_budget breaks out of the admission loop rather than clamping. (It is not the only break on a budget comparison — the speculative-decode padding path has one too, when padding a decode up to a uniform size would overrun what is left.) Chunked prefill is on by default, so on a default engine the common path clamps.

In the admission loop, num_computed_tokens is a local — and that is where prefix caching shows up. Read that subtraction carefully: it is request.num_tokens - num_computed_tokens, not request.num_tokens - request.num_computed_tokens. For a request being admitted for the first time, the local was just set from the number of its prompt tokens the KV cache already holds, as reported by KVCacheManager.get_computed_blocks, plus anything a KV connector says it can fetch from elsewhere. So the whole benefit of a prefix-cache hit reaches the scheduler as one smaller number on the right-hand side of a minus sign. Where that number comes from is the next module's subject; that it lands here is this one's.

The budget is never checked against a total; it is only ever decremented. After the loops, the method asserts that token_budget has not gone negative and that the sum of num_scheduled_tokens has not exceeded self.max_num_scheduled_tokens. Those assertions are a tripwire, not the mechanism — the arithmetic above them is what actually holds the line, and the assertions exist to catch a future branch that forgets to do it.

The dict that both halves share

num_scheduled_tokens is the other local worth tracing. It maps request id to the number of tokens this pass will process for that request, and it is the payload of the whole method: it becomes a field on the SchedulerOutput that goes to the worker.

That third column is where the two halves join. update_from_output does not iterate self.running to find out what ran — it iterates the keys of num_scheduled_tokens, which the first half wrote. Step 5 picks that up.

Why req_index and not a for

The running loop is indexed manually because it is not safe to iterate. The preemption branch mutates self.running while the loop is walking it: one arm pops an element off the end, and the other removes an element from the middle and — when that element had already been scheduled earlier in this same pass — does req_index -= 1 so the walk does not skip whatever slid into the vacated slot.

The other reason is that the loop uses continue rather than break when a request cannot be scheduled. Four separate continue statements do this, and the widest of them fires when the request's num_new_tokens came out as zero — which the source attributes to any of four causes, including "no new tokens to schedule" and "the encoder budget is exhausted". Skipping one running request and carrying on means the pass is deliberately not strictly first-come-first-served, and a comment on that continue says exactly that: lower-priority requests behind a stuck one do get scheduled.

Which raises the question the next step answers: if a request can be skipped, deferred, blocked, or evicted, then "waiting" and "running" are not enough words. There are three collections, and the request's status field is a fourth axis on top of them.

Which Queue a Request Is In

How many collections does the scheduler keep?

Four that a request can be queued in — three that outlive a pass, and one that does not — plus a dict that holds every live request regardless. "Two queues" is the right mental model of the policy; it is the wrong count of the state. A request whose Scheduler.add_request call has returned may be in self.waiting, or in a second waiting collection called self.skipped_waiting, and telling them apart is the difference between "this request is queued" and "this request is queued and cannot be looked at yet".

(The class keeps other request sets too — in-flight prefills, ids finished since the last send, ids preempted since the last pass. Those are bookkeeping, not queues: nothing is ever scheduled out of them. The five below are the ones that answer "where is my request right now".)

The two waiting collections are not plain lists. They come from a factory keyed on the engine's scheduling policy. Under the default first-come-first-served policy the object is a double-ended queue, so "front" and "back" are literal positions and insertion order is the whole ordering. Under the priority policy it is a heap ordered by (priority, arrival_time), and the queue's own docstrings note that prepending has no meaning there — the prepend operation collapses into an ordinary insert. Read the next table with that caveat: the position column is exact under first-come-first-served and advisory under priority.

Every move, and where it lands you

Read the position column and one rule appears: arriving puts you at the back; leaving self.running involuntarily puts you at the front. A preempted request is prepended, not appended. So is a request the pass declined to admit. Losing your place in the running set does not cost you your place in the queue — the code pays for the recomputation, not for the wait.

The last row is the one that surprises people, and it is this module's central claim in one line. update_from_output can remove a request from the waiting queue. Step 5 shows the branch.

The status field is a separate axis

Which collection a request is in and what request.status says are two different questions, and both halves read both.

Those three WAITING_FOR_* values are exactly the set that the enqueue helper tests for. If request.status is one of them, the request goes to self.skipped_waiting; otherwise to self.waiting. That is the whole routing rule, and it is why add_request returning does not tell you which queue you are in.

The grammar case shows how early that can bite. A request that asked for structured output has its status set to WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR in the Request constructor — before the scheduler has ever seen it. So its very first add_request routes it to self.skipped_waiting rather than self.waiting, and it stays there until its grammar is ready, at which point the promotion sets its status to WAITING and it can be admitted normally. Blocked is a phase, not a permanent class.

WAITING_FOR_REMOTE_KVS holds memory while making no progress. The admission loop allocates its blocks, sets request.num_computed_tokens, sets the status, and pushes it onto the skipped queue — all without running a forward pass.

WAITING_FOR_STREAMING_REQ is counted against the running limit even though it is not in self.running. The admission loop's own cap is computed as len(self.running) + self.num_waiting_for_streaming_input. A paused streaming session occupies a slot in the worker's batch, so it has to be counted; step 6 traces which flag that cap comes from.

Skipping is not a penalty box

When the admission loop declines a request, it pops it from its queue and prepends it onto the local step_skipped_waiting. When the loop finishes, that whole local queue is prepended onto self.skipped_waiting in one operation.

Then, on the next pass, the queue-selection helper under first-come-first-served returns self.skipped_waiting before self.waiting — the skipped queue is drained first, not last. So the collection whose name sounds like a sin bin is actually the high-priority one, and the reason is mechanical rather than moral: those requests are the ones whose blocking dependency may now have resolved, and the loop has to look at them to find out.

The Branch That Preempts

What triggers preemption in vLLM's scheduler?

A None. KVCacheManager.allocate_slots returns either a block set or the sentinel None, and Scheduler.schedule tests that return value with if new_blocks is not None. When the test fails, the else path evicts somebody. There is no exception, no callback, and no memory-pressure watchdog — one identity check on a return value selects the entire preemption branch.

You already know what preemption costs and why a serving engine wants it. Three things about this code are not derivable from that: preemption is triggered from the running loop and never from the admission loop; it retries the same allocation in a loop until it succeeds; and there are two different victim-selection branches, only one of which matches the rule the concept module states.

The allocate-or-preempt loop

# distilled — real names, reduced body: the allocate-or-preempt loop inside schedule()while True:  new_blocks = self.kv_cache_manager.allocate_slots(      request,      num_new_tokens,      num_lookahead_tokens=self.num_lookahead_tokens,  )  if new_blocks is not None:      break  if self.policy == SchedulingPolicy.PRIORITY:      preempted_req = max(          self.running, key=lambda r: (r.priority, r.arrival_time)      )      self.running.remove(preempted_req)      if preempted_req in scheduled_running_reqs:          preempted_req_id = preempted_req.request_id          scheduled_running_reqs.remove(preempted_req)          token_budget += num_scheduled_tokens.pop(preempted_req_id)          req_to_new_blocks.pop(preempted_req_id)          req_index -= 1  else:      preempted_req = self.running.pop()   self._preempt_request(preempted_req, scheduled_timestamp)  preempted_reqs.append(preempted_req)  if preempted_req == request:      break if new_blocks is None:  break

This sits inside the running loop from step 2, in place of the plain allocate_slots call that loop would otherwise make. Four observations, in the order the code makes them:

It is a retry loop, not a decision. Evicting one victim does not mean the current request can now be scheduled — a request needing four blocks may need four evictions. So the while True goes back and calls allocate_slots again with the same arguments, and keeps evicting until the sentinel stops coming back.

The two victim branches disagree about who loses. The else branch is self.running.pop() — the last element of the list, which is the most recently admitted request, because admission appends. That is the rule the concept module states, and it is the default. The other branch, taken only under the priority scheduling policy, is max(self.running, key=lambda r: (r.priority, r.arrival_time)) — the worst-priority request, oldest-arrival broken last. Same method, same pass, two completely different fairness stories decided by one config value.

The priority branch has to undo work. Its victim can be one the loop already scheduled earlier in this same pass, so if preempted_req in scheduled_running_reqs unwinds it: pull it out of the scheduled list, give its tokens back to token_budget with +=, drop its block record, and step req_index back one so the walk does not skip the request that just slid down into its slot. Nothing analogous exists in the else branch, because popping from the end can only ever hit a request the indexed walk has not reached yet.

A request can evict itself. if preempted_req == request: break is the base case: when the only thing left to preempt is the request currently being allocated for, the loop gives up. Control falls to if new_blocks is None: break, which exits the running loop entirely — so this pass schedules nothing further, not even for requests behind this one.

The mutation itself

# distilled — real names, reduced bodyclass Scheduler(SchedulerInterface):  def _preempt_request(self, request: Request, timestamp: float) -> None:      assert request.status == RequestStatus.RUNNING      self._free_request_blocks(request)      self.encoder_cache_manager.free(request)      self._inflight_prefills.discard(request)      request.status = RequestStatus.PREEMPTED      request.num_computed_tokens = 0      if request.spec_token_ids:          request.spec_token_ids = []      request.num_preemptions += 1      if self.log_stats:          request.record_event(EngineCoreEventType.PREEMPTED, timestamp)      self.waiting.prepend_request(request)      self.reset_preempted_req_ids.add(request.request_id)

request.num_computed_tokens = 0 is the line that costs money. "The request must be re-prefilled when it is readmitted" is not a policy written down anywhere — it is the consequence of that one assignment, because the admission loop computes num_new_tokens as request.num_tokens - request.num_computed_tokens. Set the second term to zero and the request asks for its whole prompt again on the next pass.

Note also what this method does not do: it does not remove the request from self.running. The docstring says so explicitly, and both branches above do the removal themselves — self.running.pop() in one, self.running.remove(preempted_req) in the other. The method's job is the state transition, not the list surgery. It ends by prepending to self.waiting, which is why step 3's table puts preempted requests at the front.

One preemption cancels every admission this pass

Recall the guard around the admission loop from step 2: if not preempted_reqs and self._pause_state == PauseState.UNPAUSED. preempted_reqs is appended to on every eviction. So a single preemption anywhere in the running loop makes that condition false, and the entire waiting-queue loop is skipped for the whole pass — no matter how much token_budget is left, and no matter how many requests are queued.

The two breaks and this guard are the same fact stated three times: once allocate_slots has returned None, this pass stops growing. The running loop stops walking, the admission loop never starts, and the leftover token budget goes unspent. Which is worth knowing operationally, because that pass is not idle: a full forward pass still runs for whatever survived in self.running, so a memory-pressured engine can burn several passes at full GPU cost while admitting nothing at all.

Sentinel versus exception — do not confuse these

Both of these are "the KV pool could not satisfy a request", and they have nothing else in common.

The sentinel is the designed path, and it exists precisely so the exception never fires: allocate_slots decides up front whether the pool can satisfy the request and returns None if it cannot, rather than asking for blocks that are not there. The ValueError is therefore a bug signal, not a capacity signal. If you see it in a log, do not go looking for a preemption — go looking for whatever asked the pool for blocks without checking first.

Closing the Loop

What does the scheduler's second half do?

Three things, in this order: it walks the requests the first half scheduled and folds the worker's sampled tokens into each one; it collects the ones that stopped and removes them from whichever collection they were in; and it builds the dict[int, EngineCoreOutputs] that leaves the engine-core process. It is the same class, the same thread, and the same two collections as Scheduler.schedule — one method call later.

Nothing in the LLM Serving track's scheduler step prepares you for this method, because that step teaches admission only. Everything below is new ground rather than a second pass over old ground.

The per-request loop

# distilled — real names, reduced body: the per-request loopclass Scheduler(SchedulerInterface):  def update_from_output(      self, scheduler_output: SchedulerOutput, model_runner_output: ModelRunnerOutput  ) -> dict[int, EngineCoreOutputs]:      sampled_token_ids = model_runner_output.sampled_token_ids      outputs: dict[int, list[EngineCoreOutput]] = defaultdict(list)      stopped_running_reqs: set[Request] = set()      stopped_preempted_reqs: set[Request] = set()       for req_id in scheduler_output.num_scheduled_tokens:          request = self.requests.get(req_id)          if request is None or request.is_finished():              continue          req_index = model_runner_output.req_id_to_index[req_id]          new_token_ids = sampled_token_ids[req_index] if sampled_token_ids else []          status_before_stop = request.status           stopped = False          if new_token_ids:              new_token_ids, stopped = self._update_request_with_output(request, new_token_ids)          if stopped:              if self._handle_stopped_request(request):                  self._free_request(request)              if status_before_stop == RequestStatus.RUNNING:                  stopped_running_reqs.add(request)              else:                  stopped_preempted_reqs.add(request)          if new_token_ids or stopped:              outputs[request.client_index].append(EngineCoreOutput(...))

Notice what that loop does not do: it never removes anything from a collection. It only reads request state, mutates the request object, and adds to two sets. The list surgery is a separate block, after the loop closes, in the same method.

After the loop

# distilled — real names, reduced body: still inside update_from_output      if stopped_running_reqs:          self.running = remove_all(self.running, stopped_running_reqs)      if stopped_preempted_reqs:          self.waiting.remove_requests(stopped_preempted_reqs)       return {          client_index: EngineCoreOutputs(outputs=outs)          for client_index, outs in outputs.items()      }

Ten lines, and they are the two facts this module has been building toward: the second half removes from self.running, and it removes from self.waiting.

Six things those two blocks settle

The loop iterates what the first half wrote. for req_id in scheduler_output.num_scheduled_tokens — not for request in self.running. That dict, built by Scheduler.schedule in step 2, is the join between the two halves. A request in self.running that the pass declined to schedule is not in the dict and is not visited here at all.

Request identity is re-attached to tensor rows here. The worker does not return per-request objects; it returns arrays. req_index = model_runner_output.req_id_to_index[req_id] is the lookup that turns "row 7 of the sampled-token tensor" back into "request chatcmpl-9f2c". Every per-request read below it — logprobs, NaN counts, pooler output — goes through the same index.

Removal is batched after the loop, never inside it. The loop only collects into stopped_running_reqs and stopped_preempted_reqs. Then, once, self.running = remove_all(self.running, stopped_running_reqs) — an assignment, so self.running is rebuilt rather than mutated in place. The reason is cost, not safety: the loop walks a dict, so removing from a list inside it would be legal. But self.running is a list, so each in-loop removal would be a linear scan, and a comment above the loop warns that it can carry a thousand or more requests and is a performance-sensitive path. One rebuild replaces up to a thousand scans.

The second half can remove a request from the waiting queue. status_before_stop is the local that decides. If the request was RUNNING when the pass began, it goes in one set; if it was PREEMPTED, it goes in the other, and its removal target is self.waiting. The source flags this as a rare case, but rare is not absent: a request evicted during schedule can still have a token in flight from an earlier pass, hit its stop condition, and be pulled out of the waiting queue by a method whose name says nothing about queues.

"Stopped" is not "finished". stopped only means a stop condition matched. The very next line asks a separate question — is this request done with? — and for a resumable streaming-input session the answer is no: its status is reset, it is put back on a waiting queue, and the free call is skipped. That is also why the real method captures the request's finish reason before making that call, since the call can overwrite the status the reason is read from.

Not every scheduled request produces output. The if new_token_ids or stopped guard (the real method also admits a pooler output) is what keeps a mid-prefill chunk silent. A request three chunks into a long prompt consumed budget, ran on the GPU, and advanced its num_computed_tokens — and emits nothing, because there is no token yet. The else arm of that guard asserts the invariant out loud: the engine core returns no partial prefill outputs.

The shape of the return value

That outer key is easy to miss and it explains a design choice from module 1. client_index was stamped onto the request on its way in, at the process boundary, so the return leg could be routed back to the right frontend. This is the method that uses it: the batch is fanned out by client before it ever reaches a socket, which is what makes several API-server processes behind one engine core possible.

Two more things ride out on the same structure. When the engine is configured to report them, ids of requests that finished since the last send are attached as a finished_requests set per client — so a client can learn about a finished request even on a pass where that request emitted no token. And the scheduler's stats block is attached to exactly one of the entries, because the numbers describe the engine, not any one client.

Where the memory goes back

Both halves free KV blocks, for opposite reasons, and this is the cleanest way to hold the class in your head:

Same pool, same pass, two callers. And the ordering matters: schedule runs first, so the blocks a finished request is about to release are not available to the admission loop of the pass that finishes it. They land in the pool in time for the next one.

What the Flags Touch

Which scheduler flags change which code?

Three flags reach the code you just read, and each one reaches it in a different shape. One becomes the seed value of a local variable. One becomes an attribute that is tested in exactly one place. One is never copied onto the scheduler at all and is read straight off the config object, in two loops. Knowing which is which is the difference between "I changed a flag and throughput moved" and "I know the line that changed".

--max-num-batched-tokens — the budget's seed value

The scheduler's constructor does not copy this field straight across. It resolves it:

self.max_num_scheduled_tokens takes scheduler_config.max_num_scheduled_tokens if that is set, and falls back to scheduler_config.max_num_batched_tokens otherwise. There is a separate --max-num-scheduled-tokens flag behind that first field, and when it is set it silently outranks the flag most people reach for. If you set --max-num-batched-tokens and the budget is not the number you expected, that fallback is the first thing to check.

After that, the field's whole career is two lines: it seeds token_budget at the top of every Scheduler.schedule call, and it appears in the closing assertion that the pass did not overspend. Step 2's excerpt shows both. Nothing else in the method reads it.

--max-num-seqs — an admission cap, not a batch cap

This one becomes self.max_num_running_reqs, and exactly one statement uses it to change a decision — at the top of the admission loop (it appears once more at the end of the method, in an assertion that the pass did not overshoot it):

num_running = len(self.running) + self.num_waiting_for_streaming_input, and if that is at or above the cap, the loop breaks. Two consequences fall straight out of where the test is:

The running loop never consults it. A pass that schedules 300 already-running requests does not check this flag once. That is safe rather than sloppy, because self.running cannot get above the cap in the first place: the self.running.append(request) inside the admission loop is the only append to that list anywhere in the file, and the cap guards it. The flag bounds the door, not the room.

Paused streaming sessions count against it while sitting in a different collection. That + self.num_waiting_for_streaming_input term is a counter of requests whose status is WAITING_FOR_STREAMING_REQ — requests that are in self.skipped_waiting, not in self.running, and that still occupy a slot in the worker's batch. So on a deployment using streaming input, --max-num-seqs 256 does not mean 256 requests generating. It means 256 slots, some of which may be held by sessions producing nothing.

--long-prefill-token-threshold — a per-request clamp, not a budget

This is the one that never becomes a scheduler attribute. Both loops read self.scheduler_config.long_prefill_token_threshold directly, and both apply it the same way:

if 0 < threshold < num_new_tokens: num_new_tokens = threshold

The 0 < half of that comparison is the on/off switch — the default is 0, and zero disables the clamp rather than forbidding all work. The other half means the clamp only ever reduces a request's share; it never grants one more.

Two properties are easy to get wrong:

It bounds one request, while the token budget bounds the pass. They are applied in sequence and in that order: the threshold clamp runs first, then num_new_tokens = min(num_new_tokens, token_budget). So whichever is tighter wins, and the budget always gets the last word. Setting the threshold to 512 on a 4,096-token budget means no single request eats more than an eighth of a pass. Reach for it when short requests are stuck behind long prompts — it is aimed at that latency, and it buys the fairness by making long prefills take more passes, which is a throughput cost you are choosing to pay.

It can be non-zero even if you never passed the flag. When --max-num-partial-prefills is greater than 1 and the threshold is still 0, the scheduler config fills it in as int(max_model_len * 0.04) — four percent of the model's context length. Encoder-decoder models go the other way: the config forces the threshold back to 0 and disables chunked prefill outright, so the clamp is inert for them no matter what you passed.

Where to put the print statement

That is this track's promise, so here it is per flag. All three lines are in vllm/v1/core/sched/scheduler.py.

The flag that is not here

Notice which constraint has no flag in this list: the KV pool's size. Nothing in Scheduler.schedule configures how many blocks exist — the scheduler only ever asks and gets a block set or a None back. The flags that decide how big that pool is, and how blocks are reused across requests, belong to the next module.