vLLM's Sampler — the Logits-Processor Chain, in Source

Sampling Runs on the GPU

Where does vLLM actually sample the next token?

On the GPU, inside the worker, in GPUModelRunner.sample_tokens. The logits are already device tensors when that method starts, and they stay device tensors until after a token id has been chosen — the choice is made where the numbers already are, not on the CPU after a copy.

"Inside the worker" is a layer, not necessarily a separate process: module 4's Where the Executor Varies showed that under the default single-GPU executor the worker is an object in the engine-core process, and only the multiprocessing executor makes it a real process boundary. The device claim holds either way, and it is the one this module needs.

You already know both of those. This module does not re-explain why dividing by temperature widens a distribution, or why constraining the decoder beats repairing its output afterwards. It shows you where those two ideas are written — which method the chain lives in, which order its statements run in, and which if decides whether a knob is visible to a greedy request at all.

The one thing to carry in: the concept module draws the pipeline as a single ordered list of six math operations. It has to, because it is teaching the math. The code splits that list across two methods, and the split is not where you would guess. Almost everything surprising in this module is a consequence of which side of the split an operation landed on.

The method takes one argument, and unpacks the rest

# distilled — real names, reduced body: the parked-state unpack, the mask,# the sample. Speculative-decode drafting and bookkeeping elided.@torch.inference_modedef sample_tokens(  self, grammar_output: "GrammarOutput | None") -> ModelRunnerOutput | AsyncModelRunnerOutput | IntermediateTensors:  if self.execute_model_state is None:      # Nothing parked, so nothing to sample. Module 4 owns this branch.      kv_connector_output = self.kv_connector_output      self.kv_connector_output = None      return ModelRunnerOutput.with_kv_conn_output_only(kv_connector_output)   # Unpack ephemeral state.  (      scheduler_output,      logits,      spec_decode_metadata,      # ...six more parked values  ) = self.execute_model_state  # Clear ephemeral state.  self.execute_model_state = None   # Apply structured output bitmasks if present.  if grammar_output is not None:      apply_grammar_bitmask(          scheduler_output, grammar_output, self.input_batch, logits      )   with record_function_or_nullcontext("gpu_model_runner: sample"):      sampler_output = self._sample(logits, spec_decode_metadata)   self._update_states_after_model_execute(      sampler_output.sampled_token_ids, scheduler_output  )

Three facts, and each one is about where, not about what sampling means.

The signature is where the CPU's contribution enters. One parameter comes in — grammar_output — and everything else this method works on it takes out of self.execute_model_state, a tuple the previous call parked on the runner. So the argument list is a short answer to "what did the CPU compute while the GPU was busy?": the grammar bitmask, and nothing else. logits, the tensor this whole module is about, is not in the argument list at all; it was already on the device and stayed there. (The parked tuple is mixed — scheduler_output and the metadata beside it are ordinary Python objects, not device tensors. The claim is about the logits.)

Why the mask is an argument rather than something computed here is module 4's, and it is made in Two Calls, Not One — including the qualification that the second call is skipped entirely when there is nothing to sample, so this is the ordinary generative path rather than an invariant. Take it as given and read the order below instead.

The grammar mask is applied before anything a caller would call a sampling parameter. apply_grammar_bitmask(...) runs, and only then self._sample(...). Constrained decoding therefore sits upstream of temperature, of the penalties, of top-k and top-p — the schema decides which tokens are legal, and every knob after it redistributes probability among whatever survived. The concept module says a mask goes "between the distribution and the sample." The code is stricter than that: the mask goes in first, and the distribution is shaped afterwards.

self.execute_model_state = None happens before the sample, not after. Unpack, clear, then work. Nothing in the sampling path can leave the runner holding a stale parked tuple, because the clear is not conditional on success. That is why the state error module 4 taught you means "the second call never ran," and never means "the second call ran and failed."

Why "on the GPU" is a structural claim, not a performance one

_sample is a two-line hop: it reads self.input_batch.sampling_metadata and, on the ordinary path, calls self.sampler(logits=logits, sampling_metadata=sampling_metadata). That call is the interesting part, because of what self.sampler is.

The runner builds it once, in its own constructor:

# vllm/v1/worker/gpu_model_runner.py — GPUModelRunner.__init__
self.sampler = Sampler(
    logprobs_mode=self.model_config.logprobs_mode,
    use_fp64_gumbel=self.model_config.use_fp64_gumbel,
)

Sampler is a torch.nn.Module. Calling self.sampler(...) invokes its forward, exactly as calling a linear layer would. So sampling is not a post-processing step bolted onto the model's output — it is another module in the same process, operating on the same device tensors, with the same in-place discipline. apply_temperature is logits.div_. The allow-list is logits.masked_fill_. Top-k and top-p mask to -inf and then softmax. The greedy pick is logits.argmax. The comment above the returned object in vllm/v1/sample/sampler.py says it plainly: "These are GPU tensors."

That is the module's first delta, and it is checkable. Nothing in the concept module mentions a device, because for teaching the math the device is irrelevant. For reading a profile, changing a knob, or explaining why a "reproducible" request got slower, the device is the only thing that matters — and by step 5 you will have a concrete case where asking for a seed changes which kernel runs.

The example this module walks, and what is invented about it

Every stage in the right-hand panel follows one request generating The dog runs ___ over a four-token vocabulary: fast, again, slow, far. Consider a request where the caller set a logit_bias of +9 on again and a repetition_penalty of 2.0, with top_k = 2, and where again already appears earlier in the output.

Those numbers are chosen for this walkthrough. None of them is a vLLM default. They are small enough to check by hand and picked so that two real operations collide on the same token, which is what step 3 needs. A four-token vocabulary is a teaching fiction too — a real one is tens of thousands of entries wide, and the tensor is [num_requests, vocab_size].

vLLM's actual per-request defaults, from SamplingParams at v0.26.0, are the opposite of dramatic:

A request that sets nothing gets the model's raw distribution, sampled at temperature 1.0 with no filtering. Keep that as the baseline; the +9 and the 2.0× are the demonstration, not the norm.

What the rest of this module is

_sample hands off to a module whose whole job is the two-method split. The next step opens the first of those two methods and finds five operations in a fixed sequence, none of which the concept module's flow diagram contains.

The Logits-Processor Chain

What is vLLM's logits-processor chain?

Five operations in a fixed sequence inside Sampler.apply_logits_processors, every one of them mutating the same tensor, and all of them finishing before anything called temperature runs. It is a pipeline, not a bag of independent knobs — which is the difference the rest of this module lives on.

The concept module's flow diagram has temperature at position two, immediately after the raw logits. In the code, temperature has not happened yet when this method returns.

The five statements, in body order

# distilled — real names, full body: the five operations in order.# Speculative-decode output-token merging elided.def apply_logits_processors(  self,  logits: torch.Tensor,  sampling_metadata: SamplingMetadata,  predict_bonus_token: bool,) -> torch.Tensor:  bad_words_token_ids = sampling_metadata.bad_words_token_ids  output_token_ids = sampling_metadata.output_token_ids   # Apply allowed token ids.  if sampling_metadata.allowed_token_ids_mask is not None:      logits.masked_fill_(sampling_metadata.allowed_token_ids_mask, float("-inf"))   # Apply bad words exclusion.  if bad_words_token_ids:      apply_bad_words(logits, bad_words_token_ids, output_token_ids)   # Apply logits processors which can impact greedy sampling.  for processor in sampling_metadata.logitsprocs.non_argmax_invariant:      logits = processor.apply(logits)   # Apply penalties (e.g., freq_penalties).  logits = self.apply_penalties(logits, sampling_metadata, output_token_ids)   holder = sampling_metadata.thinking_budget_state_holder  if holder is not None and holder.has_tracked_requests():      holder.update_state(...)      logits = holder.apply_to_logits(...)  return logits

Read the sequence rather than the individual operations. Two hard masks, then a loop, then penalties, then — for the deployments that enable it — a thinking-token-budget clamp that masks logits once a reasoning model has spent its allowance. The for in the middle is the one worth stopping on, because it is not iterating what you would expect.

The two masks are absolute, and they run first. allowed_token_ids_mask is a per-request whitelist — anything outside it is filled with -inf, not scaled. apply_bad_words does the same for banned sequences: a request's bad_words are tokenised, and a token is masked when emitting it would complete one of those sequences given what has already been generated, which is why the function needs output_token_ids and a single-token ban does not cover it. A token pushed to -inf here cannot be resurrected by anything downstream, because every later operation is a division, a subtraction, or another mask. So the allow-list and the ban-list are the only sampling controls that are decisions rather than pressures. Everything after them is a pressure.

sampling_metadata.logitsprocs is not a list. This is the structural fact the whole module turns on, and it is invisible from the concept side. logitsprocs is a LogitsProcessors object holding two lists, and its constructor decides which list each processor goes into by calling a method on the processor itself:

# LogitsProcessors.__init__ — the partition
self.argmax_invariant: list[LogitsProcessor] = []
self.non_argmax_invariant: list[LogitsProcessor] = []
if logitsprocs:
    for logitproc in logitsprocs:
        (
            self.argmax_invariant
            if logitproc.is_argmax_invariant()
            else self.non_argmax_invariant
        ).append(logitproc)

apply_logits_processors iterates non_argmax_invariant and never touches the other list. The other list is iterated somewhere else entirely — inside sample, after temperature. So a single boolean, returned by a method on a processor class, decides which side of temperature that processor's knob runs on. Nothing sequences the two groups explicitly; the partition does it.

Penalties are a separate call, after the loop. self.apply_penalties(...) is not one of the processors in either list — it is its own method, invoked once, after the loop finishes. Which means the canonical order for the two knobs in this module's example is settled by two adjacent statements: the for loop that contains logit_bias runs, and then penalties run. That ordering is what the next step is about.

Which built-in lands in which list

Three processor classes ship as built-ins, constructed in a fixed order, and each declares its own side:

The two False answers each carry a docstring saying why, and they are worth quoting because they name the property rather than the mechanism. Logit bias: "Logit bias can rebalance token probabilities and change the outcome of argmax in greedy sampling." Min-tokens: "By censoring stop tokens, min-tokens can change the outcome of the argmax operation in greedy sampling." Both from the built-in processor classes at v0.26.0.

That is the meaning of "argmax-invariant": can this operation change which entry is largest? If yes, it has to run before the greedy pick is taken, so a greedy request feels it. If no, it can wait until after temperature and only ever affect a random draw. min_p scales a threshold relative to the peak, so the peak stays the peak — it cannot move the argmax, and so it never runs for a fully greedy batch at all.

Order within the loop is construction order. The built-ins are constructed as [MinTokens, LogitBias, MinP] and appended to their lists in that sequence, so min_tokens reaches the logits before logit_bias does. Custom processors are chained on after the built-ins, which means a custom non-argmax-invariant processor runs after logit_bias, and a custom argmax-invariant one runs after min_p. Writing a plugin does not let you choose your position; it puts you last in your group.

Two configurations where processors simply are not there

The chain is not a fixed five operations for every deployment, and both exceptions are worth knowing before you go looking for a knob that appears to do nothing.

The speculative-decoding row is worth reading precisely, because "the processor isn't built" and "your request is ignored" are two different outcomes and only the first is true. Validation catches it before sampling ever runs: SamplingParams._validate_spec_decode raises when a drafter is configured and the request set either knob —

if self.min_p > _SAMPLING_EPS or self.logit_bias:
    raise ValueError(
        "The min_p and logit_bias sampling parameters "
        "are not yet supported with speculative decoding."
    )

So a request that actually asks for one of them is rejected with an error, not quietly served without it. Note the threshold rather than a truthiness test on min_p: the default 0.0 is not > 1e-5, so ordinary requests pass. If you are debugging "my logit_bias did nothing" on a speculative-decoding deployment, you should be looking at a 400 in your client logs, not at the bias value.

Where this leaves the tensor

One tensor, mutated five times, returned to forward. Nothing has been normalised, nothing has been sampled, and temperature has not been divided by. The concept module would say we are two steps into a six-step pipeline; the code says we have finished the first of two methods.

And the two operations we ran are the two that do not commute. That is next.

Order Is the Contract

Why does the order of vLLM's logits processors change which token you get?

Because repetition_penalty branches on the sign of the logit it is handed, and logit_bias is the thing most likely to change that sign. Two operations that would commute if both were additive do not commute when one of them is a sign-dependent divide. Reverse the two statements in the previous step's method body and the same request samples a different token.

That is what makes the order a contract rather than a convention. The order is not documented in a comment or asserted in a test — it is the sequence of statements in Sampler.apply_logits_processors, and each statement mutates the tensor the next one reads.

The nine lines that make it non-commutative

apply_penalties reaches a CUDA kernel for the repetition term, and the kernel is small enough to read whole. From the sampler kernel at v0.26.0:

const bool is_repeated = prompt_mask[idx] || output_mask[idx];
if (is_repeated) {
  scalar_t logit = logits[idx];
  if (logit > 0) {
    logits[idx] = logit / penalty;
  } else {
    logits[idx] = logit * penalty;
  }
}

Two things in there, and both matter more than they look.

The sign branch is why the penalty is not a subtraction. A penalty of 2.0 applied to a logit of +6 gives +3; applied to -6 it gives -12. Both arms push the token down, which is the intent — but they push it down by different amounts, and which arm runs depends entirely on what the logit happens to be at the moment the kernel sees it. An additive bias upstream can move a logit across zero and hand the kernel the other branch. That is the whole mechanism.

prompt_mask[idx] || output_mask[idx] means the prompt counts. Repetition penalty fires on any token that appeared in the prompt or in the output so far. It is the only one of the three penalties that looks at the prompt at all — the other two, applied immediately after in the same function, read only the output side:

logits -= frequency_penalties.unsqueeze(dim=1) * output_bin_counts
logits -= presence_penalties.unsqueeze(dim=1) * output_mask

output_bin_counts is how many times each token was generated; output_mask is whether it was generated at all. So there is a third order inside apply_penalties: repetition (multiplicative, prompt and output), then frequency (additive, scaled by count), then presence (additive, flat). Three penalties, three different arithmetics, one in-place tensor, one fixed sequence.

The same request, twice

Take the example from step 1 — a request where the caller set logit_bias = +9 on again and repetition_penalty = 2.0, and again has already been generated once. The starting logits are fast: 7.0, again: 1.0, slow: 0.0, far: -1.0, with top_k = 2 and temperature held at 1.0 so it cannot be the variable. The +9, the 2.0, the top_k = 2 and the four logits are all chosen for this walkthrough — none of them is a vLLM default (the real ones are in step 1's table, and top_k's is 0, meaning off).

Same starting logits, same two operations, same temperature, same top_k. The two orders put a different token in front. The reversed row is not what vLLM does — it exists to show that the ordering is load-bearing, and the code's own order is the first row.

Be precise about what "leading candidate" means, because it changes with temperature. At temperature 1.0 this request is a random request — the < 1e-5 test in the next step is what makes a request greedy — so the token it actually emits is a weighted draw, and again at 9.5 against fast at 7.0 means again becomes the likely answer rather than the certain one. Send the identical request at temperature 0 and the swap is not probabilistic at all: argmax moves from fast to again, deterministically, every time. Either way the reordering changed the model's answer — for one of them, provably.

The asymmetry has an intuition worth keeping: the penalty is a proportional shrink, so it shrinks whatever it is given. Run the bias first and the penalty halves a boosted number, giving back roughly half the boost. Run the penalty first and it halves the small original, then the full boost lands on top of the shrunken value — the penalty ends up costing half a point instead of five. A discount applied before a coupon is not the same as a coupon applied before a discount, and for the same reason.

Why it cannot be normalised away

You could imagine a design where every processor declares a delta, the deltas are summed, and one write applies them all — that design has no order to get wrong. vLLM's does not work that way, and the previous step's body says why in its shape:

• logits = processor.apply(logits) inside a for loop. Each processor takes the tensor and returns it, having written into it.
• logits.masked_fill_, logits.div_, logits -=, and a kernel that assigns logits[idx] — every operation is in-place on the same [num_requests, vocab_size] buffer.
• No intermediate copy is kept. There is nowhere to hold a delta even if the design wanted to.

In-place mutation on one shared buffer is the reason the batch fits in memory at all — a 128-request batch over a 128,000-token vocabulary in float32 is a single tensor, and duplicating it per processor is not free. The cost of that choice is exactly the property this step is about: the sequence of writes is the semantics.

The knob that cannot flip the winner

Now the contrast that makes the lesson precise, because it is easy to walk away thinking "everything is order-sensitive."

apply_temperature is one operation: logits.div_(temp.unsqueeze(dim=1)), with temp clamped away from zero for greedy rows. Dividing every entry of a row by the same positive scalar cannot change which entry is largest. So temperature — the single most-tuned parameter in the concept module, the one every API exposes first — is structurally incapable of changing a greedy request's token, no matter what you set it to.

That is not a vLLM quirk; it is a property of division. But vLLM relies on it, and the next step shows where: sample computes the greedy pick before apply_temperature runs, precisely because the result cannot depend on it.

So the honest summary is narrower and more useful than "order matters":

What to take from this step

If you change the order of statements in Sampler.apply_logits_processors, you have changed model output for any request combining a bias with a penalty. Not the throughput. Not the numerics in a loose sense. The token. (The hard masks are the exception, per the table above — moving them relative to the penalties changes nothing, because -inf wins from either side.)

Which means the method body is an interface, and the next step is the method that calls it — and the second method, where the other half of the processor partition finally runs.

Inside Sampler.forward

What does Sampler.forward do, in order?

Four things: snapshot the logprobs, cast to float32, call apply_logits_processors, call sample. Everything the concept module calls "the sampling pipeline" happens inside the fourth of those, and the snapshot in the first is taken from a point in the pipeline the concept module has no name for.

Sampler.forward is the method that owns the split. Reading it is how you find out that the chain and the draw are two methods rather than one list.

The body, reduced to its four moves

# distilled — real names, reduced body: the four moves. The logprob-mode# override, the specific-token-ids path and dtype bookkeeping are elided.def forward(  self,  logits: torch.Tensor,  sampling_metadata: SamplingMetadata,  predict_bonus_token: bool = False,) -> SamplerOutput:  # NOTE(woosuk): Use the original logits (before any penalties or  # temperature scaling) for the top-k logprobs.  num_logprobs = sampling_metadata.max_num_logprobs  raw_logprobs: torch.Tensor | None = None  if num_logprobs is not None:      if self.logprobs_mode == "raw_logprobs":          raw_logprobs = self.compute_logprobs(logits)      elif self.logprobs_mode == "raw_logits":          raw_logprobs = logits.to(torch.float32)   # Use float32 for the logits.  logits = logits.to(torch.float32)   logits = self.apply_logits_processors(logits, sampling_metadata, predict_bonus_token)  # Sample the next token.  sampled, processed_logprobs = self.sample(logits, sampling_metadata)  if processed_logprobs is not None:      raw_logprobs = processed_logprobs   logprobs_tensors = self.gather_logprobs(raw_logprobs, num_logprobs, token_ids=sampled)  # These are GPU tensors.  return SamplerOutput(      sampled_token_ids=sampled.unsqueeze(-1),      logprobs_tensors=logprobs_tensors,  )

The cast happens once, before any processor. logits.to(torch.float32) runs before apply_logits_processors is called, so every penalty, every mask, every division and the draw itself run in fp32 regardless of whether the model ran in bf16 or fp16. What that buys is narrow and worth stating narrowly: the sampling arithmetic is dtype-independent, so a penalty is not quietly less precise on a bf16 deployment. It does not make two deployments agree — the logits arriving at this line already differ, because the forward pass that produced them ran in a different precision. The cast standardises what happens next, not what came in.

The logprobs you get back are, by default, from before the chain. raw_logprobs is computed at the top — and the source comment says exactly why, naming the change from the previous engine generation: "Use the original logits (before any penalties or temperature scaling) for the top-k logprobs. This is different from the V0 sampler, which uses the logits that is used for sampling (after penalties and temperature scaling)."

That is a real surprise for anyone reading logprobs to explain a sampled token. Under the default, the token was chosen from one tensor and the logprobs describe a different one. The two disagree by exactly the amount your knobs did. Which point the logprobs come from is a server-level choice, not a per-request one — step 6 puts that flag in its place — and the four possible values are raw_logprobs, raw_logits, processed_logprobs, processed_logits, where "processed" means after the whole chain including temperature and top-k/top-p.

sample is the second method, and it is where the other processor list runs. Same file, called on the line after the chain returns.

Inside sample: greedy comes before temperature

# Sampler.sample — reduced body, real order
if sampling_metadata.all_random:
    greedy_sampled = None
else:
    greedy_sampled = self.greedy_sample(logits)
    if sampling_metadata.all_greedy:
        return greedy_sampled, processed_logprobs

# Apply temperature.
logits = self.apply_temperature(
    logits, sampling_metadata.temperature, sampling_metadata.all_random
)

# Apply logits processors that only apply to random sampling
# (argmax invariant)
for processor in sampling_metadata.logitsprocs.argmax_invariant:
    logits = processor.apply(logits)

# Apply top_k and/or top_p.
random_sampled, processed_logprobs = self.topk_topp_sampler(
    logits, sampling_metadata.generators,
    sampling_metadata.top_k, sampling_metadata.top_p,
)

if greedy_sampled is None:
    return random_sampled, processed_logprobs

sampled = torch.where(
    sampling_metadata.temperature < _SAMPLING_EPS,
    greedy_sampled,
    random_sampled,
    out=greedy_sampled,  # Reuse tensor
)

Five statements, and the first one is the one people get wrong.

greedy_sample runs before apply_temperature, and it is an argmax over the post-chain logits. So the greedy pick sees the allow-list, the bad-words mask, min_tokens, logit_bias and all three penalties — and does not see temperature, min_p, top_k or top_p. Step 2's partition is exactly this line. A processor whose is_argmax_invariant() returned False had to be upstream of this call, and a processor that returned True is placed downstream of it, where a greedy request can never feel it.

That answers a question the concept module cannot: which sampling parameters does temperature=0 ignore? Not "the random ones" vaguely — precisely the ones in the argmax_invariant list plus top_k and top_p, because every one of those is applied after the greedy pick has already been taken.

"Greedy" is not a mode flag. It is temperature < 1e-5. _SAMPLING_EPS is 1e-5 and appears twice: in apply_temperature, and in the torch.where at the bottom. Nothing on the wire says "greedy" — a caller sets temperature=0, and this comparison is where that becomes a different algorithm.

A mixed batch computes both paths and then chooses per row. all_greedy and all_random are batch-level booleans, so the two early exits only fire when the whole batch agrees. When it does not, the method computes the argmax over every row and runs the full random path over every row, then torch.where selects per row from the two results. There is no per-request branch anywhere in this method, and there is no ragged tensor: uniform work over the whole batch, and one selection at the end. It is the same trade module 4's discard_request_mask made — do the work for everyone, throw away what you do not need, because a branch costs more than the arithmetic.

Which explains a 1.0 that otherwise looks arbitrary. apply_temperature opens with temp = torch.where(temp < _SAMPLING_EPS, 1.0, temp) when the batch is not all-random. The greedy rows in a mixed batch have temperature=0, and dividing by zero would poison the tensor for everyone — so their temperature is replaced by 1.0 purely so the shared division is safe. Those rows then travel the whole random path and their result is discarded by the torch.where. The 1.0 is not a default; it is a placeholder standing in for a value that is about to be ignored.

out=greedy_sampled reuses a tensor. The comment says so. The selection writes into the buffer the argmax already allocated rather than allocating a third one — a detail worth knowing only because it means you cannot inspect greedy_sampled after this line and expect the greedy answer. It now holds the mixed answer.

The two methods, side by side

The right-hand column is the concept module's six-step pipeline, minus the first step and plus a batch. The left-hand column is the part the concept module does not have, and it is where every non-commutative operation lives.

Next: what the right-hand column costs when one batch contains both kinds of request.

Greedy and Random Side by Side

How does one vLLM batch serve greedy and random requests at the same time?

By computing both answers for every row and selecting per row at the end. Step 4 showed the torch.where that does the selecting. This step is about what that costs, and about the two things the concept module's word "randomly" hides: that the random draw is also an argmax, and that asking for a reproducible draw changes which kernel runs.

Both paths end in an argmax

The concept module's last step is "draw one token randomly from the surviving probabilities." The obvious way to write that is torch.multinomial. vLLM does not use it, and says why in the docstring of the function it uses instead:

# random_sample — real names, reduced: annotations and the dtype-mismatch
# arm of the divide are elided.
def random_sample(probs, generators, use_fp64_gumbel=False):
    """Randomly sample from the probabilities.

    We use this function instead of torch.multinomial because torch.multinomial
    causes CPU-GPU synchronization.
    """
    q = empty_exponential_noise_like(probs, use_fp64_gumbel)
    if len(generators) != probs.shape[0]:
        q.exponential_()
    if generators:
        for i, generator in generators.items():
            q[i].exponential_(generator=generator)
    return sample_with_exponential_noise(probs, q)


def sample_with_exponential_noise(probs, q):
    scores = probs.div_(q)
    return scores.argmax(dim=-1).view(-1)

Draw one exponential random number per vocabulary entry, divide the probabilities by that noise, take the largest. That is the exponential-race form of the Gumbel-max trick — a standard way to turn "sample proportionally to these weights" into "take an argmax", which matters here because an argmax is a plain reduction the GPU can do alone. The docstring gives the reason for avoiding the obvious call in exactly one word: torch.multinomial "causes CPU-GPU synchronization." It does not say what gets copied, and neither will this page; what it costs is a synchronisation point on a path that runs once per decode step.

So the honest description of the two paths is not "one is deterministic and the other is random." Both are argmax. The greedy path takes the argmax of the logits; the random path takes the argmax of the probabilities divided by noise. That is the entire difference, and it is why they can share a batch, a tensor, and an output buffer.

What a mixed batch pays

Step 4 established that a mixed batch computes both paths. The cost of that is a separate question, and it has three answers rather than one, because the two batch-level predicates give two of the three cases a short circuit:

Read the bottom row as the interesting one. One temperature=0 request in an otherwise-random batch costs the batch a full-vocabulary argmax it will use for exactly one row. And one random request in an otherwise-greedy batch costs the batch the entire random path — temperature, the argmax-invariant processors, the sort-or-Triton masking, the noise draw — for rows whose results are then discarded.

That is the same bargain module 4's discard_request_mask struck for prefill: uniform work over the whole batch beats branching, because a branch on a GPU means divergence and a second kernel launch. It is worth knowing mainly so that a throughput change after a client started sending temperature=0 has an explanation that is not mysterious.

What top-k and top-p do to the tensor

Neither one shortens anything. Both mask to -inf and leave the shape alone, which is what lets thirty requests with thirty different top_k values live in one [num_requests, vocab_size] tensor:

• top-k: sort the row, find the value at position vocab_size - k, and masked_fill_ everything below it. Per-row k comes from a tensor, so each row gets its own cut.
• top-p: softmax the sorted row, cumsum, mask everything whose cumulative mass falls outside the nucleus — then force top_p_mask[:, -1] = False, commented # at least one. So top-p can never mask every candidate, however small p is.
• Both write into the sorted copy and then scatter_ it back into the original positions.

Which implementation runs depends on the batch size. On CUDA the dispatch is if HAS_TRITON and logits.shape[0] >= 8 — a Triton kernel for eight or more rows, and the PyTorch sort path below that, with a comment saying it is there for small batches. And if both k and p are unset, apply_top_k_top_p returns the logits untouched on the first line, which is what the defaults top_k = 0 and top_p = 1.0 buy you: not a cheap filter, but no filter at all.

Determinism costs two things, and neither is randomness

SamplingParams.seed is per-request. Setting it creates a torch.Generator for that request, filed in sampling_metadata.generators under its batch index. Two separate consequences follow, and both are visible in code you have now read.

One: the noise is drawn row by row. Look at the loop again — for i, generator in generators.items(): q[i].exponential_(generator=generator). The source's own comment is TODO(woosuk): This can be slow because we handle each request one by one. Optimize this. A batch with no seeds fills q in one batched call. A batch with some seeds does the batched fill and the per-row loop, because if len(generators) != probs.shape[0] is true whenever the seeds are partial.

Two: a non-empty generators dict disables the fused kernel. TopKTopPSampler.forward_cuda opens with a guard that sends the call back to the PyTorch-native path when there is nothing to filter or when per-request generators are present, and logs the reason: "FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation." So one seeded request in a batch takes the whole batch off the FlashInfer path.

Greedy needs neither. argmax over the same logits gives the same index every time, with no generator and no noise — which is the practical reason temperature=0 is the cheap way to get repeatability, and seed is the way to get repeatability with a distribution.

There is also a server-wide switch on the precision of that noise. use_fp64_gumbel makes q float64 instead of matching probs, and its own documentation states the trade: FP64 "preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs." Turning it on also forces forward_native, for the same reason a seed does. That is a per-server decision about sampling behaviour that no request can make or override — which is the subject of the next step.

Where this leaves the token

One integer per request, still on the device, wrapped in a SamplerOutput whose comment says # These are GPU tensors. The runner then folds those ids back into per-request state and hands them to the bookkeeping step module 4 traced.

Every knob that shaped that integer arrived from one of two places, and they are not interchangeable. That is the last thing to get precise about.

Request-Level vs Server-Level

Which vLLM sampling controls are per-request, and which are set once for the server?

There are three tiers, not two. Per-request fields on SamplingParams. Server-level engine configuration that no request can override. And in between, a closed list of six request-level parameters whose defaults the server sets — which is where most "I never set that" surprises come from.

Getting this split right is the difference between a five-minute fix and an afternoon. A knob in tier one is a client change. A knob in tier two is a restart. A knob in tier three looks like tier one and behaves like tier two until someone overrides it.

Tier 1 — per-request, on SamplingParams

Every row below is a field a single request sets, with its real default at v0.26.0 and the point in the chain where it lands. The "lands at" column is the payoff from the previous five steps: it tells you whether a greedy request feels the knob at all.

The structured_outputs row carries the schema itself — json, regex, choice, grammar, json_object, structural_tag. The schema is per-request. The engine that compiles it is not, and that is tier two.

Tier 2 — server-level, one value for every request

These are engine configuration. A request cannot set them, and changing one means restarting the server.

Two rows deserve a second look because they are the ones people misfile.

--seed is not a per-request seed, and the two do different jobs. SamplingParams.seed gives one request a reproducible draw. --seed exists so that tensor-parallel workers, each running the sampler on its own shard, agree — its documentation says so directly. Setting --seed does not make your requests reproducible, and setting SamplingParams.seed does not make your ranks agree.

--logprobs-mode is the reason a bug report can be unreproducible. Two operators reading the same request's logprobs on servers started with different values for this flag are looking at numbers taken from different points in the pipeline. There is nothing in the response that says which.

Tier 3 — the server sets the defaults for six request-level parameters

This is the tier that is easy to guess wrong, so it is worth reading the code rather than reasoning about it. --generation-config decides where per-model sampling defaults come from: "auto" (the default) loads them from the model directory, "vllm" loads nothing and uses vLLM's own neutral defaults, and a path loads them from there. --override-generation-config merges values on top.

The list of parameters this can reach is closed. ModelConfig.get_diff_sampling_param names it:

available_params = [
    "repetition_penalty",
    "temperature",
    "top_k",
    "top_p",
    "min_p",
    "max_new_tokens",
]

Six entries, and max_new_tokens is renamed to max_tokens on the way through because that is Hugging Face's name for vLLM's field. So:

Which turns two vague debugging questions into a lookup:

• "Temperature is 0.7 and nobody set it." Plausible. It is in the list — check generation_config.json in the model directory, or pass --generation-config vllm to opt out of it.
• "A logit_bias is being applied and nobody set it." Not possible through this path. It is not in the list, so either the request carries it or a custom processor loaded by --logits-processors is doing something the name logit_bias does not cover.

Two configurations that remove tier-1 knobs entirely

Worth repeating here, because it is a level question rather than a chain question. With speculative decoding enabled, only the min-tokens processor is built, and vLLM logs "min_p and logit_bias parameters won't work with speculative decoding." With a pooling model, no processors are built at all, and passing custom ones raises.

In the speculative-decoding case the outcome for a caller is an error, not a silent no-op: SamplingParams._validate_spec_decode raises when a request sets min_p above 1e-5 or a non-empty logit_bias while a drafter is configured. So a server-level decision does not quietly void a request-level field here — it makes the field unusable, loudly. That is the better of the two behaviours, and it is worth knowing which of your tier-1 rows have that protection and which just stop mattering.

The line to remember

Per-request fields choose how a token is picked from the distribution the model produced. Server-level flags choose what code exists to pick it with — which processors were constructed, which grammar compiler runs, which kernel does the masking, what precision the noise has, and what the logprobs you get back are even describing.

That distinction is not cosmetic. Every tier-2 row above can change output or performance for requests that did not ask for anything.