vLLM's Return Leg — Detokenization and Streaming, in Source

The Return Leg Starts in the Engine

What happens to a token after vLLM samples it?

It goes on a queue, and a dedicated I/O thread inside the engine-core process picks it up, msgpack-encodes it, and pushes it onto a ZeroMQ socket aimed at the API server. The thread that samples the token never waits for the socket, and the thread that owns the socket never touches the scheduler.

That is the first hop of the return leg — the last stretch of the same path module 1 opened. Module 1's step 5 handed you a token id sitting in an engine output; Sampler, Logits Processors & Structured Output ended with that id still a GPU tensor. This module follows it the rest of the way, to the data: line a client actually reads.

What is new here, and what module 1 already gave you

Out: Detokenization and Streaming is the map of this territory: seven hops, a table of symbols, and the stop-string back-channel that travels the wrong way down the pipe. This module is the descent into it. Take the map as read — nothing below re-derives the hop list.

The two things module 1 could not fit, and this module owes you:

The socket shape is not symmetric

Requests came in over a ROUTER/DEALER pair, because the client had to address a specific engine and the engine had to answer a specific client — module 1's Across the Boundary walks that handshake. Outputs go out over PUSH.

The asymmetry is not an oversight. On the way in, the routing problem is "which of several engines gets this request," and ZeroMQ's identity frame solves it. On the way out, the routing problem is already solved before the socket is reached: the engine popped a (client_index, outputs) tuple off its own queue, and client_index is a plain list index into the sockets it holds. No identity frame, no lookup, one PUSH per client.

The output I/O thread

# distilled — real names, reduced body: the thread's loop and its# zero-copy send. Socket construction, the coordinator-socket branch# (client_index == -1) and encoder setup are elided.class EngineCoreProc(EngineCore):  def process_output_sockets(self, output_paths, coord_output_path, engine_index):      """Output socket IO thread."""      encoder = MsgpackEncoder()      # Send buffers to reuse.      reuse_buffers: list[bytearray] = []      # Keep references to outputs and buffers until zmq is finished      # with them.      pending = deque[tuple[zmq.MessageTracker, Any, bytearray]]()       # We must set linger to ensure the ENGINE_CORE_DEAD      # message is sent prior to closing the socket.      with ExitStack() as stack, zmq.Context() as ctx:          sockets = [make_zmq_socket(ctx, p, zmq.PUSH, linger=4000) for p in output_paths]          max_reuse_bufs = len(sockets) + 1           while True:              output = self.output_queue.get()              if output == EngineCoreProc.ENGINE_CORE_DEAD:                  for socket in sockets:                      socket.send(output)                  break              client_index, outputs = output              outputs.engine_index = engine_index               # Reclaim buffers that zmq is finished with.              while pending and pending[-1][0].done:                  reuse_buffers.append(pending.pop()[2])               buffer = reuse_buffers.pop() if reuse_buffers else bytearray()              buffers = encoder.encode_into(outputs, buffer)              tracker = sockets[client_index].send_multipart(                  buffers, copy=False, track=True              )              if not tracker.done:                  ref = outputs if len(buffers) > 1 else None                  pending.appendleft((tracker, ref, buffer))              elif len(reuse_buffers) < max_reuse_bufs:                  # Limit the number of buffers to reuse.                  reuse_buffers.append(buffer)

Three things to notice, and none of them is "it sends a message."

The first statement of the loop is a blocking get

self.output_queue.get() is an ordinary blocking queue read. This thread spends almost all of its life parked on that line. That is the entire reason it exists: the engine's busy loop does not want to know how long a socket write takes, so it drops a tuple on a queue and goes back to scheduling. The seam between "the engine produced output" and "the output left the process" is one queue, and it is here.

If you want to prove to yourself that a stall is on the sending side rather than in the model, this is the line to instrument. A growing output_queue means the engine is producing faster than the socket drains; a queue that is empty while clients wait means the problem is downstream, in the API server.

The buffer bookkeeping exists because the send is zero-copy

send_multipart(buffers, copy=False, track=True) hands ZeroMQ pointers to memory this thread owns instead of letting it take a copy. That is cheap, and it comes with a rule: you may not touch that memory again until ZeroMQ says it is finished with it. track=True is what asks for the receipt — the returned zmq.MessageTracker flips done when the send is complete.

So the loop keeps two collections. pending holds every (tracker, ref, buffer) triple still in flight; reuse_buffers holds byte arrays ZeroMQ has released. Each iteration drains what it can from the first into the second, then pops a buffer to encode into — allocating a fresh bytearray only when there is nothing to recycle. max_reuse_bufs caps the recycling pile at one more than the number of sockets, so a burst does not leave the thread hoarding memory forever.

The everyday version: this is a coffee shop with a fixed set of reusable cups. You cannot refill a cup while a customer is still drinking from it, so you track which cups are out, take back the ones that come home, and only buy a new cup when the shelf is empty — and you cap how many spare cups you keep on the shelf.

There is a poison pill, and linger exists to deliver it

ENGINE_CORE_DEAD is a sentinel value put on the same output_queue as real outputs. When the loop pops it, it forwards it on every socket and breaks. The linger=4000 argument on those sockets is what makes that work: without a linger, closing a ZeroMQ socket can discard messages that have not been flushed, and the death notice would be exactly the message most likely to be dropped — it is the last one. The source says so in its own comment.

The consequence for debugging: when an engine dies, the API server finds out through the same socket that carries tokens, as one more message. There is no separate health channel to check.

Where this leaves the request

An EngineCoreOutputs for req-42 is bytes on a socket, stamped with an engine_index, addressed by a client_index the frontend put on the request when it submitted it. Nobody has decoded it. Nobody has turned a token id into a character.

The next step picks it up on the other side — and the method that looks like it reads the socket turns out not to.

Back Across the Boundary

How does an engine output get from the socket into the API server?

Through a queue, not through the call that looks like it reads the socket. AsyncMPClient.get_output_async is the method every later hop awaits, and its body never mentions a socket: it awaits an asyncio.Queue. A separate background task owns the socket, decodes each message, and puts the result on that queue.

This is the crossing. Everything in the previous step ran inside the engine-core process. Everything from this call onwards — for the rest of the module, and for the rest of the request — runs in the API server.

Why a task in the middle at all

Reading a socket is I/O; decoding msgpack is CPU. If the same coroutine did both, then between "bytes have arrived" and "the next recv is posted" there would be a decode nobody can overlap with anything. Splitting them means the socket task can be waiting on the next recv_multipart while the consumer is still working through the batch it was handed.

The source states the intent in one line above the split: "Perform IO in separate task to parallelize as much as possible."

Three hops, then, not one:

The task and the await

# distilled — real names, reduced body: the lazily-started socket task# and the await that consumes what it produces. Utility-output routing,# the elastic-EP notification branch and exception formatting are elided.class AsyncMPClient(MPClient):  def _ensure_output_queue_task(self):      resources = self.resources      if resources.output_queue_task is not None:          return       # Perform IO in separate task to parallelize as much as possible.      # Avoid task having direct reference back to the client.      decoder = self.decoder      outputs_queue = self.outputs_queue      output_socket = resources.output_socket       async def process_outputs_socket():          try:              while True:                  frames = await output_socket.recv_multipart(copy=False)                  resources.validate_alive(frames)                  outputs: EngineCoreOutputs = decoder.decode(frames)                  if outputs.outputs or outputs.scheduler_stats:                      outputs_queue.put_nowait(outputs)          except Exception as e:              outputs_queue.put_nowait(e)          except asyncio.CancelledError:              outputs_queue.put_nowait(EngineDeadError())       resources.output_queue_task = asyncio.create_task(          process_outputs_socket(), name="EngineCoreOutputQueueTask"      )   async def get_output_async(self) -> EngineCoreOutputs:      self._ensure_output_queue_task()      # If an exception arises in process_outputs_socket task,      # it is forwarded to the outputs_queue so we can raise it      # from this (run_output_handler) task to shut down the server.      outputs = await self.outputs_queue.get()      if isinstance(outputs, Exception):          raise self._format_exception(outputs) from None      return outputs

An exception is a value on the queue, not a raise

Look at what the two except clauses do. They do not log and continue, and they do not let the exception escape. They put it on the queue.

That is deliberate, and the comment inside get_output_async names the reason: an exception raised inside a fire-and-forget asyncio task lands nowhere — no caller is awaiting that task, so the failure surfaces as a warning at garbage-collection time, long after the server has stopped producing tokens and long after anyone could act on it. Putting the exception on the queue moves the raise onto a stack that somebody is awaiting.

Two flavours, and they mean different things:

Note the ordering of the clauses: Exception is caught first, and asyncio.CancelledError does not inherit from Exception in modern Python, so it reaches its own clause. The narrower-looking clause is the one that fires for cancellation.

The task deliberately does not hold the client

Before the inner function is defined, every attribute it needs is copied into a local: decoder, outputs_queue, output_socket, resources. The comment above them says why — "Avoid task having direct reference back to the client."

A closure that reads self.decoder captures self. A long-lived task holding self is a strong reference that outlives every other reference, so the client object can never be garbage-collected and neither can anything it owns. Copying the attributes out first breaks that edge. Where a callback genuinely needs the client back, the real code takes a weak reference and gives up if it has already gone.

You will see the identical discipline, with an almost identical comment, at the top of the next step's method. Two files, two authors, one rule — worth recognising as a house style rather than a local trick.

Not everything that arrives reaches the queue

if outputs.outputs or outputs.scheduler_stats: is a filter. Messages that carry neither — a utility-call response, an elastic-expert-parallel notification — are handled on their own branches and never reach outputs_queue. The socket is shared; the queue is not.

There is one more subtlety in how the task starts. The constructor tries to start it immediately, and falls back to starting it lazily only if there is no event loop running yet. The comment gives the reason: start it late and you can miss an EXECUTOR_FAILED message the engine sent before the first request ever arrived — an engine that died during warm-up, reporting to a listener that was not there.

What you have now, and what you do not

You have an EngineCoreOutputs — a decoded batch object, in the API server, containing one EngineCoreOutput per request the engine produced tokens for this step. Possibly hundreds of them, from hundreds of unrelated clients, in one object.

You do not have anything a client can read. Not one token id has become a character, and nothing has been routed to the coroutine that is actually serving req-42. Both of those are the next step's problem, and the second one turns out to be the harder half.

The Output Handler Loop

What connects one shared socket to thousands of separate coroutines?

A single background task and one small mailbox per request. AsyncLLM._run_output_handler is that task: it awaits a batch, slices it, and hands each slice to the output processor — which does not return the results to it. The results are pushed sideways, into a per-request collector that the request's own generate() coroutine is already waiting on.

That sideways push is the shape of the whole return leg on the API-server side, and this step is about the seam rather than the loop.

What module 1 already established

Out: Detokenization and Streaming showed this same method and made three points that still stand: it is one loop for the whole server, not one per request; the chunking is a fairness device that keeps a large batch from monopolising the event loop; and a stop string detected here travels backwards across the boundary as an abort. None of that is repeated below.

What module 1 did not show is the line that proves the fan-out, and the object on the other side of it.

The prologue is the design

# distilled — real names, reduced body: the closure prologue and the# chunk loop's contract. Logging, the stop-string abort branch and# scheduler-stats bookkeeping (module 1's story) are elided.class AsyncLLM(EngineClient):  def _run_output_handler(self):      """Background loop: pulls from EngineCore and pushes to AsyncStreams."""      if self.output_handler is not None:          return       # Ensure that the task doesn't have a circular ref back to the AsyncLLM      # object, or else it won't be garbage collected and cleaned up properly.      engine_core = self.engine_core      output_processor = self.output_processor      log_stats = self.log_stats      # We use a mutable list for logger_manager so that it can be updated      # during elastic EP scaling without creating a circular reference      # via self.      self._logger_ref = [self.logger_manager]      chunk_size = envs.VLLM_V1_OUTPUT_PROC_CHUNK_SIZE       async def output_handler():          while True:              # 1) Pull EngineCoreOutputs from the EngineCore.              outputs = await engine_core.get_output_async()              num_outputs = len(outputs.outputs)              iteration_stats = IterationStats() if (log_stats and num_outputs) else None               engine_core_outputs = outputs.outputs              for start in range(0, num_outputs, chunk_size):                  end = start + chunk_size                  # 2) Process EngineCoreOutputs.                  processed_outputs = output_processor.process_outputs(                      engine_core_outputs[start:end], outputs.timestamp, iteration_stats                  )                  # NOTE: RequestOutputs are pushed to their queues.                  assert not processed_outputs.request_outputs                   # Allow other asyncio tasks to run between chunks                  if end < num_outputs:                      await asyncio.sleep(0)       self.output_handler = asyncio.create_task(output_handler())

Everything above async def output_handler() is preparation for the same rule the previous step's socket task followed: do not let the task capture self. Same reason, near-identical comment, different file.

One detail here goes further. self._logger_ref = [self.logger_manager] is a one-element list, and the comment explains it: the logger can be swapped at runtime during elastic expert-parallel scaling. A local copy of the logger would go stale; reading self.logger_manager inside the closure would recreate the reference cycle the rest of the prologue exists to avoid. A mutable list is the compromise — the closure captures the box, and whoever needs to swap the contents writes into the box. If you have written Go or C, this is a pointer-to-pointer; if you have not, it is a labelled pigeonhole: the task remembers which hole to look in, not what was in it at the time.

chunk_size comes from an environment variable, VLLM_V1_OUTPUT_PROC_CHUNK_SIZE, whose default is 128 at v0.26.0. A batch of 400 requests is therefore processed as four slices — start takes the values 0, 128, 256 and 384, so the slices are 128, 128, 128 and 16 outputs long. if end < num_outputs is true for the first three (end is 128, 256, 384) and false for the last (end is 512), which makes three await asyncio.sleep(0) hand-backs, one after every slice but the final one — not one uninterrupted 400-request pass.

The assert is the contract

# NOTE: RequestOutputs are pushed to their queues.
assert not processed_outputs.request_outputs

OutputProcessor.process_outputs has two possible destinations for a finished output, and it picks between them per request based on whether that request has a queue. This loop asserts it always got the queue-based one.

So this one method serves both the streaming server and the batch-scripting API, and the branch that distinguishes them is per request, not per deployment. The assert is the online path stating, in executable form, which branch it expects — and it is the cheapest place to look when adding a code path that returns outputs and wondering why the server crashes instead of streaming them.

The mailbox on the other side of the push

Each request's collector is small enough to read in full:

# vllm/v1/engine/output_processor.py — RequestOutputCollector
"""
When streaming deltas, RequestOutputs are merged if the
producer gets ahead of the consumer.
"""

def put(self, output):
    if self.output is None or isinstance(output, Exception):
        self.output = output
        self.ready.set()
    elif isinstance(self.output, RequestOutput) and isinstance(output, RequestOutput):
        # This ensures that request outputs with different request indexes
        # (if n > 1) do not override each other.
        self.output.add(output, aggregate=self.aggregate)

It holds one output, not a list. If the consumer has not collected the previous one yet, the new one is merged into it rather than queued behind it — and aggregate is true exactly when the request asked for deltas, because deltas are the thing that can be concatenated meaningfully.

This is the answer to a question the next three steps will keep raising: one sampled token does not guarantee one message to the client. If the output handler runs twice before a slow consumer wakes up, two tokens' worth of text arrive as one merged output, and the client sees one frame containing both characters. Nothing is lost; the boundaries between tokens are.

Note also what put never does: block, or grow. A client that stops reading cannot make this loop wait, and cannot make it allocate. The engine's output path is decoupled from the slowest client on the server by exactly one object slot per request.

Where the token id is now

It is inside an EngineCoreOutput, inside a slice, being passed into process_outputs — and it is still an integer. The next step is the one that turns it into characters, and it does so in the same loop iteration that builds the object the client will eventually see.

Token IDs Into Text

What turns token ids back into text in vLLM?

OutputProcessor.process_outputs — one loop over the batch, and inside each iteration two calls in a fixed order: detokenize, then build the object the client will see. It is the only place in the V1 engine that is allowed to loop over the whole batch, and its own docstring says so.

Detokenization is the inverse of the step that started the whole request path.

Inverting it is not as simple as looking each id up in the vocabulary and gluing the pieces together, and the next step is entirely about why. This step is about the loop those calls sit in — and about the fact that building an output is allowed to produce nothing.

One loop, by policy

# distilled — real names, reduced body: one iteration of the batch loop.# Stats, logprobs, pooling models, streaming input and the finish/abort# branch are elided.class OutputProcessor:  def process_outputs(self, engine_core_outputs, engine_core_timestamp=None,                      iteration_stats=None) -> OutputProcessorOutput:      """      ... vLLM V1 minimizes the number of python loops over the full      batch to ensure system overheads are minimized. This is the      only function that should loop over EngineCoreOutputs.      """      request_outputs = []      reqs_to_abort = []      for engine_core_output in engine_core_outputs:          req_id = engine_core_output.request_id          req_state = self.request_states.get(req_id)          if req_state is None:              # Ignore output for already-aborted request.              continue           new_token_ids = engine_core_output.new_token_ids          pooling_output = engine_core_output.pooling_output          finish_reason = engine_core_output.finish_reason          stop_reason = engine_core_output.stop_reason           # 2) Detokenize the token ids into text and perform stop checks.          stop_string = req_state.detokenizer.update(              new_token_ids, finish_reason == FinishReason.STOP          )          if stop_string:              finish_reason = FinishReason.STOP              stop_reason = stop_string           # 4) Create and handle RequestOutput objects.          if request_output := req_state.make_request_output(              new_token_ids, pooling_output, finish_reason, stop_reason          ):              if req_state.queue is not None:                  # AsyncLLM: put into queue for handling by generate().                  req_state.queue.put(request_output)              else:                  # LLMEngine: return list of RequestOutputs.                  request_outputs.append(request_output)       return OutputProcessorOutput(request_outputs, reqs_to_abort)

The docstring is not decoration. "This is the only function that should loop over EngineCoreOutputs" is a rule imposed on future contributors, and the reason is in the sentence before it: a Python-level for over the full batch costs the same whether it does one thing or ten, so V1 pays for exactly one and asks everything that needs per-request work to fold itself into this body. When you add a feature that needs to touch every request's output, this loop is where it goes — not a second pass beside it.

An output for a request nobody is waiting for is dropped

self.request_states.get(req_id) returns None, and the loop moves on with a two-word comment: "already-aborted request."

This is the far end of an ordering module 1 established in phase 2: the frontend registers a RequestState before it submits the request, so a reply can always find its receiver. Aborting removes that state. Between the abort and the engine noticing it, the engine may still produce a step or two of output for the request — and this line is what makes those harmless. The abort is asynchronous; the drop is what makes an asynchronous abort safe.

Detokenize and build, same iteration

The two calls sit one after the other in the same body. req_state.detokenizer.update(...) mutates that request's detokenizer state; req_state.make_request_output(...) reads it. There is no second pass over the batch between them, and no queue in between — which matters, because the second call reads text the first call just appended.

update also returns something: a matched stop string, or None. The two lines that follow promote a stop-string match into a STOP finish reason the engine never issued. That back-channel is module 1's story — Out: Detokenization and Streaming traces it across both processes — and it is worth noticing here only because it explains the signature: update returns a stop string, not text. The text it produced is a side effect.

The walrus is doing real work

if request_output := req_state.make_request_output(...) reads like an assignment with an idiom around it. It is a branch: the method can return None, and when it does, nothing is put on the request's queue and nothing is appended to the list. The engine produced a token, the detokenizer consumed it, and the client is told nothing at all.

# distilled — real names, reduced body: the two early returns and the# delta bookkeeping. Pooling outputs and the n>1 parent-request branch# are elided.class RequestState:  def make_request_output(self, new_token_ids, pooling_output, finish_reason,                          stop_reason, kv_transfer_params=None,                          ec_transfer_params=None):      finished = finish_reason is not None      final_only = self.output_kind == RequestOutputKind.FINAL_ONLY       if not finished and final_only:          # Only the final output is required in FINAL_ONLY mode.          return None       if self.stream_interval > 1:          # Send output request only when          # 1. It has finished, or          # 2. It is the first token, or          # 3. It has reached the stream interval number of tokens          if not (              finished              or self.sent_tokens_offset == 0              or self.detokenizer.num_output_tokens() - self.sent_tokens_offset              >= self.stream_interval          ):              return None           if self.output_kind == RequestOutputKind.DELTA:              # Send tokens from the offset in DELTA mode.              new_token_ids = self.detokenizer.output_token_ids[self.sent_tokens_offset:]              self.sent_tokens_offset = self.detokenizer.num_output_tokens()       output = self._new_completion_output(new_token_ids, finish_reason, stop_reason)      return self._new_request_output(self.external_req_id, [output], finished)

Three ways this returns None, each a different feature:

The stream_interval branch is the one to internalise, because it is a server-side setting that changes client-visible behaviour. Set it above 1 and a streaming client gets a frame for its first token, then one frame per interval, then a final frame. The first token is always sent — that is the self.sent_tokens_offset == 0 clause — so time-to-first-token is untouched while everything after it is coarsened.

Two different ways to see nothing

Keep these apart, because they look identical from the client's side and have nothing to do with each other:

The right-hand column is the one nothing else on this site covers, and it is the next step.

Why Partial Tokens Wait

Why does vLLM's detokenizer hold bytes back?

Because a single character can be spelled across two tokens, and half a character is not a character. When the bytes accumulated so far do not yet spell a complete one, the detokenizer appends nothing to the output and leaves the bytes where they are. The next token completes the character, and both bytes' worth of text are released at once.

So a token can arrive, be counted, be charged for, and produce zero visible output. That is not a bug, a buffer flush setting, or a network artifact. It is the only correct behaviour available.

The everyday version: someone is sliding a strip of paper out from under a card, a centimetre at a time, and you read aloud whatever has become fully visible. Some centimetres reveal a whole letter. Some reveal the left half of one, and you say nothing — and then the next centimetre gives you the letter you were waiting for. You are never wrong, and you are sometimes silent.

Where the two halves come from

The LLM Internals track's Byte-Level BPE step is the setup: a tokenizer that keeps all 256 possible byte values in its base vocabulary can represent any text at all, because every file is bytes. When a stretch of text has no dedicated vocabulary entry of its own, such a tokenizer emits its raw bytes as individual tokens instead. That is byte fallback, and it is what makes an accented character or an emoji tokenizable rather than unknown.

SentencePiece-based tokenizers — Llama's and Mistral's among them — print those tokens in the literal form <0xC3>, which is the notation this walkthrough uses. GPT-2-style byte-level BPE reaches the same place by a different route and displays it differently; the decoding problem below is identical either way.

It is also what splits characters across tokens. Take é:

0xC3 on its own is not a character. It is the lead byte of a two-byte sequence, and UTF-8 says nothing follows from it until the continuation byte arrives. The walkthrough this module follows generates the word café as three tokens: caf, then <0xC3>, then <0xA9>.

The loop that appends nothing

# distilled — real names, reduced body: the per-token decode loop.# Stop-token skipping, the min_tokens offset adjustment and the# stop-string evaluation are elided.class BaseIncrementalDetokenizer(IncrementalDetokenizer, ABC):  def update(self, new_token_ids: list[int], stop_terminated: bool) -> str | None:      """      Update RequestState for the request_id by:          1) Detokenize the new token ids incrementally.          2) Evaluate stop criteria.       Return matched stop string or None.      """      if not new_token_ids:          # Skip detokenization if no new token ids.          return None       # 1) Detokenize the new token ids incrementally.      stop_check_offset = len(self.output_text)      for new_token_id in new_token_ids:          self.token_ids.append(new_token_id)          self.output_text += self.decode_next(new_token_id)       # 2) Evaluate stop strings.      stop_string = None      ...      return stop_string   @abstractmethod  def decode_next(self, next_token_id: int) -> str:      raise NotImplementedError

Read self.output_text += self.decode_next(new_token_id) carefully. There is no branch here, no "if this token is complete" test, no pending-bytes list. decode_next returns a string and it is appended. The only way a token can produce no text is for decode_next to return the empty string — and appending "" to a string is a no-op that leaves no trace.

Notice also what update returns: a stop string or None. The text never appears in the return value. It accumulates on self.output_text, which is per-request state that has existed since before the first token was generated — the RequestState module 1 registered in phase 2. This is what "incremental detokenizer" means concretely: stateful, per request, appended to.

Where the empty string comes from

decode_next is abstract. The readable implementation delegates to a helper that does the actual re-decode:

# vllm/v1/engine/detokenizer.py — SlowIncrementalDetokenizer.decode_next
def decode_next(self, next_token_id: int) -> str:
    new_tokens, decoded_text, prefix_offset, read_offset = detokenize_incrementally(
        tokenizer=self.tokenizer,
        all_input_ids=self.token_ids,
        prev_tokens=self.tokens,
        prefix_offset=self.prefix_offset,
        read_offset=self.read_offset,
        ...
    )
    self.tokens.extend(new_tokens)
    self.prefix_offset = prefix_offset
    self.read_offset = read_offset
    return decoded_text

Two offsets go in; two offsets come back and are written straight over the old ones. Those offsets are the buffer. There is no separate "pending bytes" variable anywhere in this file — holding bytes back is expressed as not moving the read window.

detokenize_incrementally decodes the whole token window into new_text, decodes the shorter prefix window into prefix_text, and ends with a two-branch return:

if len(new_text) <= len(prefix_text) or new_text.endswith("�"):
    # utf-8 char at the end means it's a potential unfinished byte sequence
    # from byte fallback tokenization.
    # If it's in the middle, it's probably a real invalid id generated
    # by the model
    return new_tokens, "", prefix_offset, read_offset

new_text = new_text[len(prefix_text):]
return new_tokens, new_text, read_offset, len(output_tokens)

Compare the two returns and the mechanism is right there.

The detection is the U+FFFD replacement character. Decoding an incomplete UTF-8 sequence does not raise — it substitutes � for the bytes it could not resolve. So the test for "this token did not finish a character" is a string test on the decoded result, and the source comment says exactly which case it is guarding: "a potential unfinished byte sequence from byte fallback tokenization." The comment also names the case it is knowingly conflating with — a genuinely invalid token id — and distinguishes them by position: a replacement character in the middle of the text is a real decoding failure, one at the end is probably just early.

The walkthrough, token by token

Row 3 is the part worth sitting with. One new byte released two bytes' worth of text, because the offsets never advanced past row 2, so row 3 re-decoded 0xC3 0xA9 together. The stream did not lose a token in row 2 and it did not double-emit in row 3. It emitted the character exactly once, at the first moment emitting it was correct.

Two hold-backs, and only one is about bytes

There is a second buffer in the same class, and it exists for a completely different reason. If a request sets stop strings and does not want them echoed back, the detokenizer withholds the tail of the text on every streamed read:

# vllm/v1/engine/detokenizer.py — BaseIncrementalDetokenizer
# Number of chars to hold back when stop strings are to be excluded
# from streamed output.
if self.stop and not self.include_stop_str_in_output:
    self.stop_buffer_length = max(len(s) for s in self.stop) - 1
else:
    self.stop_buffer_length = 0

def get_next_output_text(self, finished: bool, delta: bool) -> str:
    # We return the full output text if the sequence is finished.
    buffer_length = 0 if finished else self.stop_buffer_length
    ...
    length = len(self.output_text) - buffer_length
    last_offset = self._last_output_text_offset
    if last_offset < length:
        self._last_output_text_offset = length
        return self.output_text[last_offset:length]
    return ""

The logic: if a stop string is N characters long, then any suffix of up to N - 1 characters could turn out to be its beginning. Stream those characters now and you can no longer take them back when the match completes. So they wait.

The practical consequence is checkable and surprises people:

finished sets buffer_length to zero, which is how the withheld tail flushes at the end. So a stream with a long stop string runs permanently a few characters behind and then catches up in one go — a lag that is easy to misread as a network or client-side buffering problem.

Keep the two apart. The byte hold-back is forced by UTF-8, is never longer than a few bytes, and no setting turns it off. The stop-string hold-back is a consequence of a request parameter, is exactly max(len(s)) - 1 characters, and turning it off is a request-level choice.

Which of the two implementations you are actually running

decode_next has two real implementations, and the one you get is decided by a version comparison against the installed tokenizers library rather than by any vLLM flag:

# vllm/v1/engine/detokenizer.py — IncrementalDetokenizer.from_new_request
if USE_FAST_DETOKENIZER and isinstance(tokenizer, TokenizersBackend):
    # Fast tokenizer => use tokenizers library DecodeStream.
    return FastIncrementalDetokenizer(tokenizer, request)

# Fall back to slow python-based incremental detokenization.
return SlowIncrementalDetokenizer(tokenizer, request)

FastIncrementalDetokenizer pushes each id into a Rust DecodeStream from the tokenizers library and gets back a token or None — the same contract, the same hold-back, implemented in a dependency this walkthrough cannot quote. The Python path is the one traced above because its source is directly readable, and both satisfy the same abstract decode_next. If you want to see the offsets move under a debugger, force the slow path; if you want to know what production is doing, assume the fast one.

What the client has been sent so far

Nothing. Everything in this step happened inside one process_outputs iteration, before make_request_output was even called. The step that actually writes bytes onto an HTTP response is next — and it has its own rule about which outputs get skipped, which is not the rule you just learned.

Streaming to the Client

How does a RequestOutput become an SSE frame?

Two coroutines, in the API server, connected by the per-request collector from step 3. AsyncLLM.generate drains that collector and yields each RequestOutput; OpenAIServingChat.chat_completion_stream_generator iterates what it yields and turns each one into a data: line on the HTTP response. The stream ends with data: [DONE].

Server-Sent Events is a one-directional HTTP format: the response never closes, and each message is the literal text data: , a payload, and a blank line. That is why the two escaped newlines below are load-bearing rather than cosmetic — the blank line is the frame delimiter.

The consumer side, and what happens when a client leaves

# distilled — real names, reduced body: the queue-draining loop and the# cancellation path. Most of the signature and four of the five except# branches are elided.class AsyncLLM(EngineClient):  async def generate(self, prompt, sampling_params, request_id,                     **kwargs) -> AsyncGenerator[RequestOutput, None]:      q: RequestOutputCollector | None = None      try:          q = await self.add_request(request_id, prompt, sampling_params, **kwargs)           # The output_handler task pushes items into the queue.          # This task pulls from the queue and yields to caller.          finished = False          while not finished:              # Note: drain queue without await if possible (avoids              # task switching under load which helps performance).              out = q.get_nowait() or await q.get()               assert isinstance(out, RequestOutput)              finished = out.finished              if out is not STREAM_FINISHED:                  yield out       # If the request is disconnected by the client, generate()      # is cancelled or the generator is garbage collected. So,      # we abort the request if we end up here.      except (asyncio.CancelledError, GeneratorExit):          if q is not None:              await self.abort(q.request_id, internal=True)          raise      finally:          if q is not None:              q.close()

q.get_nowait() or await q.get() is a fast path, not a poll. If the collector already holds an output, take it without yielding to the event loop at all; only suspend when it is genuinely empty. The comment gives the motivation — "avoids task switching under load which helps performance" — and the reason it works is the collector's merge rule from step 3: whatever is sitting there is already everything that has accumulated, so taking it synchronously never skips anything.

The except clause is how a closed browser tab frees GPU memory. When a client disconnects, the HTTP framework cancels the task that is iterating this generator, which raises asyncio.CancelledError inside it; if the generator is instead abandoned and collected, Python throws GeneratorExit. Both land here, and both call self.abort(...) before re-raising.

That abort travels back across the process boundary to the scheduler, which finishes the request and returns its KV blocks to the pool — the free-list return that KV Cache Manager & Block Pool traced from the allocation side. There is no reaper, no timeout, and no polling of connection state anywhere in this path. The whole mechanism is an exception handler on a generator.

It is also why the drop in step 4 exists. Between this abort and the engine acting on it, the engine may still emit output for the request; process_outputs finds no RequestState and moves on.

The serving layer

# distilled — real names, reduced body: the per-output body of the# streaming loop. The first-chunk role frame, tool and reasoning# parsers, logprobs, usage accounting, the finish-reason branch with# its own guard, and error handling are elided.class OpenAIServingChat(OpenAIServing):  async def chat_completion_stream_generator(      self, request, result_generator, request_id, model_name, **kwargs  ) -> AsyncGenerator[str, None]:      created_time = int(time.time())      chunk_object_type: Final = "chat.completion.chunk"      num_choices = 1 if request.n is None else request.n      previous_num_tokens = [0] * num_choices      previous_texts = [""] * num_choices       async for res in result_generator:          for output in res.outputs:              i = output.index              delta_text = output.text               if (                  not delta_text                  and not output.token_ids                  and not previous_num_tokens[i]              ):                  # Chunked prefill case, don't return empty chunks                  continue               delta_message = DeltaMessage(content=delta_text)              previous_texts[i] += delta_text              previous_num_tokens[i] += len(output.token_ids)               chunk = ChatCompletionStreamResponse(                  id=request_id, object=chunk_object_type, created=created_time,                  choices=[ChatCompletionResponseStreamChoice(index=i, delta=delta_message)],                  model=model_name,              )              data = chunk.model_dump_json(exclude_unset=True)              yield f"data: {data}\n\n"       # Send the final done message after all response.n are finished      yield "data: [DONE]\n\n"

result_generator is the async generator AsyncLLM.generate() returns. The two loops are nested for a reason: one RequestOutput carries one CompletionOutput per requested completion, so a request with n=3 produces three frames per iteration, each stamped with its own index.

The skip, precisely

One bare continue above drops an output entirely, and its condition is a conjunction of three things:

All three at once, and the comment names the case: "Chunked prefill case, don't return empty chunks." A long prompt processed across several forward passes produces engine outputs that carry no generated token at all, and emitting an empty frame for each of them would be noise.

This is not the held-back token from step 5, and it is worth being exact about why. A held-back token is a token: the engine sampled it, it has a real id, and that id rides along in output.token_ids. The second clause is therefore false, the conjunction fails, and the continue does not fire. The output takes the ordinary path and a frame is written — one with "content": "" in it.

So a client streaming café over a byte-fallback tokenizer receives three frames, and the middle one carries nothing. A client that assumes every frame advances the text will handle that correctly by accident — it concatenates an empty string. A client that logs a warning on empty deltas, or treats one as end-of-stream, will not.

The round trip, closed

Module 1 opened with eleven phases and a promise that every one of them lands on a real file. Phase 11 is this step, and the path is now traced end to end:

The integer that module 5 left on a GPU is now a character on a wire. The last thing left is to check that you can run the path backwards without the diagram.