What is batch-sharded sampling?

It splits a decode step's batch across the tensor-parallel ranks so each rank materializes and samples only its own slice of the logits, instead of every rank holding the full batch-by-vocabulary tensor. vLLM added it in PR #50465 as part of Model Runner V2.

Why did every rank hold the full logits before?

The LM head is sharded like every other matrix, so the ranks all-gather its output to reconstruct a complete score row per request. Once that gather has happened, the simplest thing is for each rank to sample from the copy it already has - which means the work is done TP times and TP-1 results are discarded.

How much memory does it save?

The vLLM 0.29 release notes state that it cuts per-step logits memory by 1/TP. The absolute figure depends on your batch size, vocabulary size and dtype, since the tensor is batch x vocabulary entries wide.

Does it make generation faster?

Not directly - sampling was never the bottleneck. The saving is memory, and on a serving engine memory determines how many requests fit in a batch, so the effect surfaces as throughput rather than latency.

Do I need to enable it?

No. It arrives with Model Runner V2, which vLLM 0.29 made the default for all models. A few ROCm models and features that Model Runner V2 does not yet support still run on the older runner.

LLM· 2026-09-10

vLLM 0.29 — Batch-sharded sampling — What does it mean?

Jargon

Tensor parallelism: Splitting each weight matrix across several GPUs so a model too large for one card still runs. TP is the number of GPUs in that group — TP=8 means eight ranks each hold an eighth of every layer.
Rank: One GPU worker inside a parallel group. Ranks in a tensor-parallel group run the same layers in lockstep on different slices of the weights.
Logits: The raw score the model assigns to every token in the vocabulary at one position, before any sampling. One row per request, one column per vocabulary entry.
LM head: The final matrix that turns a hidden state into logits. Its output width is the vocabulary size, which is why its result is the largest per-step tensor in the model.
All-gather: A collective operation where every rank ends up with the concatenation of what all ranks produced. It is how a tensor-parallel model reassembles a sharded result — and it is what batch-sharded sampling avoids doing to the logits.
Model Runner V2: vLLM's rewritten execution layer, default for all models as of 0.29. What the model runner does
Sampler: The component that turns logits into one chosen token per request, through an ordered chain of logits processors. Inside the sampler
Decode step: One pass that computes next-token logits for the scheduled positions in the batch. Plain decoding commits one token per request per step; speculative decoding can commit several. Either way the logits tensor is rebuilt on every step.

The news. On September 9, 2026, vLLM 0.29 made Model Runner V2 the default for all models (#53183), finishing a rollout that began with pooling models. Among the pieces it brings with it is batch-sharded sampling (#50465), which the release notes describe as cutting per-step logits memory by 1/TP — a factor equal to the tensor-parallel world size. It landed alongside CUDA-graph memory profiling for KV-cache auto-sizing (#53306) in the same execution layer. Read the release →

Start in the exam hall, because the wasteful arrangement is easier to see there than in a stack trace.

Tensor parallelism cuts every weight matrix across several GPUs, so no single card ever holds the whole model. That works cleanly layer after layer — until the last one. The LM head turns each request's hidden state into a score for every token in the vocabulary, and vocabularies run to a hundred thousand entries and more. Whatever slice of that matrix a rank owns, a sampler needs the complete row before it can pick a token, so the ranks reassemble it. Now every teacher at the table is sitting with the same complete stack of papers.

Then every one of them marks all of it. Sampling itself is not expensive — walk the ordered chain of logits processors, scale the scores by temperature, keep only the top-k most likely tokens, draw one — but running it identically on every rank means all but one of those results is computed and immediately discarded. The wasted arithmetic is the smaller problem. The real cost is desk space: the full batch-by-vocabulary logits tensor has to exist on every rank, every decode step, and every megabyte it occupies is a megabyte the KV cache does not get.

Batch-sharded sampling splits the stack instead of copying it. Rank 0 takes the first slice of requests, rank 1 the next, and so on; each rank materializes and samples only its own share of the batch, which is where the release notes' 1/TP figure comes from. What has to travel afterwards is trivially small — one token id per request, a handful of bytes, against a tensor measured in hundreds of megabytes. The teachers hand back grades, not papers.

One caveat on how far to read this. The release notes state the result — per-step logits memory cut by 1/TP — and not the collectives underneath it. So read the account above as the ordinary tensor-parallel arrangement rather than a documented walk through this patch. What the release states is the outcome, and a per-rank saving of exactly 1/TP is the shape you get when a per-rank copy of the whole batch becomes a per-rank copy of one slice.

The thing to notice is where the saving lands. The win is not a faster sampler; sampling is rarely what a serving engine is short of. It gives memory back, and on a serving engine memory is what sets how many requests fit in a batch at all — the same release also added CUDA-graph memory profiling so the KV cache can be auto-sized from what is left over (#53306). This is the currency prefix caching and quantization trade in, arriving from an unexpected direction.

Tensor-parallel size	Per-rank logits tensor	Freed vs. replicated
TP = 1	~148 MiB (illustrative: batch 256, vocab 152k, fp32)	nothing to shard
TP = 2	~74 MiB (illustrative, same setup)	~74 MiB per rank
TP = 4	~37 MiB (illustrative, same setup)	~111 MiB per rank
TP = 8	~18.5 MiB (illustrative, same setup)	~130 MiB per rank

Put numbers on it, holding the batch fixed. Take a batch of 256 requests on a model with a 152,000-entry vocabulary, logits in fp32 (illustrative — both the vocabulary size and the batch size are deployment choices, not figures from the release).

One step's logits are 256 × 152,000 × 4 bytes, which is about 148 MiB, re-materialized on every decode step. Replicated at TP = 8, that is 148 MiB resident on all eight ranks at once, to produce a result that is 256 integers. Shard the batch and each rank holds 32 × 152,000 × 4 bytes, or about 18.5 MiB — the 1/TP the release notes name — freeing roughly 130 MiB on every rank — every step, for the life of the server.

Goes deeper in: Inside vLLM → Sampler → Sampling Runs on the GPU

Related explainers

vLLM 0.29 — Mamba prefill checkpoints — What does it mean? — the other memory tradeoff in the same release: how you prefix-cache a layer that keeps no per-token record to cache.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based