LLM Batching — Static vs Continuous Batching

Why GPUs Need Batching

What is LLM Batching?

GPUs are massively parallel processors, but serving one user at a time wastes most of that capacity. Batching groups multiple requests together so the GPU processes them simultaneously, multiplying throughput. Static batching waits for all requests in a batch to finish before starting new ones — simple but wasteful when requests have different lengths. Continuous batching (used by vLLM, SGLang, TensorRT-LLM) inserts new requests as soon as a slot opens, keeping the GPU busy. The tradeoff is memory: each request in the batch needs its own KV cache, so batch size is limited by available GPU memory.

Why GPUs Need Batching

GPUs are not like CPUs. A modern CPU has 8–32 cores optimized for sequential, low-latency work. A modern GPU has thousands of cores designed to do the same operation on many pieces of data at once — a style called SIMD (Single Instruction, Multiple Data).

That architectural choice pays off enormously for matrix multiplications — the core operation in a transformer. But it creates a catch: you need enough work to fill all those cores at once. One request, no matter how complex, won't do it.

The Factory Analogy

Imagine a factory floor with 10,000 workers. You hire them all to assemble a product. If you only give them one order at a time, 9,999 workers stand idle while one assembles. The factory is "running" but operating at 0.01% capacity.

Batching is giving the factory 10,000 orders at once — all workers stay busy, throughput skyrockets, and the cost per unit drops dramatically.

Memory-Bound vs Compute-Bound

There's a more technical reason batching matters. GPU work falls into two categories:

• Memory-bound: The GPU is waiting for data to arrive from memory. Cores sit idle.
• Compute-bound: The GPU is busy doing math. Cores are fully utilized.

With a single request, the weight matrices for each transformer layer need to be loaded from GPU memory for every token step. That's a lot of data movement — and with only one request using the result, you're paying the memory bandwidth cost for very little compute. The GPU is memory-bound.

Add more requests to the batch, and the same weight matrices get used to compute outputs for all of them simultaneously. Now you're doing much more compute per memory load. The GPU becomes compute-bound — which is where it's most efficient.

This is the decode phase

The situation just described has a name. When a model is writing its answer, it produces one word at a time. To produce a word it drags every weight out of memory. Then for the next word it drags the same weights out all over again. Each trip to memory buys exactly one word. That is decode, and it is the case that runs badly.

Reading your question is the other case, and it does not have this problem. There the model already holds all 500 words of your prompt. One trip to memory, and those weights get used on all 500 words at once. Same trip, 500 times the payoff. That is prefill.

Both are covered in Prefill vs Decode, two modules earlier.

Now here is the useful part. Batching does for decode what prefill gets for free.

One trip to memory is worth more when more words ride along with it. Prefill gets its words from one long prompt. Batching gets them from many users at the same time. The GPU cannot tell the difference — it just counts how many words go through the same weights:

Look at the last column. Decoding for one person is the worst row on the page. Decoding for 32 people costs the same trip to memory and gets 32 times as much done. That is why the number below is counted in words in flight, not in requests.

One catch. Batching shares the cost of the weights, because everyone uses the same ones. It does not share the cost of the KV cache, because each conversation has its own. So 32 users means one weight load but 32 separate cache reads.

With short conversations the weights are the bigger cost, so batching helps a lot. With long conversations the caches grow until they are the bigger cost, and adding more users stops helping. That is the point where GQA and cache quantization start to matter.

There is a harder limit too. Those 32 caches all have to fit in GPU memory at the same time, and a GPU only has so much. That is what really caps how many people a server can hold at once — Memory Limits the Batch puts a number on it.

Researcher Kipply's "Transformer Inference Arithmetic" identifies roughly 208 tokens in flight as the threshold where a typical 7B model transitions from memory-bound to compute-bound. Below that, you're wasting capacity.

The simulation shows static and continuous batching side by side. You'll use it in the next few steps to see exactly how different batching strategies affect GPU utilization.

Static Batching

Static Batching: The Blocking Problem

The simplest way to batch requests is static batching: collect N requests, run them all together as one batch, and wait until every single request in the batch finishes before accepting new ones.

How It Works

1. Collect a fixed batch of N requests (e.g., N=3: R0, R1, R2)
2. Run all N requests in parallel, one token step at a time
3. Wait until the longest request finishes
4. Only then: release all slots and start the next batch

This works, but it has an obvious flaw.

The Idle Slot Problem

Requests have wildly different lengths. A simple "What's 2+2?" might complete in 5 tokens. A detailed "Explain quantum computing" might need 300 tokens.

In a static batch:

• R0 finishes at token 20 — its slot sits idle
• R1 finishes at token 20 — its slot sits idle
• R2 needs 200 tokens — the batch can't end until it's done

For 180 token steps, two out of three GPU slots are doing nothing. The GPU is running at 33% utilization while paying 100% of the memory bandwidth cost.

Try It

In the simulation, the Static Batching side shows the batch in progress. Watch what happens to R0 (short, 2 steps) after it finishes — its progress bar stalls while R1 (long, 6 steps) is still running. The GPU utilization bar drops as more requests complete.

The GPU utilization metric at the bottom shows the real cost: the more length variation in a batch, the more wasted capacity.

The fix turns out to be conceptually simple: instead of waiting for the whole batch to finish, check after every token step whether any slot has freed up — and if so, immediately fill it with a new request.

Continuous Batching

Continuous Batching

Continuous batching — also called iteration-level scheduling — solves the idle slot problem by making a simple change: check after every single token step whether any request has finished, and if so, immediately admit a new one from the queue.

The Key Insight

Static batching treats the batch as the unit of work. Continuous batching treats the token step as the unit of work.

# Static batching (simplified pseudocode)
def run_batch(requests):
    while not all_done(requests):
        run_one_step(requests)  # all requests, together
    return results              # only return when ALL done

# Continuous batching
def run_continuous(queue):
    active = []
    while queue or active:
        run_one_step(active)            # step all active requests
        finished = [r for r in active if r.is_done()]
        for r in finished:
            active.remove(r)
            yield r.result              # return immediately
            if queue:
                active.append(queue.pop())  # fill the slot now

The loop runs once per token step. Done requests leave, new ones enter. The GPU is always at full capacity.

Step through both loops and watch the last row — the one that says what has actually been handed back to a user:

The Numbers

The Orca paper (Yu et al., 2022) introduced iteration-level scheduling. Combined with vLLM's PagedAttention (which we'll cover in the next module), this approach delivers roughly 23× higher throughput than naive static batching — without any changes to the model itself.

vLLM's 2023 blog reported 24× higher throughput than HuggingFace transformers on LLaMA and OPT models, with the same GPU hardware.

Try It

In the simulation, compare the Static and Continuous utilization bars side by side. Step through time using the controls:

• In Static: watch R0 finish early and its slot go idle
• In Continuous: when R0 finishes, R3 immediately fills the slot — utilization stays high throughout

The throughput summary at the bottom shows real-time utilization for both approaches at each time step.

What Sits On Top Of It

Continuous batching is where a modern server starts, not the whole of what it does. It answers one question: which requests share the next token step. Two other techniques answer a different question — how much work each of those requests brings to that step — and both are switched on by default in the major engines.

Chunked prefill splits a long prompt across several steps, a few hundred tokens at a time, so one large request does not hold up everyone else's tokens while it is being read.

Prefix caching skips the reading entirely for any opening text the server has already processed for someone else. A shared system prompt is usually most of that text.

So a server you deploy in 2026 runs all three at once. Continuous batching is the one you no longer have to think about.

But continuous batching at the token level introduces a new scheduling challenge: some requests are in their prefill phase (processing the prompt) while others are in their decode phase (generating tokens). These two phases have very different compute characteristics — and they compete for the same GPU resources.

Prefill vs Decode Scheduling

Prefill vs Decode Scheduling

A Quick Recap

From Prefill vs Decode in the KV Cache module: every request goes through two distinct phases.

Prefill — the model processes your entire prompt in one parallel pass. All tokens are handled simultaneously. This is compute-intensive: the GPU does a lot of math at once, similar to training.

Decode — the model generates one new token at a time, sequentially. Each step attends to all previous tokens via the KV cache. This is memory-intensive: the GPU is mainly reading cached key/value tensors rather than doing heavy computation.

These two phases have fundamentally different performance characteristics. Prefill is compute-bound — it benefits from a large batch of prompt tokens processed together. Decode is memory-bound — it benefits from many requests being decoded in parallel so the memory bandwidth cost is shared.

The Problem: Prefill Blocks Decode

When a new request arrives with a long prompt — say, a 4,000-token PDF — the serving system must run prefill for those 4,000 tokens before it can generate the first response token.

During that prefill, the GPU is fully occupied. Every other request in the batch — including ones already in their decode phase, where users are waiting for the next token — must pause and wait.

For a user whose request is mid-decode, this shows up as a noticeable stall: their stream of tokens stops, then resumes. The longer the new arrival's prompt, the longer everyone else waits.

Chunked Prefill: One Pass Carries Both

Chunked prefill reads a long prompt a slice at a time, spread over several forward passes, instead of reading all of it in one pass.

The name suggests the prompt is cut into fixed pieces — 512 tokens each, say. That is not what happens. The size of each slice is decided by the engine's token budget.

A single pass is filled in a fixed order. The engine first walks the requests it is already running and gives each one what it needs, which for a decoding request is exactly one token. Whatever budget remains after that goes to the prompt being read. The slice is the remainder, not a preset number.

Here is the part that is easy to picture wrongly. The prefill slice and the other requests' decode tokens are not separate turns. They go through the model in the same forward pass, as one batch. With a budget of 2,048 tokens, a single pass can carry three decode tokens and 2,045 prompt tokens at the same time.

Two things follow from filling the pass in that order. The requests already running are served before the new prompt is, so they are never skipped — that is why the stall goes away. And the arriving prompt can never take the whole GPU for itself, because it only ever receives what is left over.

The Cost, and Who Pays It

Chunked prefill is not free, and the token budget decides who pays for it.

Raise the budget. Each pass now carries more prompt tokens, so the arriving request is read in fewer passes and its first token comes sooner. But each pass is also longer, so the requests that were already decoding wait longer between their tokens.

Lower the budget and it goes the other way. Passes get shorter and the decoding requests get their tokens more smoothly, while the arriving prompt needs more passes before it has been read at all.

vLLM's tuning guide describes the same trade: a smaller value such as 2,048 gives better inter-token latency, because less prefill work is slowing the decodes down, while larger values give a better time to first token and better throughput.

There is no value that is best for both. Which one you want depends on whether your users are more annoyed by a slow start or by an uneven stream.

Where It Stands Now

Chunked prefill is no longer something you switch on. vLLM introduced it in the v0.4 series as an opt-in flag, --enable-chunked-prefill; since the V1 engine it is on by default wherever the model supports it. TensorRT-LLM ships the same idea under the name "in-flight batching," and SGLang has it too.

It is a concrete example of how scheduler design — not just model architecture — determines real-world serving quality.

Interleaving the two phases on one GPU is not the only answer to this problem. The other is to stop sharing the GPU at all and run prefill and decode on separate pools of machines. The LLM Serving track covers both, starting with Chunked Prefill and then the disaggregated alternative.

Memory Limits the Batch

Memory Limits the Batch

We've seen that continuous batching keeps GPU utilization high by filling slots immediately. But there's a hard physical constraint on how large a batch can actually be: GPU memory.

The KV Cache Memory Cost

Recall from the KV Cache module: every active request stores a KV cache — the key and value tensors for every attention head at every layer, for every token processed so far. This cache grows with context length.

A rough estimate for a 7B-parameter model (32 layers, 32 heads, 128 head dimension, FP16):

• Per token: 2 (K, V) × 32 layers × 32 heads × 128 dims × 2 bytes ≈ 512 KB per token
• 512-token context: ~256 MB per request
• 4,096-token context: ~2 GB per request
• 32,768-token context: ~16 GB per request

How Context Length Shrinks Your Batch

An A100 80GB GPU has roughly 40 GB available for KV cache after loading model weights (~14 GB for a 7B model in FP16, plus overhead).

The useful way to hold that number is as one pot of tokens, not as a budget per request. At 512 KB each, 40 GB holds about 80,000 tokens of cache in total — shared by everyone on the GPU at that moment.

Every row below is that same pot, divided a different way:

The diagram above illustrates the tradeoff: the same GPU memory budget can accommodate many short-context requests or very few long-context ones.

Why This Matters for Throughput

From Kipply's analysis, you need roughly 208 tokens in flight simultaneously to keep a 7B GPU compute-bound. At short context lengths, you have plenty of batch slots to achieve this. At 32K context, you're down to 2 concurrent requests — which may be below the compute-bound threshold, meaning the GPU is memory-bound again for a different reason: not enough parallelism.

This is why long-context models are expensive to serve. It's not that the model itself is slower — it's that the KV cache for a 128K-token context fills most of the GPU's memory, leaving room for at most one or two concurrent users.

This memory constraint is also why the next-generation serving technique — PagedAttention — matters so much: it lets you manage KV cache memory more efficiently, fitting more concurrent requests into the same physical memory.

Batching in Production

Batching in Production

Understanding batching theory is one thing. Seeing how it plays out at scale clarifies why these engineering choices matter so much.

Throughput vs Latency: The Core Tradeoff

Larger batches = higher throughput, but potentially higher latency per request. Every request waits slightly longer to be scheduled because the system is trying to accumulate a full batch.

Throughput (requests per second) and latency (time to first token, time to complete) pull in opposite directions:

• Aggressive batching: maximize GPU utilization, higher average latency
• Conservative batching: lower latency, underutilized GPU, higher cost per request

Production systems tune this based on SLAs. A real-time chat interface prioritizes time-to-first-token (TTFT). A batch summarization pipeline prioritizes throughput and cost.

Priority and Preemption

Not all requests are equal. A paid subscriber's request might need to preempt a free-tier request. A system under load might need to prioritize short requests to keep median latency low.

Modern serving frameworks support:

• Request priority levels — high-priority requests jump the queue
• Preemption — a running request can be paused and its KV cache evicted to free memory for a higher-priority request (then resumed later via recomputation or swapping to CPU memory)

Real-World Numbers

vLLM (UC Berkeley, 2023): Reported 24× higher throughput than HuggingFace Transformers' naive serving for LLaMA and OPT models on identical hardware — achieved through continuous batching + PagedAttention.

LMSYS Chatbot Arena: Served 30,000–60,000 requests per day from a single A100 cluster using vLLM in 2023, with batching enabling a roughly 50% reduction in GPU costs compared to earlier serving approaches.

Speculative Decoding: A Different Lever

Batching improves throughput by filling GPU capacity. Speculative decoding improves decode speed differently: a small "draft" model proposes several tokens ahead, and the main model verifies all of them in one forward pass (which is fast because verification is parallel, like prefill).

Two parts of that sentence are worth slowing down on: how a draft model "proposes several tokens ahead," and why the big model can check all of them at once. Step through one round.

The proposing is the unglamorous part. The draft model does not produce four tokens together — it generates them the same way any language model does, one at a time, feeding each guess back in before making the next. Four proposals means four passes, in order. They are cheap because the model is small, not because they were combined.

The verifying is where the saving comes from. By the time the big model runs, all four guesses already exist, so it can compute what it would have predicted at every one of those positions in a single pass — the same reason prefill can read a whole prompt at once. Ordinary decode cannot do that, because it does not know the next token until it has finished computing the current one.

If the draft is right, you get multiple tokens for the cost of one verification step — roughly 2–3× speedup on decode-heavy workloads.

The word "different" in the heading is the part worth holding on to. Both techniques get more out of one forward pass, but they fill it along different directions. Batching adds requests to the pass. Speculative decoding adds tokens per request to the same pass. Neither one uses up what the other needs, so switching both on multiplies rather than splits — which is why production systems run both.

Further Reading

• Anyscale: Continuous Batching — One of the Most Important LLM Inference Optimizations
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
• Kipply: Transformer Inference Arithmetic
• Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., 2022)