A city bus costs roughly the same to drive across town whether it carries one passenger or forty. The driver, the fuel, and the route don't change much. What changes is how many people got where they were going for that cost.
The machines that run language models have a very similar property, and request batching is how we take advantage of it. In this article, we'll see why serving many requests together is nearly as cheap as serving one, why the first obvious way of doing it wastes a lot, and how modern model servers fix that.
Onwards!
One GPU, Many People
Let's say we run a model on our own GPU server (GPUs are the chips that do the heavy math for most models) to power a writing assistant called Quill. At any moment, several people are waiting on answers: Lena wants a subject line, Omar wants a paragraph rewritten, Kai wants a long product description, and more requests keep arriving.
The simplest way to serve them is one at a time. Finish Lena, then start Omar, then Kai. That's easy to reason about and terribly wasteful, and to see why, we need to look at what the GPU does to produce each token.
What One Step of Decoding Costs
A model writes its answer one token at a time. For each new token, the GPU runs the whole model once. Running the model means doing math with its weights, the billions of numbers that make up the model, and those weights live in the GPU's memory.
As it turns out, doing the math is fast. Fetching the weights from memory to where the math happens is the slow part. Let's say Quill's model has about 14 GB of weights (a 7-billion-parameter model stored at 2 bytes per number). To produce one token, the GPU streams roughly all 14 GB through its compute units. Then it does it again for the next token.
Now, what if Lena's, Omar's, and Kai's requests are all in the middle of decoding? The GPU could fetch the weights once and use them for all three requests in the same pass. The math triples, but the expensive memory trip happens only once.
Here is that difference:
This is why batching works so well for language models. When decoding a single request, the GPU spends most of its time waiting on memory, with its math units mostly idle. Batching fills those idle math units with other people's work. Up to a point, a batch of eight or sixteen requests takes only a little longer per step than a batch of one, so total tokens per second goes way up.
That's the bus. Here's what it looks like with our passengers on it:
The First Try: Waiting for Everyone
The obvious way to batch is to gather a group of requests, run them together until they're all done, then gather the next group. This is called static batching.
It has a problem, and it comes from the fact that answers have different lengths. Lena's subject line is 10 tokens. Omar's paragraph is 120. Kai's product description is 400. In a static batch, all three start together, and the batch isn't done until Kai's 400th token.
Here is that batch over time:
Look at the hatched area. Lena's and Omar's slots sit empty for most of the batch, so the GPU is doing less useful work than it could. And Priya and Theo, who arrived a moment after the batch started, can't get in until Kai is completely done. Lena finished long ago, but her spot does nobody any good.
The Fix: Let People on and Off at Every Stop
The fix is to stop treating the batch as a fixed group. Since the model runs one step at a time anyway, the server can reconsider the batch at every step. When a request finishes, it leaves right away. When there's room, a waiting request joins on the very next step.
This is called continuous batching (also known as in-flight batching or iteration-level scheduling). It was described in the Orca paper from OSDI 2022, and it's now standard in open-source servers like vLLM.
Here is the same moment with continuous batching:
Going back to our bus, a static batch is a tour bus that won't let anyone off until the whole tour ends. A continuous batch is a regular city bus that stops constantly, so people hop off at their stop and new riders climb on.
The Catch: Memory and Patience
If batching is so good, why not batch a thousand requests at once?
As it turns out, memory is the first wall we hit. Every request in the batch carries its own KV cache, which is the model's saved notes about that request's tokens, and it grows with every token. Long prompts and long answers take a lot of it, and when memory runs out, the batch can't grow. The vLLM paper found that servers were wasting much of this memory through fragmentation, and its PagedAttention technique (which borrows the idea of memory pages from operating systems) packs it more tightly so more requests fit at once.
The other limit is each person's patience. A bigger batch does more total work per step, so each step takes a bit longer. Total throughput goes up, but every individual user sees their own tokens arrive a little more slowly. New requests joining the batch also need their prompts processed (prefill), which can briefly slow everyone else's step and affect time to first token. Servers expose settings, like a maximum batch size or maximum tokens per step, to pick a balance between how much total work gets done and how snappy each user's experience feels.
Batching From the App Side
So far we've been talking about what a model server does internally. If we call a hosted model API, the provider is doing this batching for us, across many customers at once.
There are still two kinds of batching we control from our own code:
- Some endpoints accept many inputs in one request. Embedding APIs, for example, typically let us send a list of texts in one call instead of one call per text. Fewer round trips means less overhead.
- Providers also offer batch APIs for work that doesn't need an answer right away. We upload a file of requests, results come back within a longer window, and the price is usually lower. That's a different trade (giving up speed for cost), and it's the heart of choosing between batch and real-time inference.
Conclusion
Producing a token means streaming the model's weights through the GPU, and that memory trip costs about the same whether it serves one request or several. Batching lets many requests share each trip, which is why serving lots of users at once is so much cheaper per token than serving them one by one.
Static batching captures some of that benefit but wastes slots on finished requests and makes new ones wait. Continuous batching fixes both by reshuffling the batch at every step, like a city bus that stops everywhere. The limits are memory for each request's KV cache and the slight slowdown each user feels as the batch grows. For Quill, that meant Lena gets her subject line and leaves, and Priya climbs aboard on the very next step!
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
