Every system has a speed limit. A coffee shop can make so many lattes per minute. A model server can produce so many tokens per second. A model provider will only accept so many tokens per minute from our account.
Most of the time we never notice, because the traffic is comfortably under the limit. Then one day it isn't. What our system does in the next few minutes decides whether a few people get a polite try again in a bit or everybody gets nothing at all.
Backpressure and load shedding are the two habits that keep it on the polite side. In this article, we'll watch a system get overwhelmed, see why the obvious response makes it worse, and then add both habits one at a time.
Onwards!
Ten O'Clock on Sale Day
Let's say we run a concert-ticket site, and we added an AI helper named Encore. Fans ask it things like Which sections still have two seats together? and Can I transfer a ticket to my sister? On a normal morning, Encore handles a steady trickle of questions.
Here is Encore on one of those normal mornings, sitting in the corner of the ticket site:
Today, a very popular band's tour goes on sale at 10:00. At 9:59, a few hundred thousand fans are refreshing the page. Here's roughly what 10:00 looks like from Encore's point of view:
Let's put numbers on it. These are illustrative, chosen so the arithmetic is easy. Encore's servers, together with the token limits on our model provider account, can finish about 50 questions per second. At 10:00, questions start arriving at 200 per second.
What Happens If We Accept Everything
The most natural code in the world is to accept every request and put it in line. Nobody gets turned away, so that seems kind.
Here's the problem. If 200 questions arrive each second and we finish 50, the line grows by 150 every second. After one minute, there are 9,000 questions waiting. A question that joins the back of that line will wait 9,000 ÷ 50 = 180 seconds before Encore even starts on it.
Here is that line, a minute in:
Three minutes is longer than anyone will stare at a chat box. So fans give up, and many of them hit refresh and ask again. Their original question is still sitting in the line, so now each of those fans is in line twice.
Look at the dashed cards near the front of the line. Those are questions from people who already left. When Encore finally gets to them, it pays the model provider to write a careful answer that nobody will ever read, while fans who are still waiting sit further back.
By accepting everything, we didn't serve more people. We served the same 50 per second, made most of them wait minutes, and spent a chunk of our capacity on people who were gone. As it turns out, a system that's slightly overloaded can stop doing useful work almost entirely. Google's SRE book has a whole chapter on this pattern, Handling Overload.
Backpressure: Say "Not Yet" Early
The first fix is to stop the line from growing forever. We give the queue a maximum size. When it's full, new requests are turned away immediately, with a message that tells the sender to slow down.
On the web, there are two standard ways to say this, and they mean slightly different things. When the whole service is temporarily too busy, the usual answer is 503 Service Unavailable (defined in RFC 9110). When one particular caller is sending more than its share, the answer is 429 Too Many Requests (defined in RFC 6585). Either one can carry a Retry-After header that says how many seconds to wait before trying again. Encore's queue filling up on sale day is the whole service being busy, so Encore answers with a 503.
That push back toward the sender is called backpressure. The pressure inside the system flows backward to whoever is adding work, so they slow down instead of piling on. Here's our queue with a limit:
With a maximum of 500 waiting, the longest wait is about 500 ÷ 50 = 10 seconds. That's a wait a fan can live with. Fans who get the 503 learn right away that they should try again shortly, instead of staring at a spinner for three minutes.
Here is what a turned-away fan sees in Encore's chat:
The clients have a job here too. If every rejected fan's app retries at exactly the 5-second mark, all of them come back at once and we're right back where we started. Well-behaved clients wait a little longer after each failure (exponential backoff) and add a random wobble to the delay (jitter), so the retries spread out. The AWS Builders' Library article on timeouts, retries, and backoff with jitter explains why that randomness matters so much.
Backpressure also chains. Our model provider does the same thing to us: when we exceed our account's rate limit, it sends us a 429. Encore shouldn't respond to that by retrying instantly in a tight loop. It should pass the pressure along, slowing its own intake or answering its own callers with a 503. Here is that chain:
Load Shedding: Choose What to Drop
Backpressure keeps the line short, but it treats every request the same. The first 500 get in and everyone else gets a 503, regardless of what they asked. That's fair in a first-come sense, but it isn't smart.
Load shedding is the practice of deliberately dropping some work so that the most important work still finishes on time. The difference from plain backpressure is the word deliberately. We decide which requests are worth our limited capacity.
For Encore on sale day, a few rules make a big difference:
- Questions from fans who are in the middle of checking out get priority. A fan asking whether their payment went through matters more than a fan asking what the opening song will be.
- Requests whose callers have already given up get dropped. If a fan's app closed the connection, or the request has waited longer than the app's own timeout, there's no point answering it.
- Unusually expensive requests get turned away first. A request with a huge pasted document costs many times more tokens than a one-line question, so it's the first to go when we're over capacity.
Here is Encore sorting requests at the door:
The other important detail, and this is a HUGE detail, is that shedding has to be cheap. If Encore runs retrieval and calls the model before deciding to drop a request, it has already spent the capacity it was trying to save. The decision has to happen at the door, using information we already have: who is asking, what part of the site they're on, how long the request has been waiting, and roughly how many tokens it will need. The AWS Builders' Library article Using load shedding to avoid overload goes deep on this.
Counting Tokens, Not Requests
There's one twist that's specific to AI systems. A web server can often treat requests as roughly equal in cost. A model server can't.
One fan asks Is the 9pm show sold out? and that's a few hundred tokens. Another pastes a long email thread and asks Encore to summarize it, and that's tens of thousands of tokens. If our queue limit is 500 requests, fifty giant requests can occupy the same slots as fifty tiny ones while costing a hundred times more.
Here are those two requests side by side:
So for admission and shedding, Encore estimates each request's tokens (prompt plus a reasonable allowance for the answer) and keeps the queue's total tokens under a limit, not just its count of requests. That lines up with how our model provider rate-limits us anyway, which is usually in tokens per minute as well as requests per minute.
Dropping Gracefully
Shed doesn't have to mean nothing. Between a full answer and a 503, there's a lot of room:
- send a shorter answer by lowering the reply's token budget
- route low-priority questions to a smaller, cheaper model
- serve a stored answer for common questions
- show a helpful static page (Here's how ticket transfers work) instead of a model response
These soften the blow for the people we can't fully serve. The general idea is called graceful degradation, and it gets a closer look of its own in Graceful Degradation. Switching to a backup model is the fallback model idea we saw earlier.
Conclusion
When work arrives faster than a system can finish it, an unlimited queue doesn't help anyone. Every request waits longer, people give up and retry, and the system spends its capacity answering people who already left.
Backpressure fixes the runaway line by capping the queue and telling senders to slow down with a quick 503 or 429, and good clients respond by backing off with some randomness. Load shedding goes further by choosing which work to drop, at the door and cheaply, so the most important requests still finish on time. In AI systems, we count that work in tokens, since one request can cost a hundred times more than another.
For Encore, that meant the 10:00 rush turned into a ten-second wait for fans checking out, a polite try again soon for fans asking about setlists, and no money spent on answers to empty rooms. Your sale day will look different, but it's much easier to decide these rules calmly on a Tuesday than at 10:01!
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
