Timeouts, Retries, and Circuit Breakers

by kirupa   | filed under AI Fundamentals

Every AI feature we build leans on a model API running in somebody else's data center. Most of the time it answers. Sometimes it answers slowly, sometimes it answers with an error, and every once in a while it doesn't answer at all, and our request just sits there, politely waiting for a reply that is never going to show up.

This article is about staying calm when that happens. A timeout puts a limit on how long we wait. A retry gives a request that failed for a temporary reason a second chance. A circuit breaker notices when the failures aren't temporary anymore, and it keeps us from making a bad situation worse. These ideas have protected databases and payment systems for decades, but LLM calls stretch them in a few interesting ways.

Onwards!

Meet Pantry Pal

Imagine we are building a little app called Pantry Pal. You type in whatever is sitting in your fridge, and Pantry Pal asks a model to suggest a dinner. Here is what that looks like when everything is going well:

Here is that same happy path from Omar's side, in the Pantry Pal app:

Take a mental snapshot of this happy path, because we're about to break it. It's 6pm on a Tuesday, which is peak what-do-I-make-for-dinner time, and the model API is having a rough evening. Some of our requests come back fine. Some come back with an error. Some never come back at all.

Here is one of those requests from Omar's side:

Timeouts: Deciding How Long to Wait

Let's start with the requests that never come back. Without a timeout, our server will wait... and wait, while Omar stares at a spinner. What makes this worse is that every stuck request holds on to a connection and a bit of memory on our server. When enough of them pile up, there's no room left for new requests, and even the requests that would have succeeded can't get in:

One slow dependency has turned into an outage of our own. A timeout fixes this by putting a ceiling on the wait. After some number of seconds, we give up, free the connection, and do something more useful.

The tricky part with LLMs is picking that number. A model's response time depends on how much it writes, because output tokens come out one after another. A two-line answer might take a second, while a full recipe with a shopping list might take 30, and those 30 seconds are just a model typing. A single short timeout would kill healthy long answers, and a single long one would leave us waiting ages on requests that died early.

The way out is to use more than one clock:

• A short connect timeout (a couple of seconds), because if we can't even reach the API, waiting longer won't help.
• A time-to-first-token timeout, if we stream. A healthy stream starts producing tokens quickly, even when the full answer will be long.
• An idle timeout between chunks. If a stream goes silent for, say, 15 seconds mid-answer, something is wrong.
• An overall deadline that matches what Omar will actually put up with. If he gives up after 45 seconds, our server shouldn't wait for 120.

Here is how those clocks treat a healthy stream and a stalled one:

These numbers are illustrative. The right values come from measuring your own healthy requests and leaving a comfortable margin above them.

Retries: Giving Failures a Second Chance

Now let's look at the requests that come back with an error. A lot of errors are temporary. The API was briefly overloaded, a connection got reset, or we bumped into a rate limit. Trying again a moment later often just works.

Before we retry anything, though, we need to ask whether trying again could possibly help, because a broken request will fail the exact same way the second time. Roughly, the split looks like this:

The 429 row deserves a second look. Some providers use the same 429 status for slow down for a few seconds and for you've run out of quota. At the time of writing, OpenAI's billing and quota errors arrive as 429s with an insufficient_quota error type, and Anthropic documents that its spend-cap 429 arrives without a Retry-After header. Retrying those just burns time, so it pays to read the error body as well as the status code.

Next is how we retry. Retrying immediately in a tight loop is like repeatedly ringing the doorbell of someone who is clearly busy. The standard fix is exponential backoff, where each wait doubles: 1 second, then 2, then 4. That helps, but it hides a subtle problem. If 10,000 Pantry Pal users all failed at 6:02:00, they'll all retry at 6:02:01, and again at 6:02:03, and again at 6:02:07, in giant synchronized waves. The fix is jitter, which means picking a random wait between zero and the backoff value. Marc Brooker's 2015 post, Exponential Backoff And Jitter, showed how well this spreads the load out:

We also need to cap the retries at two or three attempts, and check what our SDK already does before writing our own loop. At the time of writing, Anthropic's official SDKs retry connection errors, 429s, and 5xx errors twice by default, with backoff. That matters because retries multiply when they stack up in layers:

So pick one layer to own retries, and turn them off everywhere else.

Two more things are worth keeping in mind. Retrying a plain suggest a dinner call is harmless, because nothing in the world changes. An agent that already called a tool like place_grocery_order is a different story, because retrying the whole turn could order the eggs twice. So we retry just the model call, or we give the order an idempotency key so the store ignores a repeat. The other thing is money. Depending on the provider, an attempt that dies halfway through can still be billed for the tokens it already produced (we saw how output tokens are billed in Tokens and How They Set Your Bill).

Here are the two ways to retry that turn:

Circuit Breakers: Knowing When to Stop

Retries assume that failures are rare and brief. By 6:15pm, the model API isn't having a blip anymore. It's down. Every Pantry Pal request now times out, retries, and fails, and all those retries pound on an API that is trying to get back on its feet.

What we want is something that notices this pattern and stops sending requests for a while. That something is a circuit breaker, and it borrows its name from the box of switches in your house. When too much current flows, a breaker trips and cuts the circuit, so a short in the toaster doesn't set the kitchen on fire. The software version was popularized by Michael Nygard's book Release It! and Martin Fowler's write-up of it. It wraps every call to the model API, and it is always in one of three states:

In the closed state, calls go through and the breaker counts recent failures. When failures cross a threshold (say, more than half of the last 20 calls), the breaker trips open, and for a cooldown of maybe 30 seconds, every call fails immediately without touching the API. Then the breaker goes half-open and lets a single trial call through. If that call works, the breaker closes. If it fails, the breaker opens again for another cooldown.

Failing fast sounds harsh, but it's a kindness. Omar hears back in milliseconds, our server keeps its connections free, and the provider gets some breathing room. Best of all, because we know right away that the API is down, we can switch to a different model or a simpler, non-AI version of the feature, which is the whole idea behind Graceful Degradation.

Two details make breakers behave well. Keep one breaker per dependency (say, per provider and model), because a big model being down says nothing about whether a small one is fine. And only count failures that are the provider's fault. If we ship a bug that sends malformed requests, those 400 errors shouldn't take a healthy provider out of rotation.

Putting It All Together

What we just did was look at how long to wait, whether to try again, and when to stop trying. In real code, these nest inside each other. The breaker sits on the outside, retries sit inside it, and every attempt carries its own timeouts within one overall deadline:

async function askModel(prompt) {
  if (breaker.isOpen()) {
    return fallbackAnswer(prompt);            // fail fast
  }

  const deadline = Date.now() + 45_000;       // what Omar will tolerate

  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      const answer = await callModelAPI(prompt, {
        connectTimeout: 2_000,
        idleTimeout: 15_000,
        deadline,
        maxRetries: 0                         // we own retries, not the SDK
      });
      breaker.recordSuccess();
      return answer;
    } catch (error) {
      if (!isRetryable(error)) throw error;   // our bug: don't trip the breaker
      breaker.recordFailure();
      if (breaker.isOpen()) break;

      const wait = retryAfter(error) ?? backoffWithJitter(attempt);
      if (Date.now() + wait > deadline) break;
      await sleep(wait);
    }
  }

  return fallbackAnswer(prompt);
}

function backoffWithJitter(attempt) {
  const cap = 1_000 * 2 ** (attempt - 1);     // 1s, 2s, 4s
  return Math.random() * cap;                 // "full jitter"
}

The helpers (callModelAPI, isRetryable, retryAfter, and breaker) are stand-ins for whatever your SDK and libraries provide. Every path out of this function ends with either an answer or a fallback, so Omar never stares at a spinner forever.

Conclusion

A model API is a dependency like any other, and it will have bad evenings. Timeouts keep a slow API from holding our server hostage, and with LLMs that means several clocks instead of one. Retries rescue temporary failures, as long as we only retry errors that can succeed, add jitter, and keep retries in one layer. Circuit breakers handle the failures that aren't temporary by failing fast and checking back in later.

What none of these decide is what Omar actually sees when the model is unavailable. Failing fast only helps if we have something good to fail over to, and that's the problem we look at next in Graceful Degradation.

Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.

Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.

Your support keeps this site going! 😇