Fallback Models

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

Every model provider has bad days. An API that answered every request this morning might start returning errors this afternoon, because of an outage, a traffic spike, or our own account hitting its limits. When our app depends on exactly one model at exactly one provider, its bad day becomes our bad day.

A fallback model is a backup plan. When the first-choice model can't answer, we send the request to a second choice, and maybe a third. In the following sections, we'll set up a fallback chain for one feature, figure out which errors should trigger it (and which really shouldn't), and look at why a backup that's never tested isn't much of a backup.

Onwards!

Quill Has a Bad Afternoon

Quill is a Rewrite this paragraph button in a notes app. You highlight a messy paragraph, click Quill, and get back a cleaner version. Quill calls one model at one provider, which we'll call Provider A. One afternoon, Provider A starts returning errors that say it's overloaded. Here is what Quill's users see:

[Image: Quill sends a rewrite request to Provider A, which is overloaded and answers with a 503 error. Quill shows Something went wrong, and a handwritten note says every single click.] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_one_provider.png)

Every click fails. The model still exists, and our code is fine too. There's just nowhere else for the request to go.

On a good afternoon, here is what Quill does with a messy paragraph:

[Image: Quill sits in a side panel in the corner of a notes app. The panel header shows a robot avatar with a feather badge, the name Quill, and the tagline Rewrite this paragraph. A user highlights a messy note: so the meeting moved to thurs i think?? 3pm. bring slides. Quill replies, The meeting has moved to Thursday at 3 pm. Please bring the slides. A handwritten note says This is Quill!] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_meet_quill.png)

Building a Fallback Chain

What we are going to do is give Quill a list of models to try, in order. If the first one fails in a way that suggests trying somewhere else, we move on to the next. Here is that chain on Quill's bad afternoon:

[Image: A fallback chain of four steps joined by if it fails arrows. 1, Provider A: overloaded. 2, Provider A in another region: timeout. 3, Provider B: answered. 4, a friendly busy message, which was not needed.] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_chain.png)

Each step in the chain is a little further from our first choice. The same model in a different region (if our provider offers that) behaves almost identically and only sidesteps a regional problem. A comparable model from another provider protects us when the whole first provider is down. The last step isn't a model at all. It's an honest, friendly Quill is busy, try again in a minute message, which is still a lot better than an error.

We should also always record which model actually answered. When users report that Quill got weird this afternoon, that record is how we figure out why.

Which Errors Should Trigger a Fallback?

This is where fallbacks go wrong most often. Not every error means try somewhere else. Some errors mean this request is broken, and sending a broken request to three providers just gets us three errors and a longer wait. Let's sort the common errors into two piles:

[Image: Two groups of errors. Try the next model: timeout, 429 rate limited, 500 server error, and 503 overloaded. Don't fall back: 400 bad request, 401 bad key, content refused, and context too long, unless the next model has a bigger context window.] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_which_errors.png)

The first pile is problems on the provider's side, or temporary capacity problems. Somewhere else might well succeed, so falling back makes sense.

The second pile is mostly problems with our request. A 400 Bad Request usually means we sent something malformed, and that bug comes with us to every provider. A 401 means a key is wrong, and that's a configuration bug we need to fix. Sending the request somewhere else would only hide it. A content refusal deserves extra care. If one model declined a request on safety grounds, shopping it around until some model says yes is a choice we'd have a hard time defending.

Context too long is a special case. If the next model in the chain can read more text in one request (it has a bigger context window (https://www.kirupa.com/ai_fundamentals/context_window_limits.htm)), falling back to it can genuinely help. If it can't, we're back to failing three times.

Retry First, or Fall Back First?

For a quick blip, like one random 500, retrying the same model once or twice (with a short, growing delay between tries) often works, and it keeps users on the model we tested the most. For a real outage, retries just make every user wait longer before reaching the fallback.

Our total time budget matters here too. If a user is waiting, each attempt eats into their patience:

[Image: A timeline over 12 seconds. Provider A times out after 4 seconds, the other region times out at 8 seconds, and Provider B answers at about 9 seconds, just before the user gives up at 10 seconds. A handwritten note says barely made it.] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_time_budget.png)

If each attempt waits up to 4 seconds before giving up, the third choice doesn't even start until 8 seconds in. So the timeout for each attempt has to be short enough that the whole chain still fits inside what a user will tolerate.

There's a neat trick for outages that last a while. If Provider A has failed for the last 20 requests in a row, there's no reason to make request 21 wait for its timeout too. A circuit breaker (https://www.kirupa.com/ai_fundamentals/timeouts_retries_circuit_breakers.htm) notices the run of failures and skips Provider A entirely for a little while, sending requests straight to the next choice. Every so often, it lets one request through to Provider A to check whether it has recovered.

Here is how a circuit breaker handles it:

[Image: Three steps. Provider A fails the last 20 requests in a row. The circuit breaker skips Provider A for a little while and sends requests straight to the next choice. Every so often, it lets one request through to Provider A to check whether it has recovered. Request 21 doesn't wait for a timeout that's sure to fail.] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_circuit_breaker.png)

The Fallback Model Is a Different Model

Fallbacks are trickier than they first look, because the fallback model is a different model. It might respond very differently to the exact same prompt.

Our Quill prompt was tuned on Provider A's model. On Provider B's model, the same prompt might produce longer rewrites, add a cheery Here's your rewritten paragraph! at the top, or ignore a formatting rule. If Quill uses tool calling (https://www.kirupa.com/ai_fundamentals/tool_calling.htm) or structured output (https://www.kirupa.com/ai_fundamentals/structured_outputs_json_schema.htm), the request format and the level of support can differ between providers too.

Here is what that can look like when Provider B answers:

[Image: Quill's panel in a notes app. A user highlights a messy note: so the meeting moved to thurs i think?? 3pm. bring slides. A small note says the request was answered by Provider B. Quill replies, Here's your rewritten paragraph! The meeting has moved to Thursday at 3 pm. Please bring the slides. A handwritten note flags that reply: Extra chatter!] (https://www.kirupa.com/ai_fundamentals/images/fallback_models_chat_provider_b.png)

So each model in the chain may need its own version of the prompt. More importantly, each one needs to be tested. If we keep a set of example paragraphs and quality checks (a golden dataset (https://www.kirupa.com/ai_fundamentals/golden_datasets.htm)), we should run them against every model in the chain, not just the first one.

And then we need to keep testing. Fallback code only runs during incidents, which makes it easy to forget about. The model gets retired, a key expires, or the prompt drifts, and we find out on the worst possible afternoon. Sending a small slice of everyday traffic through the fallback path, or exercising it on a schedule, keeps it honest.

We could write the chain into Quill's code, but it's even nicer in a model gateway (https://www.kirupa.com/ai_fundamentals/model_gateway.htm). Quill keeps asking for a name like rewrite-default, and the gateway owns the list of models to try, the error rules, the timeouts, and the circuit breakers. Every feature gets the same protection, and changing the chain is a configuration change.

Conclusion

Quill started with one model and no plan B, so when Provider A had a bad afternoon, so did every Quill user. A fallback chain gave Quill somewhere else to go: the same model in another region, a similar model at another provider, and finally a friendly message.

What makes a fallback chain work is the details. Fall back on timeouts, rate limits, and server errors, but not on broken requests or refusals. Keep each attempt's timeout short enough that the whole chain fits inside the user's patience, and use a circuit breaker so a long outage doesn't make everyone wait. Most of all, treat each fallback model as the different model it is, with its own prompt and its own tests, and run that path often enough that it still works on the day we need it.