Graceful Degradation

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

The comedian Mitch Hedberg had a bit about escalators:

An escalator can never break. It can only become stairs. There would never be an 'Escalator Temporarily Out of Order' sign, only 'Escalator Temporarily Stairs. Sorry for the convenience.'

Mitch Hedberg, as quoted by 99% Invisible (https://99percentinvisible.org/episode/episode-43-the-accidental-music-of-imperfect/transcript/)

Here is that idea in picture form:

[Image: A stopped escalator has become an ordinary staircase, and a person is walking up it. A sign at the bottom reads Escalator Temporarily Stairs. Sorry for the convenience.] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_escalator.png)

That joke is a surprisingly good description of how we want AI features to fail. When the model behind a feature is slow, down, rate limited, or over budget, the feature shouldn't vanish and take the page with it. It should become a slightly less fancy version of itself, the way the escalator becomes stairs. You still get to the top. You just do a bit more of the work.

That idea is called graceful degradation, and in this article we'll look at how to plan for it and, just as importantly, how to keep the less fancy versions honest.

Onwards!

Meet the Tent Page

Imagine we run TrailHead Outfitters, an online camping store. Our product page for the Ridgeline 3 tent has the usual things on it (photos, price, an Add to Cart button), plus two AI features. One summarizes the tent's 1,200 customer reviews into a few sentences. The other is an Ask about this tent assistant where you can type questions like Will three adults fit in this?:

[Image: The Ridgeline 3 tent page has a photo, a price, and an Add to Cart button, plus two AI panels on the right: a review summary and an Ask about this tent assistant.] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_product_page.png)

The thing to notice is that the AI panels are extras. People come to this page to decide whether to buy a tent. If the review summary is missing for an hour, that's a bit of a bummer. If the Add to Cart button is missing for an hour, that's a disaster.

The Failure Without a Plan

Let's say the first version of our page builds the review summary on the server before sending anything to the browser. Then one evening the model API slows to a crawl. Our server waits for the summary, the page waits for the server, and shoppers are left looking at a blank screen:

[Image: On the left, the whole product page stays blank while it waits for the AI summary. On the right, the product details and Add to Cart button render right away, and only the AI panels are still loading.] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_critical_path.png)

The first rule of graceful degradation falls right out of this picture. The AI feature should never sit on the critical path of the page's main job. We render the page first and load the AI panels separately, each with its own timeout (the Timeouts, Retries, and Circuit Breakers (https://www.kirupa.com/ai_fundamentals/timeouts_retries_circuit_breakers.htm) article covers how to pick those). If a panel is slow or broken, only that panel is affected.

That gets the page back. But it leaves an empty hole where the summary used to be, and we can do better than a hole.

Building a Degradation Ladder

What we are going to do is decide, ahead of time, what each AI feature turns into when things go wrong. The easiest way to picture this is as a ladder. The top rung is the full experience. Each rung below it is less impressive, but cheaper and more likely to work. Here is a ladder for our review summary:

[Image: Five rungs for the review summary, stepping down from a fresh summary by the main model, to a smaller model, to last night's cached summary, to a star chart with the top reviews, and finally to hiding the panel. Each rung down is less helpful but more likely to work.] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_ladder.png)

Each rung earns its spot for a reason. A smaller fallback model (https://www.kirupa.com/ai_fundamentals/fallback_models.htm) is often available when the big one isn't, and it's quite good at summarizing. Last night's summary is already sitting in a cache, so it costs nothing and almost never fails, as long as we tell people how old it is. The star chart and the three most helpful reviews don't involve a model at all. They come straight from our database. And hiding the panel is the bottom rung that is always available.

The assistant gets its own ladder, because a different feature degrades differently:

1. The full assistant, using the main model with access to the product specs and reviews.
2. A smaller model that only answers from the spec sheet.
3. A plain keyword search over the product FAQ, showing the closest matching question and answer.
4. A friendly note that the assistant is taking a break, with links to the spec sheet and to customer support.

Notice that the bottom rung of both ladders is something we can serve with no model at all. That's on purpose, because a bottom rung that still needs the model would fail at exactly the moment we need it most.

Deciding When to Step Down

A ladder is only useful if something decides which rung we're standing on. That decision is driven by signals we already have:

[Image: Four signals feed one decision about which rung of the ladder to serve: a request timing out, the circuit breaker opening, the daily budget running out, and an engineer flipping a kill switch.] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_signals.png)

Some of these signals are about one request. A timeout says that this particular summary isn't coming, so serve this shopper the next rung down. Others are about the whole system. An open circuit breaker means everyone should skip the main model for a while. A daily token budget running out (the cap we set on how much a feature can spend in a day, which we'll keep an eye on in Token and Cost Metrics (https://www.kirupa.com/ai_fundamentals/token_and_cost_metrics.htm)) might mean we serve cached summaries for the rest of the day. The kill switch is for humans. If the assistant starts saying something embarrassing at 2am, the on-call engineer should be able to drop it to the bottom rung with one flag flip, without a deploy.

Keeping the Lower Rungs Honest

This is the part that makes AI degradation trickier than regular degradation. A cached image is either there or it isn't. A smaller model, on the other hand, can produce something that looks just as confident as the main model's answer and is wrong.

Imagine a shopper asks, Is the Ridgeline 3 waterproof? Our main model reads the spec sheet and says it has a 1,500 mm rated rainfly and taped seams. Our smaller fallback model skims the same sheet and says Yes, it's fully waterproof! That sounds helpful, and it's the kind of answer that ruins someone's camping trip. A wrong answer is a worse outcome than a missing one, so every rung needs to be more reliable than the one above it, even if it's less capable. A few habits help with that:

- Narrow the job as you go down. The fallback model answers only from the spec sheet and is told to say I'm not sure otherwise.
- Route the high-stakes questions around the fallback. For questions about returns, safety, or warranty, a link to the actual policy page beats a guess.
- Label what people are seeing. Summary from Nov 10 or Quick answers mode costs a few words and keeps trust intact.
- Never fill a gap with invention. This one matters most for agents, as we'll see next.

Here are the two answers to that shopper's question:

[Image: The Ask about this tent chat on the Ridgeline 3 product page. A shopper asks, Is the Ridgeline 3 waterproof? A small note says the main model is answering. The assistant replies, It has a 1,500 mm rated rainfly and taped seams. Another note says the smaller fallback model is answering. The assistant replies, Yes, it's fully waterproof! A handwritten note flags that reply: Confident guess!] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_chat_waterproof.png)

Agents that call several tools can lose one tool while the others work fine. Say our assistant gets the question Can I get this by Friday, and will it fit three sleeping pads? It calls a stock tool, a size tool, and a shipping tool, and the shipping tool times out:

[Image: The assistant calls three tools. check_stock and check_fit succeed, but shipping_eta times out. The answer uses the two results and says plainly that it could not get a delivery estimate instead of guessing one.] (https://www.kirupa.com/ai_fundamentals/images/graceful_degradation_partial_tools.png)

The degraded answer still helps. It's honest about the missing piece, and it doesn't make up a delivery date to paper over the gap. A model left to its own devices will happily guess, so the instruction to name missing information belongs in the prompt, and the tool's failure should be passed to the model as an explicit result like {"error": "timed out"} rather than silently dropped.

Practice Falling Down

There is one last uncomfortable truth here. A fallback path that never runs is probably broken. The job that builds cached summaries stopped running three weeks ago, or the smaller model's API key was never added to production, and nobody noticed because the main model was healthy the whole time.

The fix is to exercise the ladder on purpose. In a staging environment, force the circuit breaker open, flip the kill switch, and expire the budget, then walk through the page and look at every rung. Some teams also send a small slice of real traffic down a lower rung all the time, just to keep it warm and to keep an eye on its quality.

Conclusion

Graceful degradation is the escalator becoming stairs. In our case, we started by moving the AI panels off the critical path, so a slow model can't take the Add to Cart button down with it. Then we gave each AI feature a ladder of rungs, from the full experience down to something that needs no model at all, and let timeouts, breakers, budgets, and a human kill switch decide which rung to serve.

What makes this harder with AI than with plain old web features is that a weaker rung can fail by being confidently wrong instead of being missing. That's why the lower rungs get narrower jobs, honest labels, and permission to say I don't know. And to know whether any of this is working in production, we need to see what our system is doing request by request, which is where Tracing and Prompt Logs (https://www.kirupa.com/ai_fundamentals/tracing_and_prompt_logs.htm) come in.