Routing Easy Tasks to Cheaper Models

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

Imagine hiring a senior engineer and then asking them to reset passwords all day. They'd do a great job! They'd also be the most expensive password-resetter in the building, and a new hire could have handled most of those resets just as well.

That is what happens when every request in our app goes to our biggest model. The big model handles the hard questions beautifully. It also handles what are your shipping times? beautifully, at the same high price per token, when a much smaller model would have gotten that one right too.

Routing means looking at each request and sending it to the cheapest model that can handle it well. In the following sections, we'll build up the idea with one support inbox, look at the main ways to decide what counts as easy, and talk about what happens when the router guesses wrong.

Onwards!

Tilly's Inbox

Tilly runs an online shop called Tilly's Teapots. Her support assistant answers about 1,000 customer messages a day, and right now every single one goes to a large model.

Here is Tilly's support assistant, sitting in the corner of her shop's website:

[Image: Tilly's support assistant sits in a chat panel in the corner of the Tilly's Teapots website. The panel header shows a robot avatar with a teapot badge, the name Tilly's Teapots, and the tagline Support assistant. A customer asks, Where's my order? The assistant replies, Your teapot shipped yesterday. It should arrive on Thursday! A handwritten note says Tilly's assistant!] (https://www.kirupa.com/ai_fundamentals/images/routing_to_cheaper_models_meet_tilly.png)

When we actually read through a day of those messages, they're not all alike. A lot of them look like Where's my order?, How long do returns take?, or Do you ship to Canada? Some of them look like this instead:

I ordered two teapots. One arrived cracked and the other is the wrong color. I used a discount code on the whole order, so how much do I get back if I return just the cracked one?

The first group needs a lookup and a polite sentence. The second needs the model to untangle a messy situation, apply a refund policy, and do some math. Here is a day of Tilly's inbox, sorted by how hard each message actually is:

[Image: One hundred squares, each standing for ten support messages. Seventy squares, about 700 messages, are easy questions about order status, shipping, and returns. Thirty squares, about 300 messages, are hard, messy, multi-step questions.] (https://www.kirupa.com/ai_fundamentals/images/routing_to_cheaper_models_inbox.png)

The 700 and 300 split is made up for this example. Every product's mix is different, and measuring ours is the first real step (more on that later). The shape is what matters. In a lot of apps, a big share of the traffic is routine.

What the Price Gap Looks Like

Model prices change constantly, so let's use clearly made-up numbers for the math. Say the large model costs about 1 cent to answer a typical message, and a small model costs about 0.1 cents for the same message. A real gap between model sizes can be this large or larger, so check your provider's current pricing page and plug in real numbers.

If all 1,000 messages go to the large model, Tilly pays about $10 a day. If the 700 easy ones go to the small model instead, the math changes:

[Image: Two receipts. Everything to large: 1,000 messages at 1 cent is 10 dollars a day. Routed: 700 messages at 0.1 cents is 70 cents, plus 300 messages at 1 cent is 3 dollars, for a total of 3 dollars and 70 cents a day.] (https://www.kirupa.com/ai_fundamentals/images/routing_to_cheaper_models_receipts.png)

That is a big difference for a change that the customers asking easy questions shouldn't even notice. The whole trick, of course, is figuring out which messages are easy before we answer them.

Deciding What Counts as Easy

Let's look at the common ways to do this, starting with the simplest. Each one sorts messages a bit better than the one before it, and costs a bit more to run.

Rules

The simplest router is plain code. If the message came from the Track my order button, it's easy. If it's under 15 words and mentions shipping, it's easy. Everything else goes to the large model.

Rules are fast, free to run, and easy to understand when something goes wrong. They're also brittle. Where's my order? Also the lid doesn't fit and I want to swap it for the blue one is short and mentions an order, and it is not an easy message.

A Classifier in Front

The next step up is to ask a model to sort the message before anyone answers it. A small, fast model reads the message and returns a label like easy or hard, and our code sends the message to the matching model:

[Image: A customer message goes to a router, a small model that labels it. Messages labeled easy go on to the small model, and messages labeled hard go to the large model.] (https://www.kirupa.com/ai_fundamentals/images/routing_to_cheaper_models_classifier.png)

The router costs a little on every request, but since it is a small model producing a single word, it is usually a small fraction of what we save. Researchers have also trained dedicated routers for this job. One example is RouteLLM (https://github.com/lm-sys/RouteLLM), which learns from data about which questions a strong model answered better than a weaker one.

Try Small First, Then Escalate

The third approach flips things around. Instead of predicting how hard a message is, we just try the small model first, check its answer, and only call the large model if the check fails. This pattern is usually called a cascade, and the FrugalGPT paper (https://arxiv.org/abs/2305.05176) explored it in detail:

[Image: A message goes to the small model, and a check looks at its answer. If the answer is good enough, it is sent. If not, the large model answers too, so hard messages pay for both calls.] (https://www.kirupa.com/ai_fundamentals/images/routing_to_cheaper_models_cascade.png)

The interesting part is the check. It can be simple code, like did the answer mention the order number we looked up? or did the JSON parse? It can be the small model reporting how confident it is (which is useful, but models are not always good judges of their own answers). It can also be another small model acting as a grader.

What makes cascades tricky is that hard messages now pay twice: once for the small model's attempt and again for the large model. They also take longer, since there are two calls in a row. Cascades work best when most traffic is easy and the check is cheap and trustworthy.

When the Router Gets It Wrong

No router is perfect, so it is worth thinking about the two ways it can be wrong. They are not equally bad:

[Image: A grid of routing outcomes. An easy message sent to the small model saves money, and a hard message sent to the large model is money well spent. An easy message sent to the large model wastes a little money. A hard message sent to the small model gives a real customer a bad answer, and a handwritten note marks it as the one that hurts.] (https://www.kirupa.com/ai_fundamentals/images/routing_to_cheaper_models_mistakes.png)

If an easy message goes to the large model, we waste a fraction of a cent. If a hard message goes to the small model, the customer with the cracked teapot gets a confident, wrong refund amount. That costs a lot more than a fraction of a cent.

So when the router isn't sure, it should lean toward the large model. Most classifier routers give us a score we can tune, and setting that threshold is really a business decision about how many bad answers we'll accept in exchange for how much savings. It also helps to give the customer a way out. If someone replies that didn't answer my question, we can send the rest of that conversation to the large model.

Measure Before and After

None of this works without measuring. Before turning routing on, we can take a few hundred real messages, run them through both models, and have people (or a carefully checked grading model) score the answers. That tells us whether the small model is really good enough on the easy bucket, and it gives us a test set (a golden dataset (https://www.kirupa.com/ai_fundamentals/golden_datasets.htm)) to rerun whenever we change the router or a model changes underneath us.

After launch, we keep checking quality on live traffic too (Offline vs. Online Evals (https://www.kirupa.com/ai_fundamentals/offline_vs_online_evals.htm) covers both kinds of checking), and there are two numbers worth watching. The first is the share of traffic going to each model. If it suddenly shifts, something about the messages or the router changed. The second is a quality signal on each path, like thumbs-down rates or how often conversations get handed to a human.

A router can live inside our app, but a nicer home is a model gateway (https://www.kirupa.com/ai_fundamentals/model_gateway.htm). Our features keep asking for one model name like support-answer, and the gateway decides whether that particular request goes to the small model or the large one. When we want to tune the threshold or try a new small model, nobody's app code has to change.

Conclusion

We started with Tilly paying large-model prices for do you ship to Canada? By looking at each message first, we sent the easy ones to a model that costs a fraction as much and saved the large model for the questions that need it. Rules, a classifier in front, and a small-first cascade are three ways to make that call, and each one trades a bit more cost and complexity for better sorting.

The thing to keep in mind is that the router's two kinds of mistakes aren't equally bad. An easy question sent to the large model wastes a fraction of a cent. A hard question sent to the small model can cost Tilly a customer. So when the router isn't sure, it should send the message to the bigger model, and we should keep measuring to make sure the savings are real.