# Small vs. Large Model Trade-offs

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

If you open the model list for almost any AI provider, you'll find the same family of models offered in a few sizes. They have names like mini, nano, Flash, Haiku, or Pro, and they're priced very differently. The big ones are smarter. The small ones are faster and cheaper. Picking one sounds simple until we realize we're making that choice for every task in our app, and that the right answer is often different for each one.

In the following sections, we'll look at what ***size*** actually means for a model, what we gain and give up at each end, and a practical way to pick the smallest model that does a job well.

Onwards!

## What "Size" Means

A model's size usually refers to its number of **parameters**. Parameters are the numbers a model learned during training, and together they hold everything it knows and every pattern it can recognize. Small models have fewer of them. Large models have many more.

More parameters give a model more room to store knowledge and pick up on subtle patterns, which is why large models tend to do better at hard reasoning, tricky instructions, and topics that need a lot of background knowledge. The cost is that producing every single token means doing math across those parameters. More parameters means more math, which means more expensive hardware, slower replies, and a higher price per token. (Some newer models only use part of their parameters for each token, a design called mixture of experts, which softens this. The general trade still holds.)

Here is a quick way to picture the two ends of the lineup:

![Two cards. A small model, drawn as a bicycle courier, gives fast replies, is cheap per token, and runs almost anywhere. A large model, drawn as a delivery truck, handles harder tasks but gives slower replies and is pricier per token.](https://www.kirupa.com/ai_fundamentals/images/small_vs_large_models_two_ends.png)

A bike courier is the right call for a letter across town. A truck is the right call for a couch. Nobody sends the truck for the letter, but with AI models, it's very easy to do exactly that without noticing.

## Three Jobs at Lumen Lamps

To make this concrete, imagine an online lamp store called Lumen Lamps. Its support team gets a steady stream of tickets, and it wants AI help with three jobs:

1. **Tagging.** Read each new ticket and label it `shipping`, `damaged`, `billing`, `how_to`, or `other`.
2. **Summarizing.** Turn a long back-and-forth email thread into a five-line summary so the next agent can catch up quickly.
3. **Drafting dispute replies.** Write a first draft for messy cases, like a customer who was charged twice, already got a partial refund, and is quoting a return policy that changed last month.

These jobs are very different. Tagging is a short read and one word out. Summarizing needs good reading comprehension over a longer input. The dispute reply needs the model to untangle a messy situation, apply the right policy, get the refund math right, and strike the right tone with an annoyed customer.

So which model should each job get? We could guess, and we'd probably guess ***small for tagging, large for disputes***. We might even be right. But guessing is how we end up overpaying for months, or shipping a model that mis-tags a fifth of our tickets without anyone noticing. A better approach is to measure.

## Measure With Your Own Examples

The most useful thing we can build before choosing a model is a small test set (often called an evaluation set, or eval set). For tagging, that's a collection of real past tickets, each paired with the tag a person agreed was correct. Fifty to a hundred examples is a fine start, as long as they include the awkward ones: the ticket that's about a damaged lamp **and** a billing problem, the one written in Spanish, and the one that just says ***???***. ([Golden Datasets](https://www.kirupa.com/ai_fundamentals/golden_datasets.htm) goes much deeper on building a set like this.)

Then we run every candidate model over the same test set and count how many tags it gets right. Here is an illustrative picture of how that might come out for tagging:

![A bar chart of correct tags on a test set for a small, medium, and large model. A dashed green line marks good enough for this job. All three bars clear it, and the large bar is only slightly taller than the medium one. A handwritten note points at the small bar: smallest one that clears the line.](https://www.kirupa.com/ai_fundamentals/images/small_vs_large_models_eval.png)

The interesting decision is where to draw that ***good enough*** line, and it's mostly a product decision, because it depends on what a mistake costs. At Lumen Lamps, a mis-tagged ticket gets bounced to the right queue a few minutes later, so a small error rate is fine. A mis-tagged message in a hospital's triage inbox would be a very different story.

Now imagine running the same exercise on the dispute replies. Here we'd grade each draft against a short checklist. Did it apply the right policy? Is the refund math right? Does it sound like a person who wants to help? It's very possible that only the large model clears the line for this job. That's fine! It means the dispute replies get the large model, and the tagging doesn't.

## The Math at Volume

Why bother with all this? Because the price gap adds up fast when a job runs all day long.

Let's use clearly made-up numbers. Say Lumen Lamps tags 20,000 tickets a day, and each tagging request uses about 400 tokens. That's 8 million tokens a day for tagging alone. If the large model costs, say, ten times as much per token as the small one, then running tagging on the large model costs ten times as much every single day, for tags that aren't meaningfully better. (If tokens and pricing are new to you, [Tokens and How They Set Your Bill](https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm) walks through how a request turns into a bill.)

Meanwhile, there might only be a few dozen dispute replies a day. Paying the large-model price on those is a rounding error, and better drafts save the support team real time. So instead of picking one model for all of Lumen Lamps, we pick one for each job:

![New work goes to a router that asks which job it is. Tagging, which happens thousands of times a day, goes to a small model. Summaries, hundreds a day, go to a medium model. Dispute replies, a few dozen a day, go to a large model.](https://www.kirupa.com/ai_fundamentals/images/small_vs_large_models_routing.png)

Sending each request to the model that fits it is called **routing**, and it's one of the cheapest wins in an AI system. In its simplest form, the ***router*** is just an `if` statement, because our code already knows which job it's doing. [Routing Easy Tasks to Cheaper Models](https://www.kirupa.com/ai_fundamentals/routing_to_cheaper_models.htm) covers the trickier case, where the same kind of request can be easy or hard.

## Speed Matters Too

Cost isn't the only reason to go small. Small models usually respond faster, both in how quickly the first token shows up and in how quickly the rest of the reply streams in. If tagging happens while a customer waits on a ***submit ticket*** page, shaving a second off every request is something people notice.

Small models also open up options that large ones can't. Some are small enough to run on our own servers, or even on a laptop or a phone. That matters when the data isn't allowed to leave the building, or when a feature needs to work offline.

## Where Small Models Stumble

Small models tend to struggle in predictable places. They're more likely to drift away from long, detailed instructions, to lose track of details in a long input, and to trip over multi-step tasks that use several [tools](https://www.kirupa.com/ai_fundamentals/tool_calling.htm) in a row. They also know less about the world, so they're more likely to confidently make up an answer to a niche question.

We can often help a small model across the line. A tighter prompt with a few worked examples, a [JSON schema](https://www.kirupa.com/ai_fundamentals/structured_outputs_json_schema.htm) that locks down the output format, or splitting one big task into two simpler steps can each make a real difference. And if a small model still falls just short, training it further on examples written by a large model is another option, which [Prompting vs. Fine-Tuning](https://www.kirupa.com/ai_fundamentals/prompting_vs_fine_tuning.htm) covers.

## Conclusion

Bigger models have more parameters, which usually means better results on hard tasks, at the cost of more money and more waiting for every token. Smaller models are faster and cheaper, and they can run in places the big ones can't, but they give out sooner on complex instructions and niche knowledge.

The way through is to stop picking one model for the whole app. Build a small test set for each job, decide what ***good enough*** means for that job, and pick the smallest model that clears the line. At Lumen Lamps, that probably means a small model tags thousands of tickets a day while a large one handles the few dozen disputes that actually need it. New models come out all the time, so it's worth rerunning those tests every so often. A job that needs a large model today may run fine on a small one next year.