# Prompting vs. Fine-Tuning: When Each Wins

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

When a model doesn't behave the way we want, there are two very different ways to fix it. We can change what we **say** to the model, which is prompting. Or we can change the **model itself** by training it further on our own examples, which is fine-tuning.

Fine-tuning sounds like the serious, grown-up option, so a lot of teams reach for it early. Most of the time, that's a mistake. But not always! In the following sections, we'll look at what each approach actually changes, and then walk through the specific situations where one clearly beats the other.

Onwards!

## The New Barista

An analogy helps here. Imagine a coffee shop hires an experienced barista. They already know how to make coffee. What they don't know is how **this** shop does things.

We could hand them a card at the start of every shift: ***Oat milk is the default. Call out the customer's name. Never more than two pumps of syrup***. That's prompting. It works on day one, and if the rules change, we just print a new card.

Or we could spend a few weeks training them, with hundreds of practice orders and corrections, until the shop's way of doing things is second nature and they don't need the card anymore. That's fine-tuning. It takes effort up front, and if the rules change, we have to train them again. Here are the two baristas side by side:

![Two panels with the same barista. In the prompting panel, the barista works from an instruction card read at the start of every shift. In the fine-tuning panel, the barista has practiced on hundreds of orders until the shop's way of doing things is habit, and no card is needed.](https://www.kirupa.com/ai_fundamentals/images/prompting_vs_fine_tuning_barista.png)

## What Actually Changes

In model terms, a prompt changes the **input**. The model's learned parameters (the numbers that hold what it knows) stay exactly the same. Everything we want lives in the text we send each time, including instructions, examples, and any documents.

Fine-tuning changes the **parameters**. We collect a set of examples, each one an input paired with the exact output we wish the model had written, and we run more training on those examples. The model's parameters shift a little so that outputs like ours become more likely. Afterward, the model behaves that way even when the prompt doesn't ask for it. This is what each approach touches:

![Two rows. In the prompting row, a card of instructions is added to what we send the model, and the model's dials stay untouched. In the fine-tuning row, a stack of examples goes into training, and two of the model's dials are turned and highlighted.](https://www.kirupa.com/ai_fundamentals/images/prompting_vs_fine_tuning_dials.png)

A lot of modern fine-tuning doesn't even touch most of the original parameters. A popular technique called LoRA, from a [2021 paper](https://arxiv.org/abs/2106.09685) by Edward Hu and colleagues, freezes the original model and trains a small set of extra parameters that sit alongside it. That makes training cheaper and storing the result much cheaper. The idea is the same, though. The behavior gets baked in.

## Meet Tidewater Travel

To see which approach wins when, let's follow a made-up travel agency called Tidewater Travel. It wants a model to help its agents with three jobs:

1. **Writing customer emails in the house voice.** Tidewater's emails are warm, short, and always end with a specific next step.
2. **Sorting incoming requests into 30 internal codes**, like `REBOOK_WEATHER`, `REFUND_FARE_CLASS_B`, and `VISA_QUESTION`.
3. **Answering agent questions about current airline rules**, like this month's carry-on limit for each partner airline.

## Start With Prompting

For all three jobs, prompting is where we start. It's fast to try, we can change it in seconds, and it keeps working when we switch to whatever new model comes out next month, with no retraining.

For the house voice, we write a short style guide in the [system prompt](https://www.kirupa.com/ai_fundamentals/system_prompts_vs_user_prompts.htm) and include three or four real emails the team loves. For the request codes, we list all 30 codes with a one-line description each, plus an example or two for the confusing ones. Then we test both against a set of real past emails and requests where we already know the right answer.

Very often, this is where the story ends. Clear instructions and a few good examples get a capable model most of the way there. OpenAI's own [guide to optimizing accuracy](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) recommends exactly this order: get a solid prompt and a set of test cases first, and use that as the baseline that any fine-tuned model has to beat.

## When Fine-Tuning Wins

Prompting runs out of road in a few specific situations.

**The behavior is easier to show than to describe.** Tidewater's house voice is something every agent recognizes instantly, but nobody can fully write down. If the style guide and four examples still produce emails that feel a bit off, a few hundred real emails used as training examples can teach the pattern more directly than any description could.

**The prompt is getting enormous.** Suppose the 30 request codes need 60 examples before the model sorts requests reliably. That's a long prompt, and we pay for every token of it on every request, thousands of times a day ([Tokens and How They Set Your Bill](https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm) shows why that adds up). A fine-tuned model has absorbed those examples, so each request can carry a short instruction instead. At high volume, that saving can pay back the cost of training.

**We want a small model to do a big model's job.** If a large model sorts requests well but is too slow or too expensive, we can have it label thousands of real requests and then fine-tune a small model on those labels. This is often called **distillation**, and it's a common way to get large-model quality on one narrow task at small-model prices (see [Small vs. Large Model Trade-offs](https://www.kirupa.com/ai_fundamentals/small_vs_large_models.htm)).

Here are those three situations together:

![Three situations where fine-tuning beats prompting at Tidewater. The behavior is easier to show than to describe, so a few hundred real emails teach the house voice. The prompt is getting enormous, so 60 examples become a short instruction. We want a small model to do a big model's job, so a large model labels thousands of requests and a small model is fine-tuned on those labels, which is called distillation.](https://www.kirupa.com/ai_fundamentals/images/prompting_vs_fine_tuning_when_it_wins.png)

## When Fine-Tuning Loses

Now for the third job, answering questions about current airline rules. It's tempting to think we could fine-tune the model on every airline's rulebook so that it ***knows*** them. This is a classic mistake.

Fine-tuning is good at teaching **how** to respond. It's unreliable at teaching new **facts**, and it's a poor fit for facts that change. The carry-on limit that was right when we trained might be wrong next month, and the fine-tuned model will keep stating the old one with total confidence. Researchers have measured part of this. In a [2024 study](https://arxiv.org/abs/2405.05904), Zorik Gekhman and colleagues found that models picked up new facts from fine-tuning much more slowly than examples that matched what they already knew, and that as the new facts were finally learned, the models became more likely to make things up. Here is how that plays out for Tidewater's carry-on question:

![Two panels. A model fine-tuned last month, with the airline rules baked in during training, confidently says the carry-on limit is 10 kilograms, with a warning sign. A model that looks up today's airline rules and gets them in its prompt says 8 kilograms, with a check mark.](https://www.kirupa.com/ai_fundamentals/images/prompting_vs_fine_tuning_stale.png)

For this job, the winner is prompting plus a lookup. Our code finds the relevant rule in an up-to-date source and puts it in the prompt for each question, which is the idea behind retrieval-augmented generation (aka RAG). When an airline changes its rule, we update the document, and the very next answer is right.

## The Hidden Costs

Even when fine-tuning would help, it comes with costs that prompting doesn't have. We need a clean set of examples, and the quality of those examples sets a ceiling. OpenAI's [fine-tuning guide](https://developers.openai.com/api/docs/guides/fine-tuning-best-practices) points out that if the people writing the examples only agree with each other 70% of the time, the model probably won't do better than that. We also need training runs, a way to test the result, and a plan for retraining when the base model is retired.

Fine-tuning also ties us to one specific base model. When a better model comes out, a prompt can usually move over in an afternoon. A fine-tune has to be redone.

Availability is worth checking too. At the time of writing, that same OpenAI guide says the company is winding down its fine-tuning platform and no longer accepts new users, while other providers and open-weight models still support fine-tuning. This changes often, so check what's available before planning around a specific option.

## Deciding

Putting it all together, here is the order of questions we'd ask for any job:

![A decision flow. Do you have a test set? If no, build one first. Does a good prompt pass it? If yes, ship the prompt. Is the gap about facts that change? If yes, add retrieval. Is it about style, format, a huge prompt, or cost at volume? If yes, try fine-tuning and compare it against the prompt.](https://www.kirupa.com/ai_fundamentals/images/prompting_vs_fine_tuning_flow.png)

For Tidewater, the airline rules go to retrieval and stay there. The request codes probably start as a prompt, and they might become a fine-tuned small model if the long prompt gets expensive at volume. The house voice starts as a prompt and only gets fine-tuned if the team keeps saying ***close, but it doesn't sound like us***.

## Conclusion

Prompting changes what we tell the model each time. Fine-tuning changes the model so it behaves a certain way without being told. Prompting wins on speed, flexibility, and anything involving facts that change, especially once we add a lookup. Fine-tuning wins when a behavior is easier to show than to describe, when a giant prompt is costing real money at volume, or when we want a small model to match a big one on a narrow task.

In every case, the deciding vote goes to a test set. Without one, we can't tell whether a fine-tune helped at all, and that's a lot of effort to spend on a guess. If you don't have a test set yet, [Golden Datasets](https://www.kirupa.com/ai_fundamentals/golden_datasets.htm) is a good place to start.

That wraps up the individual pieces. In [How an Agent Like Claude Code Works](https://www.kirupa.com/ai_fundamentals/how_an_agent_works.htm), we'll see prompts, tools, context windows, and model choices all working together in a loop.