When a model doesn't behave the way we want, there are two very different ways to fix it. We can change what we say to the model, which is prompting. Or we can change the model itself by training it further on our own examples, which is fine-tuning.
Fine-tuning sounds like the serious, grown-up option, so a lot of teams reach for it early. Most of the time, that's a mistake. But not always! In the following sections, we'll look at what each approach actually changes, and then walk through the specific situations where one clearly beats the other.
Onwards!
The New Barista
An analogy helps here. Imagine a coffee shop hires an experienced barista. They already know how to make coffee. What they don't know is how this shop does things.
We could hand them a card at the start of every shift: Oat milk is the default. Call out the customer's name. Never more than two pumps of syrup. That's prompting. It works on day one, and if the rules change, we just print a new card.
Or we could spend a few weeks training them, with hundreds of practice orders and corrections, until the shop's way of doing things is second nature and they don't need the card anymore. That's fine-tuning. It takes effort up front, and if the rules change, we have to train them again. Here are the two baristas side by side:
What Actually Changes
In model terms, a prompt changes the input. The model's learned parameters (the numbers that hold what it knows) stay exactly the same. Everything we want lives in the text we send each time, including instructions, examples, and any documents.
Fine-tuning changes the parameters. We collect a set of examples, each one an input paired with the exact output we wish the model had written, and we run more training on those examples. The model's parameters shift a little so that outputs like ours become more likely. Afterward, the model behaves that way even when the prompt doesn't ask for it. This is what each approach touches:
A lot of modern fine-tuning doesn't even touch most of the original parameters. A popular technique called LoRA, from a 2021 paper by Edward Hu and colleagues, freezes the original model and trains a small set of extra parameters that sit alongside it. That makes training cheaper and storing the result much cheaper. The idea is the same, though. The behavior gets baked in.
Meet Tidewater Travel
To see which approach wins when, let's follow a made-up travel agency called Tidewater Travel. It wants a model to help its agents with three jobs:
- Writing customer emails in the house voice. Tidewater's emails are warm, short, and always end with a specific next step.
- Sorting incoming requests into 30 internal codes, like REBOOK_WEATHER, REFUND_FARE_CLASS_B, and VISA_QUESTION.
- Answering agent questions about current airline rules, like this month's carry-on limit for each partner airline.
Start With Prompting
For all three jobs, prompting is where we start. It's fast to try, we can change it in seconds, and it keeps working when we switch to whatever new model comes out next month, with no retraining.
For the house voice, we write a short style guide in the system prompt and include three or four real emails the team loves. For the request codes, we list all 30 codes with a one-line description each, plus an example or two for the confusing ones. Then we test both against a set of real past emails and requests where we already know the right answer.
Very often, this is where the story ends. Clear instructions and a few good examples get a capable model most of the way there. OpenAI's own guide to optimizing accuracy recommends exactly this order: get a solid prompt and a set of test cases first, and use that as the baseline that any fine-tuned model has to beat.
When Fine-Tuning Wins
Prompting runs out of road in a few specific situations.
The behavior is easier to show than to describe. Tidewater's house voice is something every agent recognizes instantly, but nobody can fully write down. If the style guide and four examples still produce emails that feel a bit off, a few hundred real emails used as training examples can teach the pattern more directly than any description could.
The prompt is getting enormous. Suppose the 30 request codes need 60 examples before the model sorts requests reliably. That's a long prompt, and we pay for every token of it on every request, thousands of times a day (Tokens and How They Set Your Bill shows why that adds up). A fine-tuned model has absorbed those examples, so each request can carry a short instruction instead. At high volume, that saving can pay back the cost of training.
We want a small model to do a big model's job. If a large model sorts requests well but is too slow or too expensive, we can have it label thousands of real requests and then fine-tune a small model on those labels. This is often called distillation, and it's a common way to get large-model quality on one narrow task at small-model prices (see Small vs. Large Model Trade-offs).
Here are those three situations together:
When Fine-Tuning Loses
Now for the third job, answering questions about current airline rules. It's tempting to think we could fine-tune the model on every airline's rulebook so that it knows them. This is a classic mistake.
Fine-tuning is good at teaching how to respond. It's unreliable at teaching new facts, and it's a poor fit for facts that change. The carry-on limit that was right when we trained might be wrong next month, and the fine-tuned model will keep stating the old one with total confidence. Researchers have measured part of this. In a 2024 study, Zorik Gekhman and colleagues found that models picked up new facts from fine-tuning much more slowly than examples that matched what they already knew, and that as the new facts were finally learned, the models became more likely to make things up. Here is how that plays out for Tidewater's carry-on question:
For this job, the winner is prompting plus a lookup. Our code finds the relevant rule in an up-to-date source and puts it in the prompt for each question, which is the idea behind retrieval-augmented generation (aka RAG). When an airline changes its rule, we update the document, and the very next answer is right.
The Hidden Costs
Even when fine-tuning would help, it comes with costs that prompting doesn't have. We need a clean set of examples, and the quality of those examples sets a ceiling. OpenAI's fine-tuning guide points out that if the people writing the examples only agree with each other 70% of the time, the model probably won't do better than that. We also need training runs, a way to test the result, and a plan for retraining when the base model is retired.
Fine-tuning also ties us to one specific base model. When a better model comes out, a prompt can usually move over in an afternoon. A fine-tune has to be redone.
Availability is worth checking too. At the time of writing, that same OpenAI guide says the company is winding down its fine-tuning platform and no longer accepts new users, while other providers and open-weight models still support fine-tuning. This changes often, so check what's available before planning around a specific option.
Deciding
Putting it all together, here is the order of questions we'd ask for any job:
For Tidewater, the airline rules go to retrieval and stay there. The request codes probably start as a prompt, and they might become a fine-tuned small model if the long prompt gets expensive at volume. The house voice starts as a prompt and only gets fine-tuned if the team keeps saying close, but it doesn't sound like us.
Conclusion
Prompting changes what we tell the model each time. Fine-tuning changes the model so it behaves a certain way without being told. Prompting wins on speed, flexibility, and anything involving facts that change, especially once we add a lookup. Fine-tuning wins when a behavior is easier to show than to describe, when a giant prompt is costing real money at volume, or when we want a small model to match a big one on a narrow task.
In every case, the deciding vote goes to a test set. Without one, we can't tell whether a fine-tune helped at all, and that's a lot of effort to spend on a guess. If you don't have a test set yet, Golden Datasets is a good place to start.
That wraps up the individual pieces. In How an Agent Like Claude Code Works, we'll see prompts, tools, context windows, and model choices all working together in a loop.
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
