# Prompt Caching

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

Imagine a friend who, every time you ask them a question about a book, starts reading the book again from page one. They're happy to help. They just insist on re-reading 300 pages before answering whether the butler did it, and then again before telling you the butler's name.

That's roughly what a language model does with our prompts by default. Every request is processed from the first token to the last, even when 95% of those tokens are identical to the request we sent ten seconds ago. Prompt caching is how we let the model keep its place in the book.

In this article, we'll look at what the model actually saves, why the saved work only lines up when the start of the prompt matches exactly, and how to lay out our prompts so the cache gets used.

Onwards!

## The Handbook Assistant

Let's say our company has a 60-page employee handbook, and we've built an assistant that answers questions about it. A new hire named Marcus asks it things all day.

Each request our app sends has the same layout:

- instructions for the assistant (tone, rules, what to do when unsure)
- definitions of a couple of tools, like filing a time-off request
- the entire handbook, roughly 20,000 tokens of it
- Marcus's question, which is usually a sentence or two

Here are two of Marcus's requests side by side:

![Two requests to the handbook assistant drawn as strips: instructions, tools, the 20,000-token handbook, and a question. A bracket over the first three parts says same every time; only the question changes.](https://www.kirupa.com/ai_fundamentals/images/prompt_caching_two_requests.png)

Almost all of each request is the same. Only the last little segment changes. Without caching, the model does all of the work for all of those tokens on both requests, and we pay full input price both times.

## What the Model Works Out as It Reads

To see what can be saved, we need to peek at what the model does when it reads a prompt.

As the model processes our prompt, it works out a bundle of intermediate numbers for every token. These are called key-value states (often shortened to the KV cache). You can think of them as the model's notes on each token, recording what that token means in the context of everything before it. When the model later writes its answer, it consults these notes instead of re-reading the raw text.

The detail that makes caching possible is the direction of those notes. In the models we're talking about, a token's notes depend only on the tokens to its left. The note for the word ***vacation*** on page 12 of the handbook depends on pages 1 through 12, and not on Marcus's question at the end. Here is what that looks like:

![Six tokens, Our vacation policy gives 15 days, each with a note above it. A bracket under the first four tokens says built from these, showing that the note for gives depends only on itself and the tokens before it. The last two tokens are faded and labeled not these.](https://www.kirupa.com/ai_fundamentals/images/prompt_caching_looks_left.png)

This means that if two prompts start with the same 20,000 tokens, the notes for those 20,000 tokens come out identical. The model can work them out once, keep them, and reuse them the next time it sees that same beginning.

That beginning is called the prefix, and prompt caching is the practice of saving the notes for a prefix so the next matching request can skip straight to the new part.

## A Cache Hit, Step by Step

Let's walk through Marcus's first two questions with caching turned on.

On the first request, there's nothing saved yet. The model processes all of the tokens, writes the answer, and the provider stores the notes for the prefix. (Some providers do this automatically, and others ask us to mark where the reusable part ends. More on that shortly.)

On the second request, the provider checks whether it has notes for the beginning of this prompt. It does. The instructions, tools, and handbook match token for token, so their notes are loaded rather than recomputed. The model only has to work through Marcus's new question before it starts answering.

Here is the second request with the reused part marked:

![The second request with instructions, tools, and handbook marked with check marks and labeled loaded from the cache. Only the question is labeled computed now.](https://www.kirupa.com/ai_fundamentals/images/prompt_caching_hit.png)

Two good things happen at once. We pay less, because providers charge a reduced rate for cached input tokens. And the answer starts sooner, because skipping 20,000 tokens of processing takes a real bite out of the wait before the first word (see [Time to First Token](https://www.kirupa.com/ai_fundamentals/time_to_first_token.htm) for why that wait exists).

## What It Costs

Caching isn't free. Saving the notes has a cost, and the saved notes don't live forever.

The details vary by provider and model, so here's one documented example at the time of writing. OpenAI's [prompt caching guide](https://developers.openai.com/api/docs/guides/prompt-caching) says that for its newer models, a cache write costs 1.25 times the normal input rate and a cache read costs 0.1 times the normal input rate on most of those models. Their own worked example: across ten requests that share a prefix, one write plus nine reads costs about 2.15 times the price of that prefix, compared with 10 times if every request processed it from scratch.

Here is that comparison drawn out:

![Ten requests sharing a prefix. Without caching, ten blocks of 1x add up to 10x. With caching, one 1.25x write plus nine thin 0.1x reads add up to 2.15x. Example from OpenAI's prompt caching guide.](https://www.kirupa.com/ai_fundamentals/images/prompt_caching_cost.png)

Anthropic's version works on the same principle with different knobs. In its [prompt caching docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), we mark the reusable part with `cache_control`, the default lifetime is five minutes, and every cache hit refreshes that lifetime at no extra cost. A longer one-hour lifetime is available for a higher write price. Providers also set a minimum prefix length before anything gets cached, so caching a 40-token [system prompt](https://www.kirupa.com/ai_fundamentals/system_prompts_vs_user_prompts.htm) won't do anything.

So basically, caching pays off when the same long prefix gets reused several times within its lifetime. A handbook assistant that Marcus and his coworkers hit all day is a great fit. A prompt that's used once an hour, with a cache that expires after five minutes, would pay the write cost and rarely see a read.

## How We Accidentally Break It

The cache only helps when the prefix matches exactly, from the very first token. A single changed token anywhere invalidates the notes for everything after it, because every one of those later notes depended on it.

The most common way this happens is innocent-looking. Let's say someone adds a helpful line to the very top of the instructions:

```text
Current time: 2026-11-12 09:41:07
```

The intention was good. The effect is that the prompt now starts differently every second, so the 20,000-token handbook after it never matches anything saved. Here is that failure next to the fix:

![Avoid: a changing timestamp at the very start of the prompt, so the instructions, tools, handbook, and question after it all have to be recomputed. Better: instructions, tools, and handbook first and cached, with the timestamp and question computed at the end.](https://www.kirupa.com/ai_fundamentals/images/prompt_caching_timestamp.png)

Other common culprits look the same once you know the rule:

- putting the user's name or ID into the instructions at the top
- adding, removing, or reordering tool definitions between requests
- building the prompt from a dictionary or JSON object whose key order isn't stable
- placing freshly retrieved documents before the stable material instead of after it
- switching to a different model, since each model keeps its own cache

The fix is one layout rule. We put everything stable at the start, in a fixed order, and everything that changes at the end. Instructions, then tools, then long reference material, then the conversation, then the newest message.

## Caching the Answer Is a Different Thing

It's worth being clear about what prompt caching doesn't do. It doesn't remember answers. If Marcus asks ***How many vacation days do I get?*** twice, the model still writes a fresh answer the second time. The only thing reused is the work of reading the shared beginning.

Reusing whole answers for similar questions is a separate idea with its own risks, and we cover it in [Semantic Caching](https://www.kirupa.com/ai_fundamentals/semantic_caching.htm).

## Conclusion

A model works out notes for every token it reads, and each note depends only on the tokens before it. That's why two prompts that start the same way produce the same notes for that shared start, and why a provider can save them and skip the work next time. We pay less for those cached tokens, and the reply starts sooner.

The catch is that the match has to be exact from the first token, and the saved notes expire. For Marcus's handbook assistant, that meant keeping instructions, tools, and the handbook in a fixed order at the front, pushing the timestamp and the question to the end, and making sure the questions arrive often enough to keep the cache warm. If your app sends a long, stable prompt many times a minute, this is one of the easiest cost and speed wins available!