# Semantic Caching

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

Ask a hundred travelers about pets on planes, and you'll get a hundred different sentences. ***Can I bring my cat in the cabin?*** ***Are pets allowed on board?*** ***My dog is small, can he fly with me?*** They're all the same question wearing different outfits.

If an AI assistant is answering these, it would be nice to recognize the repeats and hand back an answer we already have, instead of paying a model to write a fresh one each time. A regular cache can't do that, because it only recognizes exact matches. Semantic caching is the version that recognizes questions by meaning.

In this article, we'll build one, see the clever part, and then spend real time on the part where it goes wrong, because that's where the interesting decisions are.

Onwards!

## Puffin Air's Help Bot

Let's say we work at a small airline called Puffin Air, and we built a help bot named Skye. Each answer takes a model call that costs us a little money and a couple of seconds. Thousands of travelers ask Skye things every day, and a big share of those questions are about the same handful of topics: pets, baggage, seat changes, and cancellations.

Here is Skye, sitting in the corner of Puffin Air's website:

![Skye sits in a chat panel in the corner of Puffin Air's website. The panel header shows a robot avatar with a bird badge, the name Skye, and the tagline Puffin Air help. A traveler asks, Can I bring my cat in the cabin? Skye replies, Yes, in a carrier! Your cat can ride with you in the cabin. A handwritten note says This is Skye!](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_meet_skye.png)

The obvious first move is a regular cache. We store each question and its answer in a lookup table, and when the exact same question arrives again, we return the stored answer.

Here is what happens when two travelers ask about pets:

![Two travelers ask about pets. An exact-match cache holds can I bring my cat in the cabin, so the identical question is a hit and Are pets allowed on board? is a miss, even though it asks the same thing.](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_exact_miss.png)

The second traveler asked the same thing and got a miss, because the text was different. In practice, exact repeats are rarer than you'd hope. People phrase things their own way, add typos, and say ***plz***.

## Matching by Meaning

To match by meaning, we need a way to turn a sentence into something we can measure distance on. That something is an [embedding](https://www.kirupa.com/ai_fundamentals/embeddings.htm): a list of numbers, produced by an embedding model, that places a piece of text as a point in a space where similar meanings land near each other. (How a model turns text into embeddings is a story of its own.)

Here is a simplified, flattened picture of where some travel questions land:

![Travel questions drawn as points by meaning. Four pet questions cluster together, three baggage questions cluster in another corner, and a seat question sits apart from both.](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_map.png)

Real embeddings have hundreds or thousands of dimensions, not two, but the idea holds. ***Can I bring my cat in the cabin?*** and ***Are pets allowed on board?*** land close together even though they share almost no words.

So a semantic cache stores entries differently. For each answered question, it saves the question's embedding alongside the answer. When a new question arrives:

1. We compute the new question's embedding.
2. We find the stored question whose embedding is closest to it.
3. We compute a similarity score between the two (cosine similarity is the usual choice, where 1.0 means pointing the same way).
4. If the score is above a threshold we picked, we return the stored answer. That's a hit.
5. If not, we call the model, return its answer, and store the new question, embedding, and answer for next time.

Here is that flow:

![A flowchart: a new question is embedded, matched to its nearest stored question, and checked against a 0.92 threshold. Yes returns the stored answer; no calls the model and then stores the new answer.](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_flow.png)

The threshold of 0.92 in that picture is illustrative. The right value depends on the embedding model and on our data, and we'll see shortly why picking it is the hard part.

When it works, it's lovely. The second traveler's question scores, say, 0.94 against the stored cat question, clears the bar, and gets the stored answer back without any call to the big model. We skip the model call entirely.

Here is that cache hit from the traveler's side:

![Skye's chat window on Puffin Air's website. A traveler asks, Are pets allowed on board? A small note shows a cache hit with a score of 0.94 and no model call. Skye replies, Yes, in a carrier! Your cat can ride with you in the cabin.](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_chat_hit.png)

## When Close Isn't the Same

Now for the curveball. Two questions can be very close in meaning and still need completely different answers.

Consider these two:

- ***Can I cancel my flight for free?***
- ***Can I change my flight for free?***

Only one word is different, and both are about the same booking. Their embeddings will land very near each other. But at Puffin Air, let's say changes are free within 24 hours and cancellations come with a fee. If the cancel question is in the cache and the change question scores above our threshold, Skye confidently gives the change-seeker the cancellation answer.

Here is the change-seeker's chat:

![Skye's chat window on Puffin Air's website. A traveler asks, Can I change my flight for free? A small note shows a cache hit on the stored cancellation answer. Skye replies, Cancellations come with a fee. A handwritten note flags that reply: Changes are free!](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_chat_cancel.png)

Here's how that feels from the traveler's side:

![A parrot on the Puffin Air counter squawks Cancellation fee applies! at a traveler holding a ticket, who thinks: I asked about changing it.](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_parrot.png)

This is the central trade-off of semantic caching. If we set the threshold low, we get lots of hits, including some wrong ones like this. If we set it high, the wrong hits mostly disappear, but so do many of the correct hits we wanted. Here is a flattened sketch of the threshold as a boundary around one stored question (the real boundary lives in hundreds of dimensions, so treat the circle as a picture of the idea rather than the geometry):

![Two panels with a stored question A at the center of a dashed match boundary. B is a true rephrasing and C asks about changing a flight. With a loose 0.85 threshold both fall inside, so C gets A's answer. With a strict 0.95 threshold only B matches and C goes to the model.](https://www.kirupa.com/ai_fundamentals/images/semantic_caching_threshold.png)

Since there's no universal right number, we pick the threshold by testing. We gather real question pairs, label which ones truly should share an answer, and see how many correct and wrong hits each threshold produces. (This is the classic trade-off between [recall and precision](https://www.kirupa.com/ai_fundamentals/recall_vs_precision.htm) in a new costume.) For a help bot where a wrong answer costs a frustrated customer, we lean strict.

## Deciding What Goes in the Cache

The threshold isn't the only guard. Some answers should never be shared at all.

***What time does my flight leave?*** is a question many travelers ask in nearly identical words. The answer is different for every one of them. If Skye cached that answer and served it to the next person, it would be wrong at best and a privacy leak at worst. Anything personalized, anything that depends on who is asking, and anything that used private data has to skip the cache or be cached per user.

A few other rules keep things sane:

- Scope the cache. Questions in different languages, different products, or different customer tiers get separate caches, so an answer for one never leaks to another.
- Expire entries. When Puffin Air changes its pet policy, every cached pet answer is stale. Give entries a lifetime, and clear related entries when the source content changes.
- Cache only answers we'd stand behind. If a response got a thumbs-down or came from a fallback path, don't save it for reuse.

## Is It Worth It?

Semantic caching adds work to every request: an embedding call and a nearest-neighbor search, even on misses. That's usually cheap next to a large model call, but it isn't free, and it adds a little latency to every miss.

It pays off when lots of users ask overlapping questions with answers that don't depend on who's asking. Help bots, documentation assistants, and FAQ-style tools fit that shape. A coding assistant where every request includes a different file probably doesn't, because almost nothing will ever match.

The way to know is to measure two numbers once it's running: the hit rate (how often we skip the model) and the wrong-hit rate (how often a hit was the wrong answer, which we find by sampling hits and reviewing them). A high hit rate means nothing if the wrong-hit rate is creeping up.

If you want to try this without building it from scratch, libraries such as [GPTCache](https://github.com/zilliztech/GPTCache) and the [semantic cache in RedisVL](https://docs.redisvl.com/en/latest/user_guide/03_llmcache.html) implement this pattern.

## Conclusion

A regular cache only recognizes exact repeats, and people rarely repeat themselves exactly. A semantic cache stores an embedding for each question, finds the closest stored question for each new one, and returns the saved answer when the similarity clears a threshold.

For Skye, that meant pet questions in many different phrasings could share one answer. It also meant facing the hard part: a question about changing a flight sits right next to one about canceling it. We handle that with a threshold chosen from labeled examples, by keeping personalized answers out of the shared cache, and by expiring entries when policies change.

One naming note before you go: semantic caching reuses whole answers, which is a completely different trick from [Prompt Caching](https://www.kirupa.com/ai_fundamentals/prompt_caching.htm), where the provider reuses the work of reading a repeated prompt and the model still writes a fresh answer every time. Many apps use both!