Embeddings

by kirupa   | filed under AI Fundamentals

Marcus is standing in a damp campsite, and the zipper on his tent won't move. He pulls out his phone and types this into the help assistant on Trailhead's website:

my tent zipper won't budge

Here is Marcus's question in Trailhead's help assistant, which sits in the corner of the store's website:

Trailhead has exactly the right help article for him. Its title is Fixing a jammed slider, and it talks about sand in the teeth, a bit of wax, and gently squeezing the slider back into shape. The problem is that Marcus's question and that article don't share a single important word. He said zipper and budge. The article says slider and jammed.

A search that matches words would walk right past the article. What we want is a search that matches meaning, and to do that we need a way to turn meaning into something a computer can compare. That something is an embedding, and it is one of the main ideas sitting under every system that answers questions from your own documents. We briefly met embeddings in How AI Generates Images and Video, where they let a model match pictures to captions. Here, we'll put them to work on text.

Here is the mismatch we need to solve:

Onwards!

Meaning as a Location

Let's start with a toy version that we can do by hand. Imagine we describe every help article with just two numbers:

1. How much is this article about tents? (0 means not at all, 1 means completely.)
2. How much is this article about fixing something that broke?

With those two questions, every article becomes a point on a map. Fixing a jammed slider is very much about tents and very much about repairs, so it lands at (0.9, 0.9). Choosing a tent for two people is all tent and no repair, so it lands at (0.9, 0.1). Patching a hole in your sleeping pad is all repair and barely tent, so it sits at (0.1, 0.9).

Here is what our little map of the help center looks like:

Now we do the same thing to Marcus's question. My tent zipper won't budge is mostly about a tent and mostly about something being broken, so it lands at (0.8, 0.7). Here is what happens when we drop it onto our map:

Marcus's question lands right next to the article he needs, even though the two share no words. The words were different, but the meaning put them in the same neighborhood.

That is the whole idea of an embedding. An embedding is a list of numbers that places a piece of text at a location, where nearby locations mean similar things.

From Two Numbers to Hundreds

Our toy has two problems. We picked the two questions ourselves, and two questions can't tell store hours apart from gift cards. Real embedding models fix both.

An embedding model is a neural network (a big pile of learned numbers, the same kind of thing that sits inside an LLM) that was trained on huge amounts of text to produce these lists of numbers. Nobody hand-picks the questions. The training process pushes texts that appear in similar contexts toward similar numbers, and the model ends up with its own internal sense of what's close to what. And instead of 2 numbers, it uses hundreds or thousands. OpenAI's text-embedding-3-small, for example, returns 1,536 numbers for every piece of text you give it (at the time of writing).

Two details change from our toy:

• The individual numbers usually don't mean anything a person could name. There is no tent-ness slot. Meaning is spread across all of the numbers together.
• We can't draw 1,536 dimensions (one axis per number, like the two axes on our toy map), so every picture you'll ever see of embeddings, including the ones in this article, is a squashed 2D sketch of something much bigger. The idea of nearby neighborhoods carries over to the real thing, even if the exact picture doesn't.

Here is what a real embedding looks like up close, and it is not very exciting:

Measuring "Nearby"

We've been saying near, so we need an actual way to measure it. The most common measure is cosine similarity, and it's friendlier than it sounds.

Picture each embedding as an arrow from the origin of our map out to its point. Cosine similarity looks at the angle between two arrows. If two arrows point in nearly the same direction, the score is close to 1. If they are at right angles, the score is 0. The length of the arrows doesn't matter, only where they point.

If we draw those arrows for our toy map and measure the angles, here is what we get:

Doing the math on our toy numbers gives us these scores:

The jammed-slider article wins, and it isn't close. With real 1,536-number embeddings, the math is exactly the same. There are just more numbers to multiply and add.

How a Search System Uses This

So basically, using embeddings for search is a two-part job. Some of the work happens ahead of time, and some happens when the question arrives:

1. Ahead of time, we run every help article (or every chunk of every article, meaning the smaller pieces we cut it into) through the embedding model and store the resulting numbers in an index. This is often called a vector database or vector store, because vector is the math name for a list of numbers like an embedding.
2. When Marcus asks, we run his question through the same embedding model, then ask the index for the stored embeddings closest to it.

Here is that flow:

The expensive part (embedding thousands of documents) happens once. At question time, we only embed one short question and look up its neighbors. For big collections, vector databases use clever approximate lookups so they don't have to compare against every single stored embedding, trading a tiny bit of accuracy for a lot of speed.

Once we have the nearest few articles, we paste them into the prompt and let the model write the answer. That is retrieval-augmented generation (aka RAG) in one sentence.

Here is that payoff in Marcus's chat:

Where Embeddings Fall Short

Embeddings are good at these mean roughly the same thing. That turns out to be a different skill from several things we might expect them to handle.

Suppose Marcus types TR-2041, the model number of his tent. To an embedding model, TR-2041 and TR-2014 look like near-identical strings of letters and digits, and they tend to land near each other on the map. A word-matching search would never confuse them. This is a big reason people combine both kinds of search, which we cover in Hybrid Search.

The word not is another weak spot. Tents that are waterproof and tents that are not waterproof are about the same topic, so their embeddings land close together. This means that similarity can tell us two texts are about the same thing, but it can't tell us whether they agree with each other.

Embeddings also know nothing about truth. An outdated return policy and the current one look almost identical as embeddings. The map has no idea which one is still true. That's a job for Metadata Filtering and for keeping the index up to date.

Here are those three weak spots together:

None of these are reasons to avoid embeddings. They are reasons to treat nearest on the map as a strong hint rather than a final answer, and to add other checks around it.

Conclusion

Marcus's problem was that his words and the help article's words didn't match, even though they meant the same thing. Embeddings get around that by turning each piece of text into a location, where distance stands in for difference in meaning. We embed the documents once, embed each question as it arrives, and grab whatever is closest.

What makes this work so well is that the model learned its sense of close from a mountain of real text, so it handles synonyms, rephrasings, and typo-ish questions without us writing a single rule. What it doesn't know is which document is current, which product code is exact, or whether a sentence says not. Those gaps are what the rest of the RAG & Retrieval articles are about, starting with a question we skipped right past: how big should each piece of text be before we embed it? That's the topic of Chunk Size and Chunk Overlap.

And Marcus? He found the wax tip, and his tent is zipped shut for the night.

Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.

Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.

Your support keeps this site going! 😇