Recall vs. Precision

by kirupa   | filed under AI Fundamentals

Theo's tent leaked during the last storm, right along the seams on the rain fly. He asks Trailhead's help assistant:

How do I reseal the seams on my tent?

Behind the scenes, our retrieval system grabs the 10 chunks it thinks are most relevant and passes them to the model. Some of those chunks are great. Some are about waterproofing jackets. One is about tent stakes for some reason.

How do we say whether that retrieval was good? Pretty good isn't something we can track, compare, or improve. We need numbers, and the two numbers that people reach for are recall and precision. They sound like jargon, but each one answers a very plain question, and once you've got those two questions in your head, a lot of retrieval tuning starts to make sense.

Onwards!

A Net in a Lake

Before we get to Theo, let's go fishing for a moment.

Imagine a lake with exactly 4 trout in it. We want the trout. We throw a net, pull it in, and find 10 things inside: 3 trout, plus an old boot, two soda cans, three clumps of weeds, and a very confused rubber duck.

There are two very different ways to judge that catch:

• Did we get the trout? There were 4 trout in the lake, and we caught 3. That's recall: 3 out of 4, or 75%.
• Is our net full of trout? We pulled up 10 things, and 3 are trout. That's precision: 3 out of 10, or 30%.

Here is our catch:

Recall looks at what we missed. Precision looks at what we dragged in by mistake. A huge net catches every trout and a lot of boots, so recall goes up and precision goes down. A tiny net held right where we spotted a fish catches almost nothing but trout, so precision goes up and recall goes down.

Back to Theo

Retrieval is the same game. The lake is Trailhead's index. The trout are the chunks that actually help answer Theo's question. Let's say there are exactly 4 of them:

1. the seam sealing steps in the tent care guide
2. the product page for Trailhead's seam sealer
3. an FAQ about leaks at the seams
4. the warranty note about factory-taped seams

Our retriever returned 10 chunks, and 3 of the 4 helpful ones are among them. It missed the warranty note.

• Recall = helpful chunks we retrieved ÷ helpful chunks that exist = 3 ÷ 4 = 75%
• Precision = helpful chunks we retrieved ÷ chunks we retrieved = 3 ÷ 10 = 30%

Here is the same idea drawn as two overlapping groups:

The Dial Between Them

The easiest knob we have is k, the number of chunks we retrieve. Let's say the 4 helpful chunks show up at positions 1, 3, 7, and 16 in the retriever's ranked list. Here's what happens as we turn the dial:

Here is that dial as three snapshots:

Pulling more chunks catches more of the good stuff and more of the junk at the same time. There's no free setting where both numbers are perfect, unless our ranking is perfect, and it never is.

Which One Matters More?

In a RAG system, both matter, but they fail in different ways.

Low recall means a helpful chunk never reached the model. The model can't use what it never saw. At best, it says I'm not sure. At worst, it fills the gap with something plausible and wrong. In Theo's case, missing the warranty note means the assistant never mentions that his tent's seams might be a free factory repair. That's an answer that's technically fine and still lets Theo down.

Low precision means the model got a pile of junk along with the good stuff. That costs tokens on every request, eats into the context window, and gives the model more chances to get distracted. If one of the junk chunks is about waterproofing a jacket with a spray, the model might helpfully suggest spraying the tent seams, which is not how seam sealer works.

Here is that junk chunk sneaking into Theo's answer:

The useful thing to notice is that the two problems aren't equally fixable later. Junk can be removed after retrieval. A reranker can push it down, and the model can ignore it. A missing chunk can't be recovered by anything downstream. So the usual strategy is:

1. Make the first retrieval step high recall. Cast a wide net, maybe 30 or 50 chunks, so the helpful chunks are almost certainly in there.
2. Make the next step high precision. Rerank and keep only the best few for the model.

Here's that two-step shape:

Measuring It for Real

There's one honest snag in all of this. To compute recall, we need to know how many helpful chunks exist, and in a real index of thousands of chunks, nobody knows that off the top of their head.

The practical answer is to build a small test set by hand. Pick a few dozen real questions, like Theo's, and for each one, have a person mark which chunks actually help. Then run your retriever on those questions and compute recall and precision at your chosen k (often written as recall@10, precision@10, and so on). It's tedious work, but it turns I think the new chunking is better into a sentence like recall@10 went from 72% to 88%, and that's a sentence you can make decisions with.

A hand-labeled set like this is often called a golden dataset, and Golden Datasets covers how to build and maintain one. Once that test set exists, you can rerun it every time you change chunk sizes, embedding models, filters, or rerankers, and see which number moved.

Conclusion

When Theo asked about resealing seams, our retriever pulled 10 chunks and found 3 of the 4 that would have helped. Recall told us what we missed (the warranty note), and precision told us how much junk came along (7 chunks). Turning the k dial trades one for the other.

For RAG, the playbook is to protect recall early, because nothing downstream can bring back a chunk that was never retrieved, and then buy precision back with reranking and trimming before the model sees anything.

As for Theo, once we widened the first net to 30 chunks and reranked, the warranty note made it into the answer. His seams may be a free factory repair, which beats a tube of seam sealer and a very sticky afternoon.

Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.

Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.

Your support keeps this site going! 😇