Reranking

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

Omar just got back from a muddy week in the mountains, and his sleeping bag smells like a wet campfire. He asks Trailhead's help assistant:

Can I put my down sleeping bag in the washing machine?

Trailhead's help center has the answer. It's in a section of the care guide for the Alpine 20 sleeping bag: use a front-loading machine on a gentle cycle with a down-specific soap, avoid top-loaders with a center agitator, and tumble dry on low with a couple of tennis balls to break up the clumps.

Our retrieval system does find that chunk. It just finds it at position #14. Above it are chunks about choosing between down and synthetic bags, washing rain jackets, and machine-washable sleeping bag liners. They're all in the same neighborhood as Omar's question, but none of them answers it. If we only hand the model the top 5, the answer never makes it into the prompt.

Here is what Omar's top results look like:

[Image: Omar's top five results are Down vs. synthetic bags, Washing a rain jacket, Machine-washable liners, Storing your sleeping bag, and Down jacket care, above a dashed line marking what goes to the model. The chunk that answers him, Alpine 20 care: washing, sits below the line at number 14, with a handwritten note: The answer is down here!] (https://www.kirupa.com/ai_fundamentals/images/reranking_before.png)

We could just send all 30 results to the model. That works sometimes, but every extra chunk costs tokens (https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm), takes up room in the context window (https://www.kirupa.com/ai_fundamentals/context_window_limits.htm), and gives the model more near-miss text to get distracted by. What we really want is to put the best chunks at the top of the list. That's what a reranker does.

Onwards!

Why the First Search Is a Bit Rough

To see why the answer landed at #14, we need to look at how the first search works.

In embedding (https://www.kirupa.com/ai_fundamentals/embeddings.htm)-based search, the question and each document are turned into embeddings separately. The document embeddings were computed days ago, long before Omar ever asked anything. At question time, we embed his question and look for nearby points. The question and the documents never actually meet. We're comparing two summaries that were made independently.

That's what makes the first search so fast. All the expensive work on the documents happens ahead of time, so searching millions of chunks takes milliseconds. The price we pay is nuance. A one-list-of-numbers summary of Can I put my down sleeping bag in the washing machine? captures the topic (down, sleeping bags, washing) but not the precise thing being asked. Lots of chunks share that topic.

Reading Them Together

A reranker takes a different approach. It's a model that reads the question and one candidate chunk together, as a single input, and outputs a score for how well that chunk answers that question.

Because it sees both at once, it can notice things a pair of separate summaries can't. It can see that the question says washing machine and the chunk says front-loading machine. It can see that the question asks can I and the chunk gives a yes-with-conditions answer. It can see that a chunk about rain jackets is about washing, but not about down sleeping bags.

Here are the two approaches side by side:

[Image: Two panels. In the first search, the question and the document are each embedded separately into lists of numbers, and only the lists are compared. In the reranker, the question and the document go in together as one input, and out comes a relevance score of 0.94.] (https://www.kirupa.com/ai_fundamentals/images/reranking_together.png)

The catch is speed. A reranker can't do anything ahead of time, because it needs the question, and the question only shows up when Omar asks it. So for every candidate chunk, we have to run the reranker once at question time. Doing that for millions of chunks would take forever. Doing it for 30 is fine.

The Two-Step Funnel

So we use both, in order:

1. Retrieve a generous shortlist with the fast first search (vector, keyword, or hybrid (https://www.kirupa.com/ai_fundamentals/hybrid_search.htm)). Say, the top 30.
2. Rerank those 30 by reading each one together with the question.
3. Keep the top few after reranking, say 5, and send those to the model.

A hiring process is a decent mental picture for this. A recruiter skims thousands of résumés for the right keywords and makes a shortlist of 30. That skim is quick and a bit crude. Then someone actually interviews the 30, which takes much longer per person but gets a much better read. Nobody interviews thousands of applicants, and nobody hires straight off a keyword skim.

Here is the funnel for Omar's question:

[Image: Four stages, each smaller than the last: an index of 40,000 chunks, a fast retrieval step that keeps the top 30, a reranker that rescores all 30, and the model, which gets the top 5.] (https://www.kirupa.com/ai_fundamentals/images/reranking_funnel.png)

After reranking, Omar's list looks very different:

[Image: After reranking, Alpine 20 care: washing is number 1, marked was number 14. Below it are Storing your sleeping bag, Down vs. synthetic bags, Down jacket care, and Machine-washable liners. The rain jacket chunk has dropped out of the top 5.] (https://www.kirupa.com/ai_fundamentals/images/reranking_after.png)

Choosing a Reranker

There are a couple of flavors.

Dedicated reranker models are trained for exactly this job: question plus passage in, relevance score out. They're often called cross-encoders, because they read (encode) the question and passage across each other in one pass. You can call hosted ones, like Cohere's Rerank, or run open models yourself. They're usually small and fast relative to a chat model.

General LLMs as rerankers also work. You give the model the question and a handful of candidates and ask it to score or order them. This can be very good, especially for tricky domain questions, but it's slower and more expensive per candidate, so it fits best when the shortlist is short.

Whichever you pick, measure two things on your own questions: how often the right chunk ends up in the final top few, and how much time reranking adds to each request. A reranker that improves accuracy but adds a noticeable pause before every answer is a trade-off you want to make on purpose. (It shows up in Time to First Token (https://www.kirupa.com/ai_fundamentals/time_to_first_token.htm), the wait before the first word of the answer appears, because the model can't start writing until reranking is done.)

What Reranking Can't Do

There's one limit worth keeping front and center. A reranker can only reorder what it's given. If the care guide chunk had been #45 instead of #14, and we only retrieved 30, the reranker would never have seen it. No amount of clever reordering can rescue a chunk that wasn't on the shortlist.

Here are the two cases side by side:

[Image: Two cases for the care guide chunk when we retrieve 30 candidates. At #14, it's on the shortlist, so the reranker moves it to #1. At #45, it was never retrieved, so the reranker never sees it.] (https://www.kirupa.com/ai_fundamentals/images/reranking_shortlist.png)

That's why the first step usually grabs more candidates than we'll end up using. The first search's job is to make sure the right chunk is somewhere in the pile. The reranker's job is to put it on top. Those two jobs have names, recall and precision, and they get their own article: Recall vs. Precision (https://www.kirupa.com/ai_fundamentals/recall_vs_precision.htm).

Conclusion

Omar's answer was in our index the whole time. The fast first search found it, but ranked it #14 because it only compared quick summaries of the question and the chunks. By adding a reranker that reads the question and each candidate together, we moved the right chunk from #14 to #1 and only sent the model the five chunks that mattered.

Here is the answer Omar gets now:

[Image: Trailhead's help assistant chat window. Omar asks, Can I put my down sleeping bag in the washing machine? The assistant replies, Yes! Use a front-loading machine on a gentle cycle with a down-specific soap, then tumble dry on low with a couple of tennis balls.] (https://www.kirupa.com/ai_fundamentals/images/reranking_chat_sleeping_bag.png)

This pattern of a quick, wide pass followed by a slower, careful pass over a few finalists isn't unique to RAG. Web search engines and recommendation feeds have used versions of it for a long time, for the same reason we did: there's too much to read carefully, so we only read carefully at the end.

Omar's bag, by the way, is tumbling in the dryer with two tennis balls. It sounds like a sneaker convention in there.