Query Rewriting

by kirupa   | filed under AI Fundamentals

Priya is chatting with Trailhead's help assistant. The conversation goes like this:

Priya: I bought the Ridgeline 2 tent last spring and I love it.

Assistant: Glad to hear it! How can I help with your Ridgeline 2?

Priya: does it come in green

To a person, that last message is crystal clear. It is the Ridgeline 2, and she wants to know about color options. Our retrieval system, though, doesn't read the whole conversation. By default, it takes the latest message, turns it into an embedding, and searches with it. So it searches for:

does it come in green

Here is what comes back:

1. Trailhead gift cards: the Green Trails design
2. Our move to green, recyclable packaging
3. Bike helmet color options

Not one mention of a tent. The search did exactly what we asked. The problem is that we asked the wrong question. The thing the user typed and the thing we should search for aren't always the same, and fixing that gap has a name: query rewriting.

Here is the mismatch:

Onwards!

Rewrite First, Then Search

The fix is to add a step before retrieval. We hand the conversation to a model and ask it to write a standalone search query, one that makes sense with no conversation around it. For Priya's message, that might be:

Ridgeline 2 tent colors green

Now the search has something to work with, and the Ridgeline 2 product page (which lists Forest green and Desert tan) comes back on top.

Here is what Priya sees once the rewrite is in place:

The rewriting prompt doesn't need to be fancy. Something like this goes a long way:

Here is a conversation between a customer and Trailhead's help assistant.
Rewrite the customer's latest message as a standalone search query.
Replace pronouns like "it" with what they refer to.
Keep product names and model numbers exactly as written.
Don't answer the question. Only output the query.

Here is the new flow:

This is the most common kind of rewriting, and in any chat-style assistant, it's close to mandatory. Real people say it, that one, what about the bigger size?, and same question for boots. Searching on those words directly is like walking into a library and asking the librarian does it come in green?

More Than One Way to Rewrite

Once we have a rewriting step, it can do more than resolve pronouns. Let's keep following Priya, who has a follow-up:

is it lighter than the Summit 3?

Splitting a Question Into Parts

That question needs two facts: the weight of the Ridgeline 2 and the weight of the Summit 3. Those facts live on two different product pages. A single search might find one of them and miss the other, because one query can only land in one spot on the embedding map.

So the rewrite step can split the question into two searches:

1. Ridgeline 2 tent weight
2. Summit 3 tent weight

We run both, combine the results, and the model gets both weights to compare. This is often called query decomposition.

Here is that split:

(Those weights are made up for our made-up store. Please don't plan a trip around them.)

Asking the Same Thing Several Ways

Another trick is to have the model write a few different phrasings of the same question, run a search for each, and merge the results. For example, Ridgeline 2 weight, how heavy is the Ridgeline 2, and Ridgeline 2 packed weight specifications. Each phrasing lands in a slightly different spot, so together they're more likely to catch a page that describes weight in unusual words. Merging the lists works just like it does in Hybrid Search, with Reciprocal Rank Fusion. This is usually called multi-query retrieval.

Searching With a Pretend Answer

The strangest trick on this list is to have the model write a short pretend answer and search with that instead of the question. The idea is that answers look more like other answers than questions do. A made-up paragraph that says The Ridgeline 2 weighs about two kilograms and packs down to the size of a loaf of bread will land right next to the real product page on the embedding map, even if its details are wrong. We only use the pretend answer to search, and we never show it to anybody. This technique is called HyDE (Hypothetical Document Embeddings), from a 2022 paper by Luyu Gao and colleagues.

Here is how it works, step by step:

The Costs

Rewriting isn't free, and it has a couple of ways to go wrong.

First, it adds a model call. Every question now waits for a rewrite before retrieval can even start. That adds time before the user sees anything. A small, fast model is usually plenty for rewriting.

A rewrite can also change the question. It's a model's guess at what the user meant, and guesses can be wrong. If Priya asks does it come in green and the rewriter decides she means olive, we've searched for something she didn't ask, and nobody will notice unless they go looking. A few habits keep this in check:

• Tell the rewriter to keep names, numbers, and codes exactly as written.
• Search with the rewritten query and the original, then merge the results. If the rewrite drifts, the original still pulls its weight.
• Log both the original and the rewrite, so when an answer goes wrong, you can see whether the problem started at the rewrite.

And sometimes rewriting isn't needed at all. A clear, standalone question like What's the return window for tents in Canada? doesn't benefit from rewriting. Some systems ask the rewriter to return the question unchanged when it's already fine, which keeps it from improving things that weren't broken.

Conclusion

Priya's does it come in green was a perfectly good message and a terrible search query. The difference between what someone types and what we should search for is the whole reason query rewriting exists. A rewrite step, sitting between the user and the retriever, can resolve pronouns from the conversation, split a comparison into separate lookups, try several phrasings, or even search with a pretend answer.

The trade-off is an extra model call and the risk of drifting from what the user actually meant, so we keep the rewriter small, tell it to preserve exact names, search with the original too, and log everything.

And yes, the Ridgeline 2 comes in Forest green. Priya is already thinking about a second one.

Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.

Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.

Your support keeps this site going! 😇