What Is RAG? by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm) Trailhead is an outdoor gear shop, and it wants a help assistant on its website. Customers should be able to ask about tents, packs, stoves, repairs, warranties, and returns, and get a real answer even at two in the morning while they're standing in a soggy campsite. So the team builds one and calls it Trailhead Help. Here is Trailhead Help, sitting in the corner of Trailhead's website and waiting for its first customer: [Image: Trailhead's help assistant sits in a chat panel in the corner of Trailhead's outdoor gear website. The panel header shows a robot avatar with a tent badge, the name Trailhead Help, and the tagline Tents, packs, and repairs. The assistant says, Hi! What can I help you with today? Rosa asks, Is my tent still under warranty? I bought it 18 months ago. A handwritten note says Trailhead's helper!] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_meet_trailhead.png) Rosa is that first customer. A seam on her tent's floor has started to peel, and before she packs the tent up and brings it in, she wants to know one thing: is it still under warranty? If we wire Trailhead Help straight to a capable model and send it Rosa's question with nothing else, here is what comes back: [Image: Trailhead's help assistant chat window, wired straight to a model with no documents. Rosa asks, Is my tent still under warranty? I bought it 18 months ago. The assistant replies, Trailhead tents come with a one-year warranty, so yours has expired. A handwritten note flags that reply: It's two years!] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_chat_guess.png) Trailhead Help sounds completely sure of itself, and it's wrong. Trailhead's warranty covers tents against manufacturing defects for two years, so Rosa has about six months left. The model didn't look anything up. It wrote the most plausible-sounding reply it could, and a one-year warranty sounds perfectly plausible. The fix for this has a slightly clunky name: retrieval-augmented generation, or RAG for short. In the following sections, we'll see why the model fumbled Rosa's question, what RAG does about it, the two halves that every RAG system is made of, and when RAG is (and isn't) the right tool. We'll finish with a map of the rest of the RAG & Retrieval articles, because each piece we meet here gets an article of its own. Onwards! Why the Model Doesn't Know Back in What Exactly Is an LLM? (https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm), we saw that everything a model knows lives in its weights, a giant pile of numbers tuned on an enormous amount of text. Two details from that article explain Rosa's wrong answer. The first detail is where that text came from: mostly the public web, books, and code. Trailhead's warranty PDF was never in there. Neither was its return policy, its repair guides, or the product pages for the tents it sells. That's private data, and the model has simply never seen it. The second detail is that the weights are frozen once training ends. Anything newer than the model's knowledge cutoff (the date its training data stops) isn't in there either, and Trailhead's help center changes every week. A new tent goes on sale, a price changes, and on one memorable Tuesday, a camp stove gets recalled. Even a model that had somehow read every page Trailhead ever published would be out of date by Friday. Here is what that looks like from the model's side: [Image: Two panels. In the weights, learned from public text: how tents work, keeping a stove lit, what a warranty is, and how returns work. Not in the weights: Trailhead's warranty PDF and return policy, which are private to Trailhead, and Tuesday's stove recall and this week's prices, which are newer than the model's knowledge cutoff.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_not_in_weights.png) Look at the two panels. The left one is full of things the model picked up from public text, like how tents work and what a warranty is. The right one holds everything Rosa actually needs. Trailhead's warranty PDF and return policy are private, so they were never in the training data. Tuesday's stove recall and this week's prices are newer than the cutoff, so they couldn't have been. So why didn't the model just say I don't know? As we saw in Why AI Makes Things Up (https://www.kirupa.com/ai_fundamentals/why_ai_makes_things_up.htm), a model always has a next token. When it doesn't know something, its odds spread across a bunch of plausible answers, one of them gets picked, and the sentence comes out sounding just as sure as a real fact. One year was one of those plausible answers. An Open-Book Exam We can't retrain a model every time Trailhead edits a help page. What we can do is hand the model the right page at the moment Rosa asks. Think about the difference between a closed-book exam and an open-book exam. In a closed-book exam, everything has to come from memory, and when memory runs out, people guess. In an open-book exam, we can flip to the right page and answer from it. RAG turns every question into an open-book exam, with one twist: before the model ever sees the question, our code has already flipped to the right page and put it on the desk. Here are the two exams side by side: [Image: Two panels with the same model. Closed book: the only thing on the desk is Rosa's question, so the model answers from memory and guesses one year. Open book: our code has already put the Tents section of Trailhead's warranty PDF on the desk next to her question, so the model reads it and answers two years.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_open_book.png) The model is exactly the same in both panels. The only thing that changed is what's on the desk, which is the context window from Context Window Limits (https://www.kirupa.com/ai_fundamentals/context_window_limits.htm). In the closed book, the desk holds Rosa's question and nothing else, so the model answers from memory. In the open book, our code adds the Tents section of the warranty PDF before the model reads a single token, and the answer comes straight off the page. Three Words, Three Steps The name sounds intimidating, but each of its three words is one step, and the steps happen in order. Retrieval means our code searches Trailhead's documents for the passages most related to Rosa's question. Augmented means we add those passages to the prompt, right next to her question. Generation means the model writes the answer, using what we handed it. Here are the three steps for Rosa's question: [Image: Three cards joined by arrows. R, Retrieval: search Trailhead's documents, which finds the Tents section of the warranty. A, Augmented: add the passages to the prompt, which makes a bigger prompt. G, Generation: the model writes the answer, two years, citing source 1.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_three_words.png) Follow Rosa's question from left to right. Retrieval finds Warranty › Tents, the passage that answers her. Augmented turns that passage and her question into a bigger prompt. Generation is the model doing what it always does, which is writing a reply. The difference is that this reply says two years and comes with a little [1] pointing back at its source. The name comes from a 2020 paper by Patrick Lewis and colleagues (https://arxiv.org/abs/2005.11401) at Facebook AI Research, University College London, and New York University. Their version searched Wikipedia, and they trained the searching part and the writing part together. Today, the term covers almost any system that looks things up and hands what it found to a model, and that's the meaning we'll use. Lewis later told NVIDIA's blog (https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/) that the team always planned to come up with a nicer-sounding name, but when it came time to write the paper, nobody had a better idea. (Naming things is famously one of the hard problems in computer science, so we'll let it slide.) The Two Halves Every RAG system is built from two halves that run at very different times. One half gets the documents ready ahead of time, long before anybody asks anything. The other half runs every single time a question arrives. Getting the Documents Ready The first half is usually called indexing, and it happens in four steps: [Image: Four steps joined by arrows, all done ahead of time. Load: gather Trailhead's documents. Chunk: cut them into passages. Embed: turn each passage into numbers. Store: save the passages and their numbers in an index.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_indexing.png) Let's walk through each step, using Trailhead's help center: 1. We load the documents. That means pulling the warranty PDF, the help articles, the product pages, and the policies out of wherever they live and turning them into plain text. 2. We chunk them. A 40-page warranty PDF is far too big to hand over whole, so we cut every document into smaller passages called chunks, often a paragraph or a few paragraphs each. 3. We embed each chunk. An embedding is a list of numbers that captures what a piece of text means, so that passages with similar meanings end up with similar numbers. This is what lets us search by meaning instead of by exact words. 4. We store every chunk, its embedding, and a few labels (like which document it came from) in an index, which is the searchable collection we'll look things up in. The index usually lives in a vector database, a database built to find the closest embeddings quickly. The expensive part, cutting up and embedding thousands of documents, happens once. After that, we only redo the work for documents that change. Answering a Question The second half runs every time someone asks something. Here is what happens when Rosa's question arrives: [Image: Rosa's question, Is my tent still under warranty?, flows through four steps. Embed: turn the question into numbers. Retrieve: find the closest chunks, read from the index built ahead of time. Assemble: put the instructions, the chunks, and the question into one prompt. Generate: the model writes a cited answer, two years, citing source 1.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_answering.png) Let's walk through these steps too: 1. We embed Rosa's question with the same embedding model we used on the chunks, so the question and the chunks can be compared. 2. We retrieve the chunks whose embeddings are closest to the question's, usually just the top few. For Rosa, the winner is the Tents section of the warranty PDF. 3. We assemble the prompt out of some instructions, the retrieved chunks with a short ID on each one, and Rosa's question. 4. The model generates the answer from those chunks and cites the ID of each chunk it used. The assembled prompt is plain text, and for Rosa's question, it might look like this: You are Trailhead Help, the help assistant for Trailhead. Answer using only the sources below. Cite a source for each claim, like [1]. If the sources don't answer the question, say so and offer to connect the customer with our team. [1] Warranty PDF › Tents Tents are covered against manufacturing defects for two years. This coverage does not include zipper damage caused by sand or debris. [2] Pole replacement guide Poles damaged by wind or misuse are not covered under warranty, but replacements can be purchased. Customer question: Is my tent still under warranty? I bought it 18 months ago. Notice the instructions at the top. The model has the real warranty text in front of it now, and telling it to stick to the sources (and to say so when the sources come up empty) is a big part of what turns a confident guess into a grounded answer. Also notice that the second chunk doesn't really answer Rosa's question. Retrieval rarely hands over only perfect passages, which is one of the many reasons the rest of this group exists. Here is what Rosa sees now: [Image: Trailhead's help assistant chat window. Rosa asks, Is my tent still under warranty? I bought it 18 months ago. A small note shows the search step, which found the Tents section of the warranty PDF. The assistant replies, Yes! Trailhead tents are covered against manufacturing defects for two years, so you have about six months left [1].] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_chat_warranty.png) Rosa asked the exact same question as before. This time, the little search note shows Trailhead Help finding Warranty PDF › Tents, and the reply comes straight from it: two years, about six months left, and a [1] she can click to check the source for herself. Making sure those citations really back up their claims is the job of Citation Grounding (https://www.kirupa.com/ai_fundamentals/citation_grounding.htm). Why Not Fine-Tune, or Paste Everything In? Two other fixes might come to mind, and both are worth a look, because they explain why RAG is so popular. The first is fine-tuning, which means training the model further on Trailhead's documents so that it knows them. In Prompting vs. Fine-Tuning (https://www.kirupa.com/ai_fundamentals/prompting_vs_fine_tuning.htm), we saw why that's a poor fit here. Fine-tuning is good at teaching a model how to respond. It's unreliable at teaching new facts, and it's a bad match for facts that change. Every time Trailhead edited its warranty, we'd have to train again, and in the meantime, the model would keep stating the old warranty with total confidence. With RAG, we update the document, re-index it, and the very next answer is right. The second is to skip searching entirely and paste every document into the prompt, every time. That actually works when the documents are small. Biscuit, the Crumb & Co. bot from earlier articles, sends its system prompt and all of its bakery facts with every request, and as we saw in Tokens and How They Set Your Bill (https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm), that's only about 1,500 tokens. Anthropic's advice on this (https://www.anthropic.com/news/contextual-retrieval) is that if a knowledge base is smaller than about 200,000 tokens (roughly 500 pages), you can put all of it in the prompt and skip RAG altogether. That works even better with prompt caching (https://www.kirupa.com/ai_fundamentals/prompt_caching.htm), which lets the provider reuse the work it already did on the unchanging start of a prompt, so repeat requests get cheaper and faster. Trailhead's help center is a different story. Let's say it gets cut into 40,000 chunks of 300 tokens each. That's 12 million tokens, which is way beyond the few hundred thousand (or, for some models, a million or more) that we saw models accept in Context Window Limits (https://www.kirupa.com/ai_fundamentals/context_window_limits.htm). Even if it did fit, that same article showed us two more problems. We'd pay for every one of those tokens on every question. And models get less reliable at finding one detail buried in a huge pile of text. Here are the three options side by side: [Image: Three options. Fine-tune trains facts into the weights: it has to be retrained on every change, and it's shaky on new facts. Paste it all in sends every document with every question: great for small documents, but Trailhead's 12 million tokens won't fit. RAG fetches what this question needs: updating a document is all it takes, and each question pays for only a few chunks.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_three_options.png) Look at the right-hand side of each row. Fine-tuning gets two crosses. It bakes facts into the weights, so every change means retraining. New facts don't stick reliably, either. Pasting it all in is great for small docs, but Trailhead's 12 million tokens won't fit. RAG fetches only what each question needs. When Trailhead updates a doc, the next answer reflects it, and each question pays for a few chunks instead of the whole help center. None of this makes RAG free. It adds moving parts, like an index we have to build, a search we have to tune, and a copy of our documents we have to keep up to date, and each of those can go wrong in its own way. The approaches also mix well. A team might fine-tune a model on Trailhead's tone of voice and still use RAG for the facts, because the tone rarely changes and the facts change all the time. A Map of What's Ahead RAG in one sentence is simple: find the right passages and hand them to the model. Making that work well, on real documents with real customers, is where things get interesting. Each piece of the pipeline we just walked through gets an article of its own, and here is where each one fits: [Image: A map of the RAG pipeline. Ahead of time: load, covered in this article; chunk, in Chunk Size and Overlap; embed, in Embeddings; and store, in Vector Databases and Keeping the Index Fresh. Every question: rewrite, in Query Rewriting; search, in Hybrid Search, Metadata Filtering, and Permission-Aware Retrieval; rerank, in Reranking; and answer, in Citation Grounding. Around it all: measure it, in Recall vs. Precision, and loop it, in Agentic RAG.] (https://www.kirupa.com/ai_fundamentals/images/what_is_rag_map.png) The Ahead of time column is the indexing half, and the Every question column is the answering half. Here are the articles in the order we'll meet them: - Embeddings (https://www.kirupa.com/ai_fundamentals/embeddings.htm) shows how text becomes numbers, where nearby means similar. - Vector Databases (https://www.kirupa.com/ai_fundamentals/vector_databases.htm) covers where those numbers live and how we find the closest ones quickly. - Chunk Size and Chunk Overlap (https://www.kirupa.com/ai_fundamentals/chunk_size_and_overlap.htm) is about how big each passage should be. - Hybrid Search (https://www.kirupa.com/ai_fundamentals/hybrid_search.htm) combines meaning search with good old keyword search. - Metadata Filtering (https://www.kirupa.com/ai_fundamentals/metadata_filtering.htm) narrows a search to the right country, product, or version. - Reranking (https://www.kirupa.com/ai_fundamentals/reranking.htm) adds a slower, more careful pass that puts the best chunks on top. - Recall vs. Precision (https://www.kirupa.com/ai_fundamentals/recall_vs_precision.htm) measures whether retrieval found the right stuff. - Query Rewriting (https://www.kirupa.com/ai_fundamentals/query_rewriting.htm) turns what people type into a better search. - Permission-Aware Retrieval (https://www.kirupa.com/ai_fundamentals/permission_aware_retrieval.htm) keeps people out of documents they aren't allowed to read. - Citation Grounding (https://www.kirupa.com/ai_fundamentals/citation_grounding.htm) checks that every claim matches its source. - Keeping the Index Fresh (https://www.kirupa.com/ai_fundamentals/keeping_the_index_fresh.htm) keeps our copy in sync as the documents change. - Agentic RAG (https://www.kirupa.com/ai_fundamentals/agentic_rag.htm) lets the model decide when, where, and how many times to search. Conclusion Rosa's first answer came from the model's memory, and Trailhead's warranty was never in there. The model knows plenty about tents in general, but it has never seen Trailhead's private documents, and it can't know about anything that changed after its training ended. When it doesn't know, it guesses, and the guess sounds every bit as confident as a fact. RAG fixes that by turning every question into an open-book exam. Ahead of time, we load Trailhead's documents, cut them into chunks, embed each chunk, and store everything in an index. When a question arrives, we embed it, retrieve the closest chunks, assemble them into a prompt along with the question, and let the model write an answer that cites its sources. Unlike fine-tuning, an edit to a document shows up in the very next answer. Unlike pasting everything in, each question only pays for the few chunks it needs. Every step in that pipeline turns out to be a little rabbit hole of its own, and the step that everything else leans on is turning text into numbers that we can search by meaning. That's where we're headed next, in Embeddings (https://www.kirupa.com/ai_fundamentals/embeddings.htm). As for Rosa, she has about six months of warranty left, which is plenty of time to get that peeling seam looked at.