Agentic RAG by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm) Felix bought a camp stove from Trailhead last month, and he's packing for a trip. Something is nagging at him, though, so he asks Trailhead Help: Is the stove I bought last month safe to use? Notice what Felix didn't say, which is which stove. Everything we've built in the RAG & Retrieval articles so far handles a message the same way. It searches once with the message's words and hands whatever comes back to the model. Searching Trailhead's help center for Felix's words turns up stove-safety pages (maybe even a recall notice), but nothing in those results says which stove is his. Here is what happens when we let Trailhead Help decide for itself how to search: [Image: Trailhead's help assistant chat window. Felix asks, Is the stove I bought last month safe to use? A small note shows the first lookup, in Felix's orders, which finds an Ember camp stove. A second note shows a help center search, which finds the Ember stove safety recall. The assistant replies, Please stop using it! Your Ember camp stove is part of a safety recall, and the recall notice asks owners to return it [1].] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_chat_stove.png) Trailhead Help searched twice, and the second search depended on the first. Its lookup_orders call found an Ember camp stove in Felix's orders. Then its search_help_center call looked for that exact stove and found the Ember recall from Keeping the Index Fresh (https://www.kirupa.com/ai_fundamentals/keeping_the_index_fresh.htm). A model that decides when to search, where to search, and how many times is the idea behind agentic RAG. In the following sections, we'll see how it works, the four decisions it hands to the model, what those decisions cost, and when plain RAG is all we need. Onwards! Plain RAG Searches Once Every RAG system we've built so far is a fixed pipeline. A message comes in, our code searches once using the message's words (maybe after a rewrite, as in Query Rewriting (https://www.kirupa.com/ai_fundamentals/query_rewriting.htm)), and the model writes an answer from whatever came back. Our code decides what happens, in the same order, every single time: [Image: Two messages go through the same fixed pipeline: one search with the customer's exact words, then the model. Thanks, that's all! still triggers a search that returns store hours, gift cards, and a packing list, which it didn't need. Is the stove I bought last month safe to use? returns camp stove safety basics, choosing stove fuel, and the Ember recall, but the model still doesn't know which stove Felix bought.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_plain_once.png) Look at the two rows. In the top one, Felix says thanks, that's all!, and the pipeline searches anyway, pulling in store hours, gift cards, and a packing list that nobody asked for. In the bottom one, the pipeline runs the search for Felix's stove question, and the results even include the Ember stove safety recall. The model still doesn't know which stove Felix bought, so it can't tell whether that recall applies to him. The best it can do is hedge, or guess. None of this makes plain RAG bad. It works great for most questions, which tend to be one-shot lookups like Rosa's warranty question in What Is RAG? (https://www.kirupa.com/ai_fundamentals/what_is_rag.htm). It struggles when the right search depends on something the model has to find out first, or when the first search comes back with nothing useful. From a Pipeline to a Loop Agentic RAG turns searching into a tool. As we saw in Tool Calling (https://www.kirupa.com/ai_fundamentals/tool_calling.htm), a tool is something the model can ask our code to run by writing a structured request. Instead of searching before the model ever sees the message, we give the model some search tools and let it decide when to use them. That turns our pipeline into the same loop we met in How an Agent Like Claude Code Works (https://www.kirupa.com/ai_fundamentals/how_an_agent_works.htm): [Image: A loop. The context holds Felix's question, the chat, and any search results. The model decides to answer or to ask for a search. Our code runs the search the model picked, in the help center, the products, or Felix's orders. The results are added to the context, and the model decides again. When the model has what it needs, it answers.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_loop.png) Follow the arrows around the loop. The context holds Felix's question, the chat so far, and any results. The model decides whether to answer or to ask for a search. When it asks, our code runs the search it picked, the results get added to the context, and the model decides again. When the model finally has what it needs, the loop ends at Model answers. The model never searches anything itself. It writes a request, and our code does the searching, just like the clerk passing notes in Tool Calling. Here are the search tools Trailhead Help gets to pick from: [Image: Three tools the model can pick from. search_help_center looks through guides, policies, and recalls. search_products looks through product pages for specs, colors, and what's in the box. lookup_orders lists the signed-in customer's orders, using their login.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_tools.png) Each tool is described the way Tool Calling showed us, with a name, a plain-English description, and a schema for its arguments. Here is what two of them might look like: [ { "name": "search_help_center", "description": "Search Trailhead's help center: guides, policies, warranties, and safety recalls.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "A standalone search query. Spell out product names." } }, "required": ["query"] } }, { "name": "lookup_orders", "description": "List the signed-in customer's recent Trailhead orders.", "parameters": { "type": "object", "properties": {} } } ] Notice that lookup_orders takes no arguments at all. Our code gets the customer from their login, never from anything they typed, which is the same rule we followed in Permission-Aware Retrieval (https://www.kirupa.com/ai_fundamentals/permission_aware_retrieval.htm). The model gets to decide whether to look at Felix's orders, but it never gets to decide whose orders it's looking at. The Four Decisions Giving the model search tools lets it make four decisions that plain RAG makes the same way every time. Let's walk through each one with Felix. Should I Search at All? Not every message needs a search. Here's how Trailhead Help sorts out three of them: [Image: Three messages and what the model decides. Thanks, that's all! needs no search, so it answers directly. Can you say that in Spanish? also needs no search, because the facts are already in the chat. Is the stove I bought last month safe to use? needs facts it doesn't have, so it searches first.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_decide.png) Start with the top row. thanks, that's all! needs no facts at all, so Trailhead Help answers directly. Can you say that in Spanish? needs only facts that are already sitting in the chat, so it answers directly too. Only the stove question needs facts the model doesn't have yet, so that's the one where it searches first. This decision cuts both ways. Skipping pointless searches saves time and money. But a model that decides it already knows the answer can slip right back into the made-up answers that RAG was supposed to fix, like the one-year warranty Rosa got in What Is RAG?. So we nudge it in the system prompt, with something like Always search before answering anything about Trailhead's products, policies, or orders. When a turn absolutely needs a search, our code can also require one. OpenAI's API calls this a tool_choice of "required" (https://developers.openai.com/api/docs/guides/function-calling), and Anthropic's calls it a tool_choice of "any" (https://platform.claude.com/docs/en/agents-and-tools/tool-use/implement-tool-use). Either way, the model has to call a tool on that turn. Which Source? Trailhead's facts live in more than one place. Policies and recalls live in the help center, specs live on the product pages, and Felix's purchases live in the orders system. With plain RAG, we have to choose which index to search ahead of time, or search all of them and hope. With agentic RAG, the model picks the tool that fits the question, guided by each tool's description. That's why Trailhead Help started with lookup_orders. Felix asked about the stove I bought, and the only thing that knows what Felix bought is his order history. This puts a lot of weight on the tool descriptions. A vague description like search stuff leaves the model guessing, which is the same lesson Tool Calling taught us with check_stock. Did That Search Work? Plain RAG takes whatever the first search returns, good or bad. An agent can look at the results and decide they aren't good enough. Let's say Felix follows up with also my pole thingy snapped in the wind, can i get a new one. Here are the two searches Trailhead Help runs: [Image: Search 1 uses Felix's own words, pole thingy snapped new one, and the best results score only 0.42, 0.39, and 0.38, so the model rewrites the query. Search 2, Ridgeline 2 replacement pole, finds the pole replacement guide with a score of 0.86, which is strong enough to answer from.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_retry.png) Search 1 uses a trimmed-down version of Felix's own words, and the best results score only 0.42, 0.39, and 0.38. The model can see those scores, because our search tool returns them, and it can also just read the titles and see that none of them are about replacing a pole. So it does what Query Rewriting (https://www.kirupa.com/ai_fundamentals/query_rewriting.htm) did as a separate step, and it writes a better query. Felix's order history also lists a Ridgeline 2 tent, so Search 2 looks for Ridgeline 2 replacement pole, and the Pole replacement guide comes back at 0.86. That's strong enough to answer from. Is the Answer Backed Up? The last decision happens right before replying. Instead of sending its first draft, the agent can check each claim against the source it came from and fix anything that doesn't match: [Image: The draft answer says Felix's pole is covered by the warranty, so Trailhead will send a new one. Source 2, the pole replacement guide, says poles damaged by wind or misuse are not covered under warranty, but replacements can be purchased. The check finds they don't match, so the final reply says wind damage isn't covered by the warranty, but Felix can buy a replacement pole.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_check.png) Compare the Draft answer with source [2]. The draft says Felix's pole is covered by the warranty. The Pole replacement guide says poles damaged by wind or misuse are not covered. The check catches that mismatch, so the Final reply tells Felix the truth: wind damage isn't covered, but he can buy a replacement pole. This is the checking from Citation Grounding (https://www.kirupa.com/ai_fundamentals/citation_grounding.htm), done as one more step in the loop. It can be a quote check in plain code, or a second model call that asks whether each passage supports each claim. What It Costs All of this flexibility has a price, and the most obvious part of it is model calls. Here's Felix's stove question, answered both ways: [Image: Plain RAG answers Felix's question with one search and one model call. Agentic RAG takes three model calls and two lookups: the model asks for his orders, then asks for a help center search, then writes the answer. Each model call sends the whole conversation so far again.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_costs.png) Plain RAG answered with one search and one model call. Agentic RAG needed three model calls, because after each lookup, the model has to decide what to do next. As we saw in How an Agent Like Claude Code Works, every trip around the loop is a full request that resends everything so far, so the cost grows faster than the number of steps. Each trip adds waiting, too. Felix sees nothing until the final answer starts, which is exactly the delay that Time to First Token (https://www.kirupa.com/ai_fundamentals/time_to_first_token.htm) is about. Predictability is part of the price as well. Two runs of the same question can take different paths, which makes agentic RAG harder to test and harder to debug than a fixed pipeline. When an answer goes wrong, we have to read through the model's choices step by step to find out where. When the Searches Come Back Empty Sometimes the thing the model is looking for just isn't there. If Felix asks Do you sell bear-proof coolers? and Trailhead doesn't, a model that's determined to find an answer can keep rewriting its query forever. That's why every agentic RAG loop needs the stop conditions from Agent Step Limits and Stop Conditions (https://www.kirupa.com/ai_fundamentals/agent_step_limits.htm): [Image: Felix asks, Do you sell bear-proof coolers? Search 1 for bear-proof cooler is weak, search 2 for bear resistant cooler is weak, and search 3 for hard-sided cooler is weak too. The step limit of three searches stops the loop, and the assistant says it couldn't find one and offers to check with the team.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_limits.png) Each search comes back weak, and each rewrite is a small variation on the last one. After Search 3, the step limit stops the loop. Instead of an error, Trailhead Help tells Felix it couldn't find a bear-proof cooler and offers to check with the team. The limit lives in our code, so the model can't talk its way past it, and a token budget or a repeat detector would work the same way. When Plain RAG Is Enough So when is all of this worth it? Anthropic's Building effective agents (https://www.anthropic.com/research/building-effective-agents) post recommends starting with the simplest solution possible, and it points out that for many applications, "optimizing single LLM calls with retrieval and in-context examples is usually enough". It also says that agentic systems "often trade latency and cost for better task performance". Here's how that shakes out for RAG: [Image: Two columns. Stick with plain RAG when one source has the answer, one search finds it, speed matters most, and costs must be predictable. Reach for agentic RAG for multi-step questions, several sources or tools, vague questions, and when wrong answers are costly.] (https://www.kirupa.com/ai_fundamentals/images/agentic_rag_when.png) Start with the left column. When one source has the answer and one search finds it, plain RAG is faster, cheaper, and easier to test, and most of Trailhead's questions look like that. The right column is where agentic RAG pays off. Felix's stove question was a multi-step question that needed several sources or tools, and since it was about safety, a wrong answer would have been costly too. It isn't all or nothing, either. As we'll see in Workflows vs. Agents (https://www.kirupa.com/ai_fundamentals/workflows_vs_agents.htm), a fixed workflow can include one agent-like step, such as a small loop that searches the help center until it finds something relevant. Plenty of real systems land right there, with plain RAG for the easy questions and an agentic loop for the hard ones. Coding agents use the same idea on code. Claude Code's search tools (https://code.claude.com/docs/en/how-claude-code-works) find files by pattern and search their contents, and the model decides what to look for next based on what it just found. If you want to dig deeper, the research world has names for some of these pieces. Self-RAG (https://arxiv.org/abs/2310.11511), from a 2023 paper by Akari Asai and colleagues, trains a model to decide when to retrieve and to critique what it retrieved and wrote. Corrective RAG (https://arxiv.org/abs/2401.15884), from a 2024 paper by Shi-Qi Yan and colleagues, adds a small evaluator that grades the retrieved documents and triggers other actions, including a web search, when they look weak. Conclusion Plain RAG searches once for every message, using the message's own words, and that works for most questions. Felix's question was different. It never named the stove, so the right search depended on something the model had to find out first. Agentic RAG handles that by turning searching into tools and running the model in a loop. The model decides whether to search at all, which source to search, whether the results are good enough or need another try with a better query, and whether its answer actually matches its sources. Our code still runs every search, keeps the customer's identity out of the model's hands, and enforces the step limits that stop a loop from spinning. The price is more model calls, more waiting, and less predictable behavior, so it's worth reaching for when the questions actually need it. That wraps up RAG & Retrieval. So far, every request we've looked at goes to one model through one API key. Real apps call several models from several providers, and keeping track of the keys, quotas, and routing for all of them is the topic of Model Gateway for Keys, Quotas, and Routing (https://www.kirupa.com/ai_fundamentals/model_gateway.htm), the first stop in AI System Architecture. As for Felix, he returned the Ember, is packing a different stove, and has a replacement pole on the way.