# How to Build an Agentic RAG System - Tool: Any LLM or framework - Difficulty: Advanced - Time required: 50-60 min - Compatibility: Python 3.10+, LangChain 1.x, Supabase with pgvector enabled, Claude or OpenAI API key - Last updated: November 2026 ## TL;DR Agentic RAG means the agent decides when and how many times to retrieve, not just retrieving once before generating an answer. This guide builds a documentation Q&A agent with a search_docs tool over a Supabase pgvector table, wired into LangChain's create_agent with a system prompt that tells it to refine and re-search when the first retrieval doesn't fully answer the question. Budget 50-60 minutes and a Supabase project with pgvector enabled. ## What you'll build You'll build a documentation Q&A agent for an internal API. Ask it something the docs only partly cover in one section, and instead of answering from whatever the first search returned, it notices the gap, searches again with a refined query, and only then answers, citing which sections it actually used. That's the whole difference agentic RAG makes over plain RAG. Plain RAG retrieves a fixed number of chunks once, stuffs them into the prompt, and generates, no judgment involved. Agentic RAG puts retrieval behind a tool call: the agent decides whether to search, what to search for, and whether the results are good enough to answer from, the same way it would decide whether to call any other tool. The retrieval logic (your embeddings, your pgvector table, your similarity function) doesn't change, what changes is who's in control of when it runs. This guide uses LangChain's create_agent with a single search_docs tool, because the agentic behavior comes entirely from the system prompt's instructions and the model's own judgment, not from a specialized RAG framework. Reach for a heavier RAG-specific stack (LlamaIndex's FunctionAgent, a dedicated reranker, a query planner) once you need routing across multiple document sets or sub-query decomposition this single-tool pattern can't express cleanly. ## Before you start - A Supabase project with the pgvector extension enabled and a documents table of embedded chunks - Python 3.10+ and a terminal - pip install langchain langgraph supabase - An embedding provider already used to populate the documents table (so query embeddings match its dimensions) ## Step-by-step guide ### 1. Decide you actually need agentic RAG, not plain RAG Plain RAG (retrieve once, generate) is simpler, cheaper, and fine when questions map cleanly to one relevant chunk, FAQ-style lookups. Agentic RAG earns its extra tool-calling turns when questions are ambiguous, span multiple sections, or a weak first retrieval is common enough that blind generation produces wrong answers often. If your docs Q&A rarely needs a second search, plain RAG is the better default. **Expected result:** A specific reason ('questions here often span two sections' or similar) for taking on agentic RAG's extra complexity and cost. ### 2. Confirm your Supabase match function exists Agentic RAG doesn't change your retrieval infrastructure, it changes who calls it. You need a documents table with an embedding column and a Postgres function (commonly named match_documents) that does the pgvector similarity search. If you already have this from a plain-RAG setup, skip ahead; if not, this is the standard shape. ``` create or replace function match_documents( query_embedding vector(1536), match_count int default 5 ) returns table (id bigint, content text, section text, similarity float) language sql stable as $$ select id, content, section, 1 - (embedding <=> query_embedding) as similarity from documents order by embedding <=> query_embedding limit match_count; $$; -- Remember RLS: documents likely needs a read policy, even if writes are service-role only. ``` **Expected result:** select * from match_documents(, 5) returns rows directly in the Supabase SQL editor. ### 3. Wrap retrieval in a tool the agent calls search_docs embeds the query, calls match_documents through the Supabase client, and formats the results as text with section tags so the model can cite them. embed_query() is whatever embedding call already populated your documents table, kept as a separate function so you can swap providers without touching the tool's shape. ``` from langchain.tools import tool from supabase import create_client import os supabase = create_client(os.environ["SUPABASE_URL"], os.environ["SUPABASE_SERVICE_ROLE_KEY"]) def embed_query(text: str) -> list[float]: # Call whichever embedding API populated your documents table. ... @tool def search_docs(query: str, match_count: int = 5) -> str: """Search the internal API docs for sections relevant to a query. Returns matching sections with content.""" embedding = embed_query(query) res = supabase.rpc("match_documents", {"query_embedding": embedding, "match_count": match_count}).execute() if not res.data: return "No matching documentation found. Try a more specific or differently worded query." return "\n\n".join(f"[{row['section']}] {row['content']}" for row in res.data) ``` **Expected result:** search_docs('rate limits') returns one or more [section] content blocks when called directly, no agent involved. ### 4. Write the system prompt that makes retrieval agentic The tool alone doesn't make this agentic RAG, the instructions do. Tell the model explicitly to judge whether the returned chunks answer the question, to refine and search again when they don't, and to cite the sections it actually used rather than answering from memory. Cap the behavior at a fixed number of searches so it doesn't re-query indefinitely on a question the docs simply don't cover. ``` SYSTEM_PROMPT = ( "You answer questions about our API using search_docs. Call it first. " "If the results only partly answer the question, refine your query and " "call search_docs again, up to 3 times total. Always cite the [section] " "tags you actually used. If you still can't find the answer after 3 " "searches, say so honestly instead of guessing." ) ``` **Expected result:** A system prompt that turns a passive retrieval tool into an agent that decides when one search isn't enough. ### 5. Wire it into create_agent and run an ambiguous question Pass search_docs and the system prompt into create_agent. Test with a question that genuinely spans two documentation sections, and watch the trace: you should see search_docs called once, then a second call with a refined query, before the final cited answer. ``` from langchain.agents import create_agent agent = create_agent( model="anthropic:claude-sonnet-5-5", tools=[search_docs], system_prompt=SYSTEM_PROMPT, ) result = agent.invoke({"messages": [ {"role": "user", "content": "How do rate limits interact with the batch endpoint?"} ]}) for m in result["messages"]: print(type(m).__name__, getattr(m, "content", "")) ``` **Expected result:** Two search_docs calls visible in the trace (rate limits, then batch endpoint), followed by an answer citing both sections. ### 6. Harden it: dedupe chunks and cap cost Two real risks show up at scale: the same chunk coming back from two refined searches (dedupe by id before formatting results), and retrieval rounds compounding cost on every ambiguous question (embedding calls plus an extra model turn per search). Log how many searches the average question takes; if most need just one, consider tightening the prompt's threshold for when a second search is worth it. ``` seen = set() deduped = [] for row in res.data: if row["id"] not in seen: seen.add(row["id"]) deduped.append(row) ``` **Expected result:** A search_docs that never returns the same chunk twice across refined queries, and a rough number for average searches per question. ## Complete code example File: `docs_rag_agent.py` ```python """Agentic RAG docs Q&A agent: search_docs over Supabase pgvector, with refinement. Run: python docs_rag_agent.py (pip install langchain langgraph supabase; set env vars) """ import os from langchain.agents import create_agent from langchain.tools import tool from supabase import create_client supabase = create_client(os.environ["SUPABASE_URL"], os.environ["SUPABASE_SERVICE_ROLE_KEY"]) def embed_query(text: str) -> list[float]: """Call whichever embedding API populated the documents table.""" raise NotImplementedError("Wire this up to your embedding provider") @tool def search_docs(query: str, match_count: int = 5) -> str: """Search the internal API docs for sections relevant to a query. Returns matching sections with content.""" embedding = embed_query(query) res = supabase.rpc( "match_documents", {"query_embedding": embedding, "match_count": match_count} ).execute() if not res.data: return "No matching documentation found. Try a more specific or differently worded query." seen = set() chunks = [] for row in res.data: if row["id"] in seen: continue seen.add(row["id"]) chunks.append(f"[{row['section']}] {row['content']}") return "\n\n".join(chunks) SYSTEM_PROMPT = ( "You answer questions about our API using search_docs. Call it first. " "If the results only partly answer the question, refine your query and " "call search_docs again, up to 3 times total. Always cite the [section] " "tags you actually used. If you still can't find the answer after 3 " "searches, say so honestly instead of guessing." ) agent = create_agent( model="anthropic:claude-sonnet-5-5", tools=[search_docs], system_prompt=SYSTEM_PROMPT, ) if __name__ == "__main__": result = agent.invoke({"messages": [ {"role": "user", "content": "How do rate limits interact with the batch endpoint?"} ]}) for m in result["messages"]: print(type(m).__name__, "->", getattr(m, "content", "")) ``` ## Common mistakes - **Calling this 'agentic RAG' when it's really one fixed retrieval step before generation** — undefined Fix: Agentic RAG requires the model to decide whether and how to re-query. If there's no instruction or mechanism for a second search, it's plain RAG with a tool-shaped wrapper. - **Not capping the number of retrieval rounds** — undefined Fix: Cap it explicitly in the system prompt (and ideally enforce it in code) or an unanswerable question can trigger search after search with no exit condition. - **Letting duplicate chunks from refined searches pile up in context** — undefined Fix: Dedupe by chunk id across all searches in a conversation before formatting results, or the model burns context re-reading the same passage. - **Mismatching the query embedding's dimension against the table's stored embeddings** — undefined Fix: This fails at the SQL level with a dimension-mismatch error, not a helpful application error. Confirm your embedding provider matches what populated the table. - **Forgetting a read policy on the documents table** — undefined Fix: Supabase tables are inaccessible by default without RLS policies. A service-role key bypasses RLS, but confirm that's actually the key this tool uses before assuming reads will just work. ## Best practices - Default to plain RAG; only add agentic retrieval once you can point at real questions a single retrieval answers badly. - Cap retrieval rounds explicitly, both in the prompt and, if the stakes justify it, in code that counts tool calls. - Dedupe chunks across refined searches in the same conversation before they reach the model's context. - Require citations (section tags, doc IDs) in the final answer so a wrong answer is traceable to a specific retrieval. - Log how many searches real questions take; a high average search count is a sign your chunking or embeddings need work, not just your prompt. - Keep embed_query() isolated from the tool logic so swapping embedding providers doesn't touch the retrieval or agent code. ## Use cases ### undefined ### undefined ### undefined ## Frequently asked questions ### What's the difference between RAG and agentic RAG? Plain RAG retrieves a fixed set of chunks once and generates from them. Agentic RAG puts retrieval behind a tool call the model decides to use, including deciding whether to search again when the first results don't fully answer the question. ### Do I need a specialized RAG framework to build agentic RAG? No. This guide builds it with a single custom tool and a system prompt on top of LangChain's create_agent. A specialized stack (LlamaIndex, a dedicated reranker) earns its complexity once you need routing across multiple document sets or multi-step query decomposition. ### How many times should the agent be allowed to re-search? Three is a reasonable default for most docs Q&A. Cap it explicitly in the prompt, and if the cost or latency matters, enforce the cap in code by counting tool calls rather than trusting the model to stop on its own. ### Why does my agent sometimes answer without citing any sources? Usually the system prompt doesn't require citations explicitly enough, or the model answered from its own training knowledge instead of calling search_docs first. State the citation requirement and the 'always search first' rule as separate, explicit instructions. ### How much does an agentic RAG query cost compared to plain RAG? Roughly the cost of one extra model turn and one extra embedding call per additional search, on top of the same per-chunk context cost plain RAG already pays. Two searches on a $2/M-input model adds a few thousand extra input tokens, typically under a cent. ### Can someone build this for me instead of doing it myself? Yes. RapidDev builds production RAG and agentic RAG systems on Supabase pgvector and LangChain, including the chunking, reranking, and citation work most tutorials skip. A free 30-minute consultation is the fastest way to scope your version. --- Source: https://www.rapidevelopers.com/build-ai-agent/how-to-build-an-agentic-rag-system © RapidDev — https://www.rapidevelopers.com/build-ai-agent/how-to-build-an-agentic-rag-system