# How to Build a Local AI Agent with Ollama - Tool: Ollama - Difficulty: Intermediate - Time required: 30-45 min - Compatibility: Ollama (macOS, Linux, or Windows), Python 3.10+, a machine with 16GB+ RAM for a mid-size local model - Last updated: November 2026 ## TL;DR A local AI agent runs entirely on your own machine through Ollama, with no cloud API calls and nothing leaving the device. This guide builds one with a tool-calling local model that answers HR policy questions by searching local markdown files, the same pattern as a cloud agent but without the network round trip or the data-residency risk. Budget 30-45 minutes and roughly 16GB of RAM to run the model. ## What you'll build You'll build an internal HR assistant that answers employee questions, like how many PTO days someone gets after two years, by searching a folder of local policy documents. Nothing in this agent, not the model, not the search step, not a single request, ever leaves the machine it runs on. For HR records, legal documents, or anything else your company can't send to a third-party API, that's not a nice-to-have, it's the whole reason to build it this way. Ollama runs an open-weight model as a local server on your own machine and exposes it through a simple chat API, the same shape as a cloud LLM API but pointed at localhost. Current tool-capable local models, like qwen3.8, can call functions the same way a cloud model does: you pass a list of tools, the model decides when to call one, and you run the matching Python function yourself and feed the result back for a final answer. This isn't a drop-in replacement for a frontier cloud model on hard reasoning. It's the right call when the data genuinely can't leave the building, when you want zero marginal cost per query, or when you need the agent to keep working with no internet connection at all. ## Before you start - Ollama installed and running (ollama.com) on a machine with at least 16GB of RAM - A tool-capable model pulled locally: ollama pull qwen3.8 - Python 3.10+ (pip install ollama) - A folder of real or sample policy documents in Markdown to search against ## Step-by-step guide ### 1. Install Ollama and pull a tool-capable model Install Ollama, then pull a model that actually supports tool calling; not every local model does. Filter Ollama's model library by the 'tools' capability before picking one, since using a non-tool-calling model will silently ignore your tools list instead of erroring clearly. ``` # Install Ollama from ollama.com, then: ollama pull qwen3.8 pip install ollama ``` **Expected result:** ollama list shows qwen3.8 downloaded, and ollama run qwen3.8 "hello" returns a normal reply. ### 2. Write a local tool with no network calls at all Define search_policies as a plain Python function that reads Markdown files from a local folder and returns matching snippets. Keep the docstring and type hints accurate and specific; the Ollama client reads them to build the tool's schema automatically, so a vague docstring produces a tool the model calls at the wrong time or with the wrong arguments. ``` from pathlib import Path POLICIES_DIR = Path("policies") def search_policies(query: str) -> str: """Search local HR policy documents for a keyword or phrase. Args: query: The word or phrase to search for across policy documents. """ hits = [] for path in POLICIES_DIR.glob("*.md"): text = path.read_text() if query.lower() in text.lower(): idx = text.lower().index(query.lower()) hits.append(f"[{path.name}] ...{text[max(0, idx-100):idx+200]}...") return "\n\n".join(hits) if hits else "No matching policy text found." ``` **Expected result:** Calling search_policies('PTO') directly returns real snippets from your local .md files, no network access involved. ### 3. Pass the function straight into the tools list Call ollama.chat with the model, a messages list, and tools=[search_policies], passing the actual function object rather than a hand-written JSON schema. The Python client inspects the function's signature and docstring and builds the tool definition for you, which keeps the tool's description and its real behavior from drifting apart. ``` import ollama MODEL = "qwen3.8" response = ollama.chat( model=MODEL, messages=[{"role": "user", "content": "How many PTO days after 2 years?"}], tools=[search_policies], ) ``` **Expected result:** response.message.tool_calls contains one call to search_policies with a query argument derived from the question. ### 4. Execute the tool call and send the result back Pull the function name and arguments off the model's response, run the matching local function yourself, and append the result as a tool-role message. Then send a second chat call with the full message history, including that tool result, so the model can write a final answer grounded in what the search actually found. ``` tool_calls = response.message.tool_calls or [] messages = [response.message] for call in tool_calls: result = search_policies(**call.function.arguments) messages.append({"role": "tool", "content": result, "name": call.function.name}) follow_up = ollama.chat(model=MODEL, messages=messages) print(follow_up.message.content) ``` **Expected result:** A final answer that quotes or paraphrases the actual policy text search_policies returned, not a generic guess. ### 5. Wrap it in a loop that generalizes past one tool As soon as you add a second tool, write the dispatch as a lookup dict instead of a chain of if statements, and loop on tool_calls until the model stops asking for any. This is the same two-call tool-use shape every cloud agent SDK automates for you; building it by hand once makes it obvious what those SDKs are actually doing underneath. ``` AVAILABLE_FUNCTIONS = {"search_policies": search_policies} def dispatch(call): fn = AVAILABLE_FUNCTIONS.get(call.function.name) return fn(**call.function.arguments) if fn else "Unknown tool." ``` **Expected result:** A dispatch function that scales to more tools without a growing if/elif chain. ### 6. Know the reliability ceiling and plan around it Third-party benchmarks have put tool-calling reliability on small local models in the 85-90% range, noticeably below a current frontier cloud model on the same task. For an HR FAQ bot, an occasional missed tool call just means a less complete answer; for anything higher-stakes, add a fallback, like asking the model to say it's unsure rather than guess, or routing the hardest queries to a cloud model when the data allows it. ``` SYSTEM_PROMPT = ( "Answer only from what search_policies returns. If nothing relevant is " "found, say the employee should ask HR directly. Never invent a policy detail." ) ``` **Expected result:** An agent that admits it doesn't know rather than fabricating a PTO policy that doesn't exist in your documents. ## Complete code example File: `local_hr_agent.py` ```python """Local HR policy agent built entirely on Ollama. Nothing leaves this machine. Run: python local_hr_agent.py (pip install ollama; ollama pull qwen3.8; put .md policy files in ./policies) """ from pathlib import Path import ollama MODEL = "qwen3.8" POLICIES_DIR = Path("policies") SYSTEM_PROMPT = ( "You are an internal HR assistant. Answer only from what search_policies " "returns. If nothing relevant is found, say the employee should ask HR " "directly. Never invent a policy detail." ) def search_policies(query: str) -> str: """Search local HR policy documents for a keyword or phrase. Args: query: The word or phrase to search for across policy documents. """ hits = [] for path in POLICIES_DIR.glob("*.md"): text = path.read_text() if query.lower() in text.lower(): idx = text.lower().index(query.lower()) snippet = text[max(0, idx - 100): idx + 200] hits.append(f"[{path.name}] ...{snippet}...") return "\n\n".join(hits) if hits else "No matching policy text found." AVAILABLE_FUNCTIONS = {"search_policies": search_policies} def dispatch(call) -> str: fn = AVAILABLE_FUNCTIONS.get(call.function.name) if fn is None: return "Unknown tool." return fn(**call.function.arguments) def run_turn(user_message: str, history: list[dict]) -> str: messages = history + [{"role": "user", "content": user_message}] response = ollama.chat(model=MODEL, messages=messages, tools=[search_policies]) tool_calls = response.message.tool_calls or [] if not tool_calls: return response.message.content messages.append(response.message) for call in tool_calls: result = dispatch(call) messages.append({"role": "tool", "content": result, "name": call.function.name}) follow_up = ollama.chat(model=MODEL, messages=messages) return follow_up.message.content def main(): history = [{"role": "system", "content": SYSTEM_PROMPT}] questions = [ "How many PTO days do I get after 2 years?", "What's the policy on remote work during probation?", ] for q in questions: print(f"Q: {q}") print(f"A: {run_turn(q, history)}\n") if __name__ == "__main__": main() ``` ## Common mistakes - **Picking any local model without checking it actually supports tool calling** — undefined Fix: Filter Ollama's model library by the 'tools' capability first. A non-tool-calling model accepts a tools argument without erroring and simply never calls one, which looks like a bug in your code when it isn't. - **Writing a vague docstring on a local tool function** — undefined Fix: The docstring and type hints are the only description the model gets, since the client auto-generates the schema from them. A vague one produces a tool called at the wrong time or with malformed arguments. - **Assuming a local model's tool-calling reliability matches a frontier cloud model's** — undefined Fix: Budget for a lower hit rate, roughly 85-90% on current small local models per third-party benchmarks, and add a fallback for when a call doesn't fire or the arguments come back wrong. - **Running a model too large for the machine's RAM and getting silent slowdowns or crashes** — undefined Fix: Match model size to available RAM. A 27B-class model needs real headroom; check Ollama's sizing guidance before deploying to a machine you don't control. - **Forgetting that 'local' only covers the model and tool execution, not every dependency** — undefined Fix: Audit every library and service your tool functions touch. A local model calling a tool that phones out to a cloud API defeats the entire privacy argument for building this way. ## Best practices - Confirm tool-calling support on Ollama's filtered model list before building around a specific model. - Write tool docstrings and type hints as carefully as you would a hand-written JSON schema, since they become one. - Keep every tool function free of outbound network calls if the whole point of the agent is staying local. - Add an explicit 'say you don't know' instruction; a smaller model is more likely to fabricate when it's not sure. - Size the model to the machine's RAM, not the other way around, especially before deploying to shared hardware. - Log tool calls and their arguments during testing so you can see exactly where a smaller model's reliability gaps show up. ## Use cases ### undefined ### undefined ### undefined ## Frequently asked questions ### Does a local AI agent with Ollama ever call out to the internet? Not for inference or tool execution if you write it this way. Ollama serves the model locally and your tool functions run as regular local code, so as long as none of your tools make outbound calls themselves, nothing about a request or its data leaves the machine. ### How much RAM does a tool-calling local model actually need? A 27B-class model like the one in this guide typically needs somewhere around 16-32GB of RAM depending on quantization. Check the specific model's page on Ollama's library for its exact size before committing hardware. ### Which local models actually support tool calling? Filter Ollama's model library by the 'tools' capability rather than guessing from a model's name. As of late 2026 that list includes qwen3.8, Gemma 4, Mistral Small 4, Nemotron 3.5, and gpt-oss-20b, among others. ### Is a local model as reliable at tool calling as a cloud model? Generally not yet. Third-party benchmarks put small local models in the 85-90% range on tool-calling accuracy, below what a current frontier cloud model achieves on the same tasks, so plan for an explicit 'I don't know' fallback. ### Can I mix a local model for most queries with a cloud model for hard ones? Yes, and it's a common pattern when privacy rules allow some data off-device. Route by task difficulty or by whether the specific query touches sensitive data, keeping the sensitive path entirely local. ### Can someone set up a local or hybrid agent like this for my company? Yes. RapidDev builds local and hybrid agents for teams with data-residency or compliance constraints that rule out a pure cloud approach. A free 30-minute consultation is the fastest way to scope what your data actually requires. --- Source: https://www.rapidevelopers.com/build-ai-agent/how-to-build-a-local-ai-agent-with-ollama © RapidDev — https://www.rapidevelopers.com/build-ai-agent/how-to-build-a-local-ai-agent-with-ollama