# How to Build a Voice AI Agent - Tool: LiveKit Agents - Difficulty: Intermediate - Time required: 60-75 min - Compatibility: Python 3.10+, livekit-agents 1.8+, a LiveKit Cloud or self-hosted project, Deepgram, Claude, and ElevenLabs API keys - Last updated: November 2026 ## TL;DR A voice AI agent chains speech-to-text, an LLM, and text-to-speech into a pipeline fast enough to feel like a real phone call. This guide builds one with LiveKit Agents, Deepgram for transcription, Claude for reasoning, and ElevenLabs for speech, taking restaurant reservations and handing off anything bigger than a party of 8 to a human. Budget 60-75 minutes and a LiveKit Cloud account to test it live. ## What you'll build You'll build a phone and web voice agent for Bella's Trattoria that takes table reservations: name, date, time, and party size, confirmed out loud before it books anything. Parties larger than 8, and anything that sounds like a complaint, get handed to a human instead of the agent trying to resolve them itself. A voice agent is really a pipeline: speech-to-text turns what the caller says into text, an LLM decides what to say and which tools to call, and text-to-speech turns the reply back into audio, all fast enough that the pause between a caller finishing a sentence and hearing a response doesn't feel like a phone system. LiveKit's AgentSession wires those three stages (plus voice-activity detection, which decides when the caller has actually stopped talking) into one object, and a function tool gives the LLM stage something real to do beyond just talking. This guide uses Deepgram for transcription, Claude for reasoning and tool calls, and ElevenLabs for speech, assembled yourself through LiveKit Agents rather than a packaged SaaS receptionist. That tradeoff buys you a system prompt and tools you fully control, at the cost of wiring the pipeline yourself; if you'd rather not, a packaged AI receptionist product is the faster path for a simple booking flow. ## Before you start - Python 3.10+ and a terminal - A LiveKit Cloud project (or self-hosted LiveKit server) with its URL, API key, and API secret - API keys for Deepgram (speech-to-text), Anthropic (the LLM), and ElevenLabs (speech) - pip install "livekit-agents[deepgram,anthropic,elevenlabs,silero]~=1.8" python-dotenv ## Step-by-step guide ### 1. Install the plugins and set your keys Install livekit-agents with the Deepgram, Anthropic, ElevenLabs, and Silero extras in one line; each extra pulls in the plugin package for that provider. Put every key in a .env.local file rather than hardcoding them, since this script will load them automatically and you never want a voice provider key committed to a repo. ``` pip install "livekit-agents[deepgram,anthropic,elevenlabs,silero]~=1.8" python-dotenv # .env.local LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=... LIVEKIT_API_SECRET=... DEEPGRAM_API_KEY=... ANTHROPIC_API_KEY=... ELEVENLABS_API_KEY=... ``` **Expected result:** pip install finishes cleanly and a .env.local file sits next to your script, untracked by git. ### 2. Write voice-tuned instructions for the agent Write the system prompt the way you'd brief a host answering the phone, not the way you'd brief a chat support agent. Short sentences read naturally out loud; a bulleted list or markdown does not, since text-to-speech reads it as awkward prose. State the escalation boundary here too, the same way the system prompt guide does for a text agent. ``` INSTRUCTIONS = ( "You are the phone reservation agent for Bella's Trattoria. Keep replies " "short, like a real phone call, one or two sentences. Confirm the date, " "time, party size, and name before calling book_reservation. For parties " "over 8 or any complaint, call escalate_to_human instead of handling it " "yourself." ) ``` **Expected result:** A short, speakable instructions string, not a formatted document. ### 3. Define the Agent class and its function tools Subclass Agent with the instructions above, then add book_reservation and escalate_to_human as methods decorated with @function_tool(). The docstring on each becomes the tool description the LLM reads, and the typed parameters become the tool's schema automatically, no separate JSON schema to maintain by hand. ``` from livekit.agents import Agent, RunContext, function_tool class ReservationAgent(Agent): def __init__(self) -> None: super().__init__(instructions=INSTRUCTIONS) @function_tool() async def book_reservation( self, context: RunContext, name: str, date: str, time: str, party_size: int, ) -> dict: """Book a table reservation. Args: name: The name the reservation is under. date: The reservation date, e.g. 2026-11-20. time: The reservation time, e.g. 7:00 PM. party_size: Number of guests. """ return {"status": "confirmed", "name": name, "date": date, "time": time} ``` **Expected result:** An Agent subclass with one working tool, ready to add a second for escalation. ### 4. Wire STT, LLM, TTS, and VAD into an AgentSession Inside the worker's entrypoint, build an AgentSession that strings Silero's voice-activity detector, Deepgram's speech-to-text, Claude as the LLM, and ElevenLabs' speech together. VAD matters more than it looks: it's what decides the caller has actually finished a sentence instead of just pausing mid-thought, which is the difference between a natural conversation and one where the agent keeps interrupting. ``` from livekit.agents import AgentSession from livekit.plugins import anthropic, deepgram, elevenlabs, silero session = AgentSession( vad=silero.VAD.load(), stt=deepgram.STT(model="nova-3"), llm=anthropic.LLM(model="claude-sonnet-5-5"), tts=elevenlabs.TTS(), ) ``` **Expected result:** An AgentSession object, not yet started, with all four pipeline stages configured. ### 5. Start the session against the room and run the worker Call session.start with your Agent instance and the room from the job context, then run the worker with LiveKit's CLI runner. This is the entrypoint LiveKit calls every time a new call or room connects, so one worker process can handle many concurrent conversations, each getting its own session. ``` from livekit import agents from livekit.agents import AgentServer server = AgentServer() @server.rtc_session(agent_name="restaurant-reservations") async def entrypoint(ctx: agents.JobContext) -> None: await session.start(agent=ReservationAgent(), room=ctx.room) if __name__ == "__main__": agents.cli.run_app(server) ``` **Expected result:** Running `python restaurant_voice_agent.py dev` starts a worker that connects to your LiveKit project and waits for a room. ### 6. Add the escalation tool for parties and complaints Give the agent a second tool that does nothing but log the handoff and tell the caller a person will take over. The point of this tool isn't automation, it's the opposite: it's the agent's way of admitting a request is outside what it should decide on its own, the same boundary the system prompt sets in prose. ``` @function_tool() async def escalate_to_human(self, context: RunContext, reason: str) -> dict: """Hand off to a human manager instead of handling the request yourself. Args: reason: Why this needs a human, e.g. 'party of 12' or 'complaint'. """ print(f"[ESCALATION] {reason}") return {"status": "escalated", "reason": reason} ``` **Expected result:** Asking for a table of 15 in a test call makes the agent say it's passing you to a manager instead of trying to book it. ### 7. Test it locally, then connect a real phone number Run the worker in dev mode and use LiveKit's test tools to join the room by voice before wiring up telephony at all; it's much faster to debug a wrong tool call over your laptop's microphone than over a live phone call. Once the conversation flow is solid, point a SIP trunk from a telephony provider at LiveKit's SIP integration and create a dispatch rule that routes the incoming number to this agent. ``` python restaurant_voice_agent.py dev # join the test room from LiveKit's agent playground in your browser ``` **Expected result:** A working voice conversation over your browser microphone that correctly books a normal reservation and escalates an oversized one. ## Complete code example File: `restaurant_voice_agent.py` ```python """Restaurant phone/voice reservation agent on LiveKit Agents. Run: python restaurant_voice_agent.py dev pip install "livekit-agents[deepgram,anthropic,elevenlabs,silero]~=1.8" python-dotenv """ from dotenv import load_dotenv from livekit import agents from livekit.agents import Agent, AgentServer, AgentSession, RunContext, function_tool from livekit.plugins import anthropic, deepgram, elevenlabs, silero load_dotenv(".env.local") INSTRUCTIONS = ( "You are the phone reservation agent for Bella's Trattoria. Keep replies " "short, like a real phone call, one or two sentences. Confirm the date, " "time, party size, and name before calling book_reservation. For parties " "over 8 or any complaint, call escalate_to_human instead of handling it " "yourself." ) class ReservationAgent(Agent): def __init__(self) -> None: super().__init__(instructions=INSTRUCTIONS) @function_tool() async def book_reservation( self, context: RunContext, name: str, date: str, time: str, party_size: int, ) -> dict: """Book a table reservation. Args: name: The name the reservation is under. date: The reservation date, e.g. 2026-11-20. time: The reservation time, e.g. 7:00 PM. party_size: Number of guests. """ # In production this writes to your reservations system/database. return { "status": "confirmed", "name": name, "date": date, "time": time, "party_size": party_size, } @function_tool() async def escalate_to_human(self, context: RunContext, reason: str) -> dict: """Hand off to a human manager instead of handling the request yourself. Args: reason: Why this needs a human, e.g. 'party of 12' or 'complaint about last visit'. """ print(f"[ESCALATION] {reason}") return {"status": "escalated", "reason": reason} server = AgentServer() @server.rtc_session(agent_name="restaurant-reservations") async def entrypoint(ctx: agents.JobContext) -> None: session = AgentSession( vad=silero.VAD.load(), stt=deepgram.STT(model="nova-3"), llm=anthropic.LLM(model="claude-sonnet-5-5"), tts=elevenlabs.TTS(), ) await session.start(agent=ReservationAgent(), room=ctx.room) if __name__ == "__main__": agents.cli.run_app(server) ``` ## Common mistakes - **Writing the system prompt like a chat agent's, with bullet points and markdown** — undefined Fix: Text-to-speech reads markdown as awkward prose. Write short, spoken-style sentences and keep any lists conversational, like 'that's either Tuesday or Wednesday' instead of a bulleted list. - **Skipping voice-activity detection and relying on silence timeouts alone** — undefined Fix: Use a real VAD stage (Silero works well as a default) so the agent knows the caller paused to think versus actually finished talking. Timeout-only turn detection makes an agent interrupt constantly. - **Testing only by typing into a text console instead of speaking into a microphone** — undefined Fix: Latency, interruptions, and transcription errors only show up in a real voice test. Join the room with your actual microphone before calling the flow done. - **Wiring up a real phone number before the conversation logic is solid** — undefined Fix: Debug the agent in LiveKit's browser test room first. It's far faster to fix a wrong tool call over your laptop mic than over a live, billed phone call. - **Giving the agent no way to hand off a request it shouldn't handle** — undefined Fix: Add an explicit escalate_to_human tool, the same way you would in a text agent. 'Just answer well' is not a strategy for an oversized party or an angry caller. ## Best practices - Keep spoken replies to one or two sentences; a long reply feels like being talked at, not talked with. - Use a real VAD stage, not a fixed silence timeout, to decide when the caller has finished speaking. - State the escalation boundary explicitly in the instructions, the same way you would for a text agent's system prompt. - Test the full voice loop with a real microphone before connecting a phone number at all. - Log every booked reservation and every escalation with enough detail to follow up without re-listening to the call. - Tune allow_interruptions for your channel; phone callers talk over the agent far more than chat users type over it. ## Use cases ### undefined ### undefined ### undefined ## Frequently asked questions ### What's the actual latency of a voice agent like this? The budget to beat is the pause between the caller finishing a sentence and hearing a reply, typically 500ms-1s end to end across transcription, the LLM call, and speech synthesis for a well-tuned pipeline. Each added stage, like a slow tool call, adds directly to that number. ### Why use LiveKit Agents instead of a packaged AI receptionist product? A packaged receptionist is faster to launch for a simple booking flow. Building on LiveKit Agents costs you the pipeline wiring yourself, in exchange for full control over the system prompt, the tools, and exactly what gets escalated versus handled. ### How much does it cost to run a voice agent like this per call? Assembling your own stack (speech-to-text, an LLM, and text-to-speech) typically lands around $0.07-$0.20 per minute all in, depending on providers, close to what dedicated voice-AI platforms charge per minute before their own markup. ### Can this same agent handle both a phone call and a web widget? Yes. The Agent and AgentSession code doesn't change; only the room's connection source does, a SIP trunk for phone calls or a browser client for a web widget, since LiveKit treats both as the same kind of room. ### Do I need Deepgram and ElevenLabs specifically, or can I swap providers? Swap freely. LiveKit Agents plugins exist for most major STT and TTS providers; change the stt= and tts= arguments in AgentSession and the rest of the agent's code, including its tools, stays the same. ### Can someone build and deploy this kind of voice agent for me? Yes. RapidDev builds production voice agents on LiveKit and other stacks, including telephony wiring and escalation routing. A free 30-minute consultation is the fastest way to scope your version. --- Source: https://www.rapidevelopers.com/build-ai-agent/how-to-build-a-voice-ai-agent © RapidDev — https://www.rapidevelopers.com/build-ai-agent/how-to-build-a-voice-ai-agent