A voice AI agent chains speech-to-text, an LLM, and text-to-speech into a pipeline fast enough to feel like a real phone call. This guide builds one with LiveKit Agents, Deepgram for transcription, Claude for reasoning, and ElevenLabs for speech, taking restaurant reservations and handing off anything bigger than a party of 8 to a human. Budget 60-75 minutes and a LiveKit Cloud account to test it live.
| Fact | Value |
|---|---|
| Tool | LiveKit Agents |
| Difficulty | Intermediate |
| Time required | 60-75 min |
| Compatibility | Python 3.10+, livekit-agents 1.8+, a LiveKit Cloud or self-hosted project, Deepgram, Claude, and ElevenLabs API keys |
| Last updated | November 2026 |
What you'll build
You'll build a phone and web voice agent for Bella's Trattoria that takes table reservations: name, date, time, and party size, confirmed out loud before it books anything. Parties larger than 8, and anything that sounds like a complaint, get handed to a human instead of the agent trying to resolve them itself.
A voice agent is really a pipeline: speech-to-text turns what the caller says into text, an LLM decides what to say and which tools to call, and text-to-speech turns the reply back into audio, all fast enough that the pause between a caller finishing a sentence and hearing a response doesn't feel like a phone system. LiveKit's AgentSession wires those three stages (plus voice-activity detection, which decides when the caller has actually stopped talking) into one object, and a function tool gives the LLM stage something real to do beyond just talking.
This guide uses Deepgram for transcription, Claude for reasoning and tool calls, and ElevenLabs for speech, assembled yourself through LiveKit Agents rather than a packaged SaaS receptionist. That tradeoff buys you a system prompt and tools you fully control, at the cost of wiring the pipeline yourself; if you'd rather not, a packaged AI receptionist product is the faster path for a simple booking flow.
Prerequisites
- Python 3.10+ and a terminal
- A LiveKit Cloud project (or self-hosted LiveKit server) with its URL, API key, and API secret
- API keys for Deepgram (speech-to-text), Anthropic (the LLM), and ElevenLabs (speech)
- pip install "livekit-agents[deepgram,anthropic,elevenlabs,silero]~=1.8" python-dotenv
Step-by-step guide
Install the plugins and set your keys
Install livekit-agents with the Deepgram, Anthropic, ElevenLabs, and Silero extras in one line; each extra pulls in the plugin package for that provider. Put every key in a .env.local file rather than hardcoding them, since this script will load them automatically and you never want a voice provider key committed to a repo.
1pip install "livekit-agents[deepgram,anthropic,elevenlabs,silero]~=1.8" python-dotenv2# .env.local3LIVEKIT_URL=wss://your-project.livekit.cloud4LIVEKIT_API_KEY=...5LIVEKIT_API_SECRET=...6DEEPGRAM_API_KEY=...7ANTHROPIC_API_KEY=...8ELEVENLABS_API_KEY=...Expected result: pip install finishes cleanly and a .env.local file sits next to your script, untracked by git.
Write voice-tuned instructions for the agent
Write the system prompt the way you'd brief a host answering the phone, not the way you'd brief a chat support agent. Short sentences read naturally out loud; a bulleted list or markdown does not, since text-to-speech reads it as awkward prose. State the escalation boundary here too, the same way the system prompt guide does for a text agent.
1INSTRUCTIONS = (2 "You are the phone reservation agent for Bella's Trattoria. Keep replies "3 "short, like a real phone call, one or two sentences. Confirm the date, "4 "time, party size, and name before calling book_reservation. For parties "5 "over 8 or any complaint, call escalate_to_human instead of handling it "6 "yourself."7)Expected result: A short, speakable instructions string, not a formatted document.
Define the Agent class and its function tools
Subclass Agent with the instructions above, then add book_reservation and escalate_to_human as methods decorated with @function_tool(). The docstring on each becomes the tool description the LLM reads, and the typed parameters become the tool's schema automatically, no separate JSON schema to maintain by hand.
1from livekit.agents import Agent, RunContext, function_tool23class ReservationAgent(Agent):4 def __init__(self) -> None:5 super().__init__(instructions=INSTRUCTIONS)67 @function_tool()8 async def book_reservation(9 self, context: RunContext, name: str, date: str, time: str, party_size: int,10 ) -> dict:11 """Book a table reservation.1213 Args:14 name: The name the reservation is under.15 date: The reservation date, e.g. 2026-11-20.16 time: The reservation time, e.g. 7:00 PM.17 party_size: Number of guests.18 """19 return {"status": "confirmed", "name": name, "date": date, "time": time}Expected result: An Agent subclass with one working tool, ready to add a second for escalation.
Wire STT, LLM, TTS, and VAD into an AgentSession
Inside the worker's entrypoint, build an AgentSession that strings Silero's voice-activity detector, Deepgram's speech-to-text, Claude as the LLM, and ElevenLabs' speech together. VAD matters more than it looks: it's what decides the caller has actually finished a sentence instead of just pausing mid-thought, which is the difference between a natural conversation and one where the agent keeps interrupting.
1from livekit.agents import AgentSession2from livekit.plugins import anthropic, deepgram, elevenlabs, silero34session = AgentSession(5 vad=silero.VAD.load(),6 stt=deepgram.STT(model="nova-3"),7 llm=anthropic.LLM(model="claude-sonnet-5-5"),8 tts=elevenlabs.TTS(),9)Expected result: An AgentSession object, not yet started, with all four pipeline stages configured.
Start the session against the room and run the worker
Call session.start with your Agent instance and the room from the job context, then run the worker with LiveKit's CLI runner. This is the entrypoint LiveKit calls every time a new call or room connects, so one worker process can handle many concurrent conversations, each getting its own session.
1from livekit import agents2from livekit.agents import AgentServer34server = AgentServer()56@server.rtc_session(agent_name="restaurant-reservations")7async def entrypoint(ctx: agents.JobContext) -> None:8 await session.start(agent=ReservationAgent(), room=ctx.room)910if __name__ == "__main__":11 agents.cli.run_app(server)Expected result: Running `python restaurant_voice_agent.py dev` starts a worker that connects to your LiveKit project and waits for a room.
Add the escalation tool for parties and complaints
Give the agent a second tool that does nothing but log the handoff and tell the caller a person will take over. The point of this tool isn't automation, it's the opposite: it's the agent's way of admitting a request is outside what it should decide on its own, the same boundary the system prompt sets in prose.
1 @function_tool()2 async def escalate_to_human(self, context: RunContext, reason: str) -> dict:3 """Hand off to a human manager instead of handling the request yourself.45 Args:6 reason: Why this needs a human, e.g. 'party of 12' or 'complaint'.7 """8 print(f"[ESCALATION] {reason}")9 return {"status": "escalated", "reason": reason}Expected result: Asking for a table of 15 in a test call makes the agent say it's passing you to a manager instead of trying to book it.
Test it locally, then connect a real phone number
Run the worker in dev mode and use LiveKit's test tools to join the room by voice before wiring up telephony at all; it's much faster to debug a wrong tool call over your laptop's microphone than over a live phone call. Once the conversation flow is solid, point a SIP trunk from a telephony provider at LiveKit's SIP integration and create a dispatch rule that routes the incoming number to this agent.
1python restaurant_voice_agent.py dev2# join the test room from LiveKit's agent playground in your browserExpected result: A working voice conversation over your browser microphone that correctly books a normal reservation and escalates an oversized one.
Complete working example
1"""Restaurant phone/voice reservation agent on LiveKit Agents.2Run: python restaurant_voice_agent.py dev3pip install "livekit-agents[deepgram,anthropic,elevenlabs,silero]~=1.8" python-dotenv4"""5from dotenv import load_dotenv6from livekit import agents7from livekit.agents import Agent, AgentServer, AgentSession, RunContext, function_tool8from livekit.plugins import anthropic, deepgram, elevenlabs, silero910load_dotenv(".env.local")1112INSTRUCTIONS = (13 "You are the phone reservation agent for Bella's Trattoria. Keep replies "14 "short, like a real phone call, one or two sentences. Confirm the date, "15 "time, party size, and name before calling book_reservation. For parties "16 "over 8 or any complaint, call escalate_to_human instead of handling it "17 "yourself."18)192021class ReservationAgent(Agent):22 def __init__(self) -> None:23 super().__init__(instructions=INSTRUCTIONS)2425 @function_tool()26 async def book_reservation(27 self,28 context: RunContext,29 name: str,30 date: str,31 time: str,32 party_size: int,33 ) -> dict:34 """Book a table reservation.3536 Args:37 name: The name the reservation is under.38 date: The reservation date, e.g. 2026-11-20.39 time: The reservation time, e.g. 7:00 PM.40 party_size: Number of guests.41 """42 # In production this writes to your reservations system/database.43 return {44 "status": "confirmed",45 "name": name,46 "date": date,47 "time": time,48 "party_size": party_size,49 }5051 @function_tool()52 async def escalate_to_human(self, context: RunContext, reason: str) -> dict:53 """Hand off to a human manager instead of handling the request yourself.5455 Args:56 reason: Why this needs a human, e.g. 'party of 12' or 'complaint57 about last visit'.58 """59 print(f"[ESCALATION] {reason}")60 return {"status": "escalated", "reason": reason}616263server = AgentServer()646566@server.rtc_session(agent_name="restaurant-reservations")67async def entrypoint(ctx: agents.JobContext) -> None:68 session = AgentSession(69 vad=silero.VAD.load(),70 stt=deepgram.STT(model="nova-3"),71 llm=anthropic.LLM(model="claude-sonnet-5-5"),72 tts=elevenlabs.TTS(),73 )74 await session.start(agent=ReservationAgent(), room=ctx.room)757677if __name__ == "__main__":78 agents.cli.run_app(server)79Common mistakes
Why it's a problem: Writing the system prompt like a chat agent's, with bullet points and markdown
How to avoid: Text-to-speech reads markdown as awkward prose. Write short, spoken-style sentences and keep any lists conversational, like 'that's either Tuesday or Wednesday' instead of a bulleted list.
Why it's a problem: Skipping voice-activity detection and relying on silence timeouts alone
How to avoid: Use a real VAD stage (Silero works well as a default) so the agent knows the caller paused to think versus actually finished talking. Timeout-only turn detection makes an agent interrupt constantly.
Why it's a problem: Testing only by typing into a text console instead of speaking into a microphone
How to avoid: Latency, interruptions, and transcription errors only show up in a real voice test. Join the room with your actual microphone before calling the flow done.
Why it's a problem: Wiring up a real phone number before the conversation logic is solid
How to avoid: Debug the agent in LiveKit's browser test room first. It's far faster to fix a wrong tool call over your laptop mic than over a live, billed phone call.
Why it's a problem: Giving the agent no way to hand off a request it shouldn't handle
How to avoid: Add an explicit escalate_to_human tool, the same way you would in a text agent. 'Just answer well' is not a strategy for an oversized party or an angry caller.
Best practices
- Keep spoken replies to one or two sentences; a long reply feels like being talked at, not talked with.
- Use a real VAD stage, not a fixed silence timeout, to decide when the caller has finished speaking.
- State the escalation boundary explicitly in the instructions, the same way you would for a text agent's system prompt.
- Test the full voice loop with a real microphone before connecting a phone number at all.
- Log every booked reservation and every escalation with enough detail to follow up without re-listening to the call.
- Tune allow_interruptions for your channel; phone callers talk over the agent far more than chat users type over it.
Still stuck?
Copy one of these prompts to get a personalized, step-by-step explanation.
I have this LiveKit Agents voice reservation agent (below) with book_reservation and escalate_to_human tools. Add a check_availability tool that the agent must call before offering a time, and have it suggest the two nearest open slots if the requested time is full. [paste complete_code]
In Claude Code, open restaurant_voice_agent.py and ask: 'Add a cancel_reservation function tool with the same pattern as book_reservation, and update the instructions to mention when to use it.'
Frequently asked questions
What's the actual latency of a voice agent like this?
The budget to beat is the pause between the caller finishing a sentence and hearing a reply, typically 500ms-1s end to end across transcription, the LLM call, and speech synthesis for a well-tuned pipeline. Each added stage, like a slow tool call, adds directly to that number.
Why use LiveKit Agents instead of a packaged AI receptionist product?
A packaged receptionist is faster to launch for a simple booking flow. Building on LiveKit Agents costs you the pipeline wiring yourself, in exchange for full control over the system prompt, the tools, and exactly what gets escalated versus handled.
How much does it cost to run a voice agent like this per call?
Assembling your own stack (speech-to-text, an LLM, and text-to-speech) typically lands around $0.07-$0.20 per minute all in, depending on providers, close to what dedicated voice-AI platforms charge per minute before their own markup.
Can this same agent handle both a phone call and a web widget?
Yes. The Agent and AgentSession code doesn't change; only the room's connection source does, a SIP trunk for phone calls or a browser client for a web widget, since LiveKit treats both as the same kind of room.
Do I need Deepgram and ElevenLabs specifically, or can I swap providers?
Swap freely. LiveKit Agents plugins exist for most major STT and TTS providers; change the stt= and tts= arguments in AgentSession and the rest of the agent's code, including its tools, stays the same.
Can someone build and deploy this kind of voice agent for me?
Yes. RapidDev builds production voice agents on LiveKit and other stacks, including telephony wiring and escalation routing. A free 30-minute consultation is the fastest way to scope your version.
Talk to an Expert
Our team has built 1,000+ apps. Get personalized help with your project.
Book a free consultation