# How to Design AI Agent Architecture - Tool: Any LLM or framework - Difficulty: Advanced - Time required: 50-65 min - Compatibility: Python 3.10+, LangGraph 1.x, any model API key, any OS - Last updated: November 2026 ## TL;DR Designing AI agent architecture means making five decisions before writing code: what the agent is scoped to do, whether one agent or several specialists fit, whether it's triggered synchronously or by an async event, how state survives between steps, and where a human must approve something consequential. This guide walks through all five for a shipping-exception agent, then encodes the result as a LangGraph state machine with a checkpointed pause for carrier updates. Budget 50-65 minutes. ## What you'll build You'll design and build the architecture for a shipping-exception agent: when a carrier reports a package delayed or damaged, it classifies the issue, handles delays on its own, and routes damage cases to a replacement decision that pauses for human approval before anything ships. The agent itself is almost beside the point here. The point is the five decisions you make before any node, tool, or prompt exists, because those decisions are what an architecture diagram is actually describing. The five: (1) scope, precisely what the agent handles and what it explicitly doesn't; (2) one agent or several, does this case genuinely split into specialist roles, or is that an unnecessary layer; (3) trigger model, does work arrive as a request waiting for a reply, or as an event (a carrier webhook) that starts a case which may sit open for days; (4) state, where does the agent's memory of a case live between steps, and does it survive a restart; (5) approval, which decisions are consequential enough that the agent proposes and a human disposes. Get these wrong and no amount of prompt tuning fixes the resulting behavior. This guide answers all five for the shipping-exception case, then encodes the answers directly as a LangGraph StateGraph: a router node that classifies the issue, two specialist nodes for delay and damage, a checkpointer so a case can sit open across multiple carrier updates, and an interrupt() call that pauses the damage path for a human's approval before a replacement ships. ## Before you start - Python 3.10+ and a terminal - pip install langgraph - A model API key (any provider; this architecture doesn't depend on one) - Willingness to answer each of the five decisions in writing before touching code ## Step-by-step guide ### 1. Decision 1: Define the scope precisely Write down exactly what counts as a shipping exception this agent handles (delayed, lost, or damaged packages on orders already shipped) and what it explicitly doesn't (returns, pre-shipment cancellations, billing disputes). A scope stated as 'handles shipping issues' invites the agent to wander into territory you haven't designed guardrails for. **Expected result:** A one-paragraph scope statement specific enough that you could point to a real case and say definitively whether it's in or out. ### 2. Decision 2: One agent or several specialists Delay and damage cases need different judgment, delay is usually a notify-and-wait; damage usually involves money. That split justifies a router plus two specialist nodes instead of one agent holding both sets of instructions. If every case in your scope needed the same handling, a single node would be the right call. Don't split roles that don't actually diverge, see the multi-agent guide for when the opposite mistake, over-splitting, shows up. **Expected result:** A short justification for exactly how many specialist paths exist and what makes each one different from the others. ### 3. Decision 3: Synchronous request or async event A chat agent is request-response: ask, wait, get an answer. This agent is event-driven: a carrier status webhook arrives whenever it arrives, possibly days after the order shipped, and the case needs to exist and remember context across that gap. Design the trigger explicitly, a webhook handler that starts or resumes a case by order_id, rather than assuming every agent looks like a chat loop. ``` # Trigger shape for this agent: an async event, not a chat turn # POST /webhooks/carrier-status { order_id, status, reported_at } # -> starts a new case or resumes an existing one by order_id as thread_id ``` **Expected result:** A documented trigger: what external event starts or resumes work, and what identifies which case it belongs to. ### 4. Decision 4: Where state lives between steps Because this agent can sit open for days between carrier updates, its state (order_id, issue type, carrier status history, resolution) has to survive a process restart, not just live in memory for one request. A checkpointer keyed by order_id is the mechanism; which checkpointer backend (in-memory for development, Postgres for production) is a deployment detail you can decide separately from the shape of the state itself. ``` from typing import TypedDict, Literal class CaseState(TypedDict): order_id: str issue_type: Literal["delay", "damage", "unknown"] carrier_status: str resolution: str ``` **Expected result:** A typed state shape that names everything this case needs to remember, independent of which node is currently running. ### 5. Decision 5: Where a human has to approve Notifying a customer of a delay is reversible and cheap; shipping a replacement item is not. Put the approval gate specifically on the damage path's replacement decision, not on every node, approval fatigue from gating low-stakes steps trains people to rubber-stamp everything, including the step that actually mattered. ``` def handle_damage(state: CaseState) -> CaseState: approved = interrupt({"action": "approve_replacement", "order_id": state["order_id"]}) if not approved: return {"resolution": "Replacement denied; offering store credit instead"} return {"resolution": f"Replacement shipped for {state['order_id']}"} ``` **Expected result:** A node that stops and waits for an external decision at exactly the one point in the flow that spends money. ### 6. Encode all five decisions as a LangGraph state machine Build the graph: a classify_issue node routes to handle_delay or handle_damage based on carrier_status, using add_conditional_edges to make the routing logic an explicit function instead of a buried if/else inside one giant node. Compile with a checkpointer so the graph can pause at handle_damage's interrupt() and resume later with the same thread_id, exactly the state-survival decision 4 called for. ``` from langgraph.graph import StateGraph, START, END from langgraph.checkpoint.memory import InMemorySaver def route_issue(state: CaseState) -> str: return "handle_damage" if state["issue_type"] == "damage" else "handle_delay" graph = StateGraph(CaseState) graph.add_node("classify_issue", classify_issue) graph.add_node("handle_delay", handle_delay) graph.add_node("handle_damage", handle_damage) graph.add_edge(START, "classify_issue") graph.add_conditional_edges("classify_issue", route_issue, {"handle_delay": "handle_delay", "handle_damage": "handle_damage"}) graph.add_edge("handle_delay", END) graph.add_edge("handle_damage", END) app = graph.compile(checkpointer=InMemorySaver()) ``` **Expected result:** A compiled graph where a damage case pauses mid-run, and resuming it with the same thread_id picks up exactly where it stopped. ## Complete code example File: `shipping_exception_architecture.py` ```python """Shipping-exception agent architecture: router + specialists + a checkpointed approval gate, encoding five architecture decisions as an explicit state machine. Run: python shipping_exception_architecture.py (pip install langgraph) """ from typing import Literal, TypedDict from langgraph.checkpoint.memory import InMemorySaver from langgraph.graph import END, START, StateGraph from langgraph.types import Command, interrupt class CaseState(TypedDict): order_id: str issue_type: Literal["delay", "damage", "unknown"] carrier_status: str resolution: str def classify_issue(state: CaseState) -> CaseState: # In production: call a model or rules engine on carrier_status. issue_type = "damage" if "damaged" in state["carrier_status"].lower() else "delay" return {"issue_type": issue_type} def handle_delay(state: CaseState) -> CaseState: return {"resolution": f"Notified customer of delay on {state['order_id']}"} def handle_damage(state: CaseState) -> CaseState: approved = interrupt({"action": "approve_replacement", "order_id": state["order_id"]}) if not approved: return {"resolution": "Replacement denied; offering store credit instead"} return {"resolution": f"Replacement shipped for {state['order_id']}"} def route_issue(state: CaseState) -> str: return "handle_damage" if state["issue_type"] == "damage" else "handle_delay" graph = StateGraph(CaseState) graph.add_node("classify_issue", classify_issue) graph.add_node("handle_delay", handle_delay) graph.add_node("handle_damage", handle_damage) graph.add_edge(START, "classify_issue") graph.add_conditional_edges( "classify_issue", route_issue, {"handle_delay": "handle_delay", "handle_damage": "handle_damage"} ) graph.add_edge("handle_delay", END) graph.add_edge("handle_damage", END) app = graph.compile(checkpointer=InMemorySaver()) if __name__ == "__main__": config = {"configurable": {"thread_id": "case-8821"}} result = app.invoke( { "order_id": "ORD-8821", "carrier_status": "Package reported damaged in transit", "issue_type": "unknown", "resolution": "", }, config=config, ) print(result) # Paused at handle_damage's interrupt(), waiting for approval # Resume the same case once a human approves the replacement: final = app.invoke(Command(resume=True), config=config) print(final["resolution"]) ``` ## Common mistakes - **Drawing an architecture diagram with boxes and arrows but no explicit state store behind any of it** — undefined Fix: Every box that needs to remember something between steps needs a named state shape and a checkpointer. If you can't say where a piece of state lives, it isn't designed yet, it's hoped for. - **Defaulting to one giant agent because splitting into specialists feels like premature complexity** — undefined Fix: Split exactly where the judgment or tools genuinely diverge (delay vs. damage here). Don't split arbitrarily, but don't avoid splitting just to keep the diagram small. - **Treating an event-driven case (a webhook that can arrive days later) like a synchronous chat turn** — undefined Fix: Design the trigger and the identifier that resumes a case explicitly. A chat-shaped request/response loop has no natural place to 'wait three days for a carrier update.' - **Gating every node behind human approval instead of just the consequential one** — undefined Fix: Approval fatigue from gating cheap, reversible steps trains reviewers to rubber-stamp everything, including the replacement decision that actually needed scrutiny. - **Deciding architecture and observability as separate projects** — undefined Fix: Bake logging of which node ran, which decision it made, and why into the graph from the first version. Retrofitting observability after an incident is much harder than designing it in. ## Best practices - Write the five decisions down in plain language before opening an editor; a diagram without written decisions behind it hides assumptions. - Make routing logic an explicit function (add_conditional_edges), never a buried if/else inside a single oversized node. - Choose a checkpointer backend based on whether a case needs to survive a process restart, not just whichever one is easiest to set up. - Put approval gates on specific consequential actions, not on every step, and name exactly what triggers one in the node itself. - Scope the agent in writing, including what it explicitly does not handle, before any node exists. - When a production agent misbehaves, check which of the five decisions was never actually made explicit; that's usually the fastest path to the bug. ## Use cases ### undefined ### undefined ### undefined ## Frequently asked questions ### What does AI agent architecture actually mean, beyond a diagram? It means five concrete decisions: what the agent is scoped to do, whether it's one agent or several, whether it's triggered by a request or an async event, where its state lives between steps, and where a human has to approve. A diagram just draws the result of those decisions. ### What should an AI agent architecture diagram actually show? Nodes with their state shape, the conditions that route between them, where a checkpointer sits, and where an interrupt or approval gate exists, not just boxes labeled 'agent' and 'tools' with arrows between them. ### What are the most common patterns for building AI agents? A single tool-calling agent for well-scoped tasks, a router-plus-specialists pattern like this guide's for tasks that genuinely split, a supervisor-and-subagents pattern for broader delegation, and a retrieval-augmented pattern when the agent's job is mostly about answering from a document set. ### How do I decide between a single agent and a multi-agent architecture? Ask whether the task's phases need genuinely different tools, instructions, or review. If yes, as with this guide's delay-vs-damage split, specialists earn their complexity. If every case gets handled the same way, one well-scoped agent is simpler and cheaper. ### Why does this architecture use a checkpointer instead of just storing state in a database myself? A checkpointer is specifically built for pausing and resuming a graph mid-execution, including at an interrupt() call, with the exact in-progress state intact. You can back it with your own database, but reimplementing the pause/resume semantics yourself is the part worth avoiding. ### Can someone design and build this architecture for me? Yes. RapidDev designs and builds agent architectures on LangGraph and other frameworks, including the state, routing, and approval decisions this guide walks through. A free 30-minute consultation is the fastest way to scope your version. --- Source: https://www.rapidevelopers.com/build-ai-agent/how-to-design-ai-agent-architecture © RapidDev — https://www.rapidevelopers.com/build-ai-agent/how-to-design-ai-agent-architecture