# How to Evaluate an AI Agent - Tool: Any LLM or framework - Difficulty: Intermediate - Time required: 30-45 min - Compatibility: Python 3.10+, claude-agent-sdk, anthropic, an Anthropic API key, any OS - Last updated: November 2026 ## TL;DR AI agent evaluation means running the agent against a fixed set of labeled test cases and scoring both what it did (which tools it called) and what it said, every time the prompt, model, or tools change. This guide builds a small harness for a support-triage agent: a labeled dataset, deterministic tool-call assertions, and an LLM judge for the free-text reply. Budget 30-45 minutes and an Anthropic API key. ## What you'll build You'll build an eval harness for a support-triage agent that reads an incoming ticket and decides which team should handle it: billing, technical, or account-security. The harness runs the agent against 10 labeled tickets, checks two different things for each one, and prints a pass rate your CI can act on. The first thing it checks is deterministic: did the agent call classify_ticket with the category you expect? That's a plain equality check, no model needed to grade it. The second is not deterministic: is the agent's explanation of why it picked that category actually coherent and specific, not generic filler? For that you need an LLM judge, a second model call given the agent's output and a strict rubric, asked to return a pass or fail and a reason. This split matters because conflating the two produces misleading scores. A harness that only checks the free-text reply with an LLM judge can pass an agent that picked the wrong category but explained itself persuasively. A harness that only checks tool calls can miss a reply that's technically on-category but incoherent. Score both, and don't let either one substitute for the other. ## Before you start - Python 3.10+ and a terminal - An Anthropic API key (ANTHROPIC_API_KEY) - pip install claude-agent-sdk anthropic - 10-30 real or realistic support tickets to seed your eval dataset ## Step-by-step guide ### 1. Install the packages and write the labeled dataset Install claude-agent-sdk and anthropic. Write your eval cases as plain Python dicts: the ticket text, the category you expect the agent to pick, and a short note on what a good explanation must mention. Keep this list in version control next to the agent's code; it's the contract that defines correct behavior. ``` pip install claude-agent-sdk anthropic export ANTHROPIC_API_KEY="sk-ant-..." EVAL_CASES = [ { "ticket": "I was charged twice for my subscription this month", "expected_category": "billing", "must_mention": "double charge or duplicate billing", }, { "ticket": "Someone logged into my account from a country I've never visited", "expected_category": "account-security", "must_mention": "unauthorized or suspicious login", }, { "ticket": "The export button just spins forever and never downloads anything", "expected_category": "technical", "must_mention": "export or download failure", }, ] ``` **Expected result:** A Python list of at least 3 (ideally 10+) labeled cases, each with an expected category and a content requirement for the explanation. ### 2. Define the triage tool and run the agent once per case classify_ticket is the only tool the triage agent has; calling it is how you'll check deterministic correctness. run_agent_on_ticket runs one eval case through the Claude Agent SDK and captures both the tool call Claude made and its final text explanation, since you need both to score the case fully. ``` import asyncio from claude_agent_sdk import query, ClaudeAgentOptions, AssistantMessage, ToolUseBlock, ResultMessage SYSTEM_PROMPT = ( "You triage support tickets. Call classify_ticket with 'billing', 'technical', " "or 'account-security', then explain your reasoning in one sentence." ) async def run_agent_on_ticket(ticket: str) -> dict: options = ClaudeAgentOptions(system_prompt=SYSTEM_PROMPT, allowed_tools=[]) called_category = None final_text = "" async for message in query(prompt=ticket, options=options): if isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, ToolUseBlock) and block.name == "classify_ticket": called_category = block.input.get("category") if isinstance(message, ResultMessage) and message.subtype == "success": final_text = message.result return {"category": called_category, "explanation": final_text} ``` **Expected result:** run_agent_on_ticket returns a dict with both a category (possibly None if no tool call happened) and an explanation string. ### 3. Write the deterministic check Comparing the called category to the expected one is a plain equality check, no model call needed. This is intentionally the simplest function in the whole harness; anything you can check without a judge, check without one. ``` def check_category(result: dict, case: dict) -> bool: return result["category"] == case["expected_category"] ``` **Expected result:** check_category returns True or False with zero ambiguity and zero added API cost. ### 4. Write the LLM judge for the open-ended explanation Build a second, separate call to Claude (not the agent itself, a fresh grading call) with a strict rubric: does the explanation mention the required topic, and is it specific rather than generic. Ask for a structured verdict you can parse reliably, not free-form prose. ``` from anthropic import Anthropic judge = Anthropic() def judge_explanation(explanation: str, must_mention: str) -> bool: resp = judge.messages.create( model="claude-sonnet-5-5", max_tokens=10, messages=[{ "role": "user", "content": ( f"Explanation: \"{explanation}\"\n" f"Required topic: {must_mention}\n" "Does the explanation clearly address the required topic, specifically " "(not generically)? Answer with exactly one word: PASS or FAIL." ), }], ) return resp.content[0].text.strip().upper() == "PASS" ``` **Expected result:** judge_explanation returns a clean boolean for each case, parsed from a one-word model response. ### 5. Run the full suite and compute a pass rate Loop over every case, run the agent, score both checks, and print a per-case result plus an overall pass rate. A case only passes if both the deterministic check and the judge agree; either one failing fails the case. ``` async def run_eval_suite() -> float: passed = 0 for case in EVAL_CASES: result = await run_agent_on_ticket(case["ticket"]) category_ok = check_category(result, case) explanation_ok = judge_explanation(result["explanation"], case["must_mention"]) if category_ok else False ok = category_ok and explanation_ok passed += int(ok) print(f"{'PASS' if ok else 'FAIL'}: {case['ticket'][:50]}... (category={result['category']})") rate = passed / len(EVAL_CASES) print(f"\nPass rate: {rate:.0%} ({passed}/{len(EVAL_CASES)})") return rate asyncio.run(run_eval_suite()) ``` **Expected result:** A printed PASS/FAIL line per case and an overall pass rate, e.g. 'Pass rate: 67% (2/3)'. ### 6. Harden it: turn the pass rate into a CI gate A pass rate nobody acts on is just a number. Set a threshold (90% is reasonable once you have 10+ cases) and exit non-zero below it, so a CI pipeline fails the build on a regression instead of letting someone eyeball a printout and move on. Grow the dataset with every bug you find in production; a bug that slipped through once and has no eval case for it will slip through again. ``` import sys async def main(): rate = await run_eval_suite() THRESHOLD = 0.90 if rate < THRESHOLD: print(f"FAILED: pass rate {rate:.0%} is below the {THRESHOLD:.0%} threshold") sys.exit(1) asyncio.run(main()) ``` **Expected result:** The script exits with status 1 and a clear message whenever the suite's pass rate drops below 90%, which a CI step can check directly. ## Complete code example File: `triage_agent_evals.py` ```python """Eval harness for a support-triage agent: a labeled dataset, a deterministic tool-call check, and an LLM judge for the free-text reply. Run: python triage_agent_evals.py (pip install claude-agent-sdk anthropic) """ import asyncio import sys from anthropic import Anthropic from claude_agent_sdk import ( query, ClaudeAgentOptions, AssistantMessage, ToolUseBlock, ResultMessage, ) judge = Anthropic() SYSTEM_PROMPT = ( "You triage support tickets. Call classify_ticket with 'billing', 'technical', " "or 'account-security', then explain your reasoning in one sentence." ) EVAL_CASES = [ { "ticket": "I was charged twice for my subscription this month", "expected_category": "billing", "must_mention": "double charge or duplicate billing", }, { "ticket": "Someone logged into my account from a country I've never visited", "expected_category": "account-security", "must_mention": "unauthorized or suspicious login", }, { "ticket": "The export button just spins forever and never downloads anything", "expected_category": "technical", "must_mention": "export or download failure", }, ] async def run_agent_on_ticket(ticket: str) -> dict: options = ClaudeAgentOptions(system_prompt=SYSTEM_PROMPT, allowed_tools=[]) called_category = None final_text = "" async for message in query(prompt=ticket, options=options): if isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, ToolUseBlock) and block.name == "classify_ticket": called_category = block.input.get("category") if isinstance(message, ResultMessage) and message.subtype == "success": final_text = message.result return {"category": called_category, "explanation": final_text} def check_category(result: dict, case: dict) -> bool: return result["category"] == case["expected_category"] def judge_explanation(explanation: str, must_mention: str) -> bool: resp = judge.messages.create( model="claude-sonnet-5-5", max_tokens=10, messages=[{ "role": "user", "content": ( f'Explanation: "{explanation}"\n' f"Required topic: {must_mention}\n" "Does the explanation clearly address the required topic, specifically " "(not generically)? Answer with exactly one word: PASS or FAIL." ), }], ) return resp.content[0].text.strip().upper() == "PASS" async def run_eval_suite() -> float: passed = 0 for case in EVAL_CASES: result = await run_agent_on_ticket(case["ticket"]) category_ok = check_category(result, case) explanation_ok = ( judge_explanation(result["explanation"], case["must_mention"]) if category_ok else False ) ok = category_ok and explanation_ok passed += int(ok) print(f"{'PASS' if ok else 'FAIL'}: {case['ticket'][:50]}... (category={result['category']})") rate = passed / len(EVAL_CASES) print(f"\nPass rate: {rate:.0%} ({passed}/{len(EVAL_CASES)})") return rate async def main(): rate = await run_eval_suite() threshold = 0.90 if rate < threshold: print(f"FAILED: pass rate {rate:.0%} is below the {threshold:.0%} threshold") sys.exit(1) if __name__ == "__main__": asyncio.run(main()) ``` ## Common mistakes - **Scoring everything with an LLM judge, including facts that have a single correct answer** — undefined Fix: Check deterministic facts, like which tool was called or which category was chosen, with plain equality. Save the judge for genuinely open-ended quality like explanation coherence. - **Letting the judge write free-form prose instead of a forced single-word verdict** — undefined Fix: A free-form judge response is harder to parse reliably and costs more tokens per case. Force PASS/FAIL or a tight enum and cap max_tokens accordingly. - **Running the eval suite once at launch and never again as the system prompt changes** — undefined Fix: A prompt or model change can shift behavior on cases you never retested. Run the full suite on every meaningful change, not just new ones. - **Keeping an eval dataset too small to catch real regressions (3-5 cases forever)** — undefined Fix: Grow the dataset every time a production bug slips through. A bug with no corresponding eval case is a bug that can recur silently. - **Printing a pass rate but never gating a build on it** — undefined Fix: Set an explicit threshold and exit non-zero below it, so CI actually blocks a regression instead of a human having to notice a printed percentage. ## Best practices - Split scoring into deterministic checks (tool calls, structured outputs) and judged checks (open-ended text quality); never substitute one for the other. - Force the LLM judge into a tight, parseable output format (PASS/FAIL, a 1-5 score) instead of free prose. - Keep the eval dataset in version control next to the agent's code, as the behavioral contract it actually is. - Grow the dataset from real production failures, not just hypothetical cases you imagined upfront. - Gate CI on a pass-rate threshold so a regression fails the build instead of waiting for someone to notice. - Re-run the full suite on every system-prompt, tool, or model change, not only on changes you assume are risky. ## Use cases ### undefined ### undefined ### undefined ## Frequently asked questions ### Do I need a dedicated eval framework like Promptfoo, or is a plain Python script enough? A plain script is enough to start, and it's what this guide builds. Reach for a dedicated framework once you need features like parallel test execution across many models, built-in dashboards, or sharing eval configs across teams; the scoring logic itself is the same either way. ### Why use a second model call as a judge instead of just reading the transcripts myself? Reading transcripts works at 5 cases and stops scaling at 50. An LLM judge with a tight rubric lets the suite run unattended in CI and catches regressions on every change, not just the ones someone remembered to spot-check. ### Can the judge grade the same thing the agent itself is trying to do, creating a blind spot? It's a real risk; a judge built on the same model family can share the same blind spots as the agent. Mitigate it by keeping the judge's rubric narrow and mechanical (does it mention X, is it under Y words) rather than asking it to evaluate overall quality broadly. ### How many eval cases do I need before the pass rate means anything? 10-20 well-chosen cases covering your known edge cases is a reasonable starting floor; below that, a single flaky case can swing the pass rate by double digits. Grow toward 50-100 as the agent matures and more production failures get added as cases. ### What's the token cost of running an eval suite like this? Each case costs one agent run plus one short judge call. With Claude Sonnet 5.5 at a few thousand tokens per agent run and a 10-token judge response, a 20-case suite typically runs well under a dollar, cheap enough to run on every pull request. ### Can someone build an eval harness for my agent? Yes. RapidDev builds eval harnesses and CI gates for production agents across the major SDKs, including the deterministic/judge split most teams skip. A free 30-minute consultation is the fastest way to scope your version. --- Source: https://www.rapidevelopers.com/build-ai-agent/how-to-evaluate-an-ai-agent © RapidDev — https://www.rapidevelopers.com/build-ai-agent/how-to-evaluate-an-ai-agent