# How to Build a Computer Use Agent - Tool: Claude computer use - Difficulty: Advanced - Time required: 60-90 min - Compatibility: Python 3.10+, anthropic SDK, a sandboxed Linux container or VM with a virtual display, Claude Opus or Sonnet 5.5 - Last updated: November 2026 ## TL;DR A computer-use agent controls an entire desktop by taking screenshots and issuing clicks and keystrokes at pixel coordinates, which lets it drive software with no API and no web version. This guide builds one with Claude's computer_toolset_20260801 that enters vendor invoices into a 20-year-old desktop accounting app inside a sandboxed container, with a human-approval gate before it saves anything. Budget 60-90 minutes and a sandbox environment to test in. ## What you'll build You'll build an agent that enters vendor invoices into a 20-year-old desktop accounting app: no API, no web version, just a Windows-era GUI with fields, tabs, and a Save button. That's the exact situation computer use exists for, and it's a different problem than a browser agent, which only ever works inside a browser tab reading a DOM. A computer-use agent instead takes a screenshot of the entire screen, sends it to Claude, and gets back pixel-coordinate actions: click here, type this, press that key. There's no DOM, no accessibility tree, no structured page content, just an image and coordinates, which means it works against any application at all, including ones built before APIs existed, at the cost of being slower and far more token-hungry per action than a browser agent reading structured page content. Use computer use only when there's genuinely no DOM, no API, and no other way in; reach for a browser agent for anything with a URL. The twist in this guide, same as the system prompt and SDK guides, is a human-approval gate. The agent can click around and fill in fields freely, but it stops and asks before it presses the key combination that actually commits the invoice. Everything before that point is reversible by just not saving; the save itself is the one action that isn't. ## Before you start - An Anthropic API key with access to a Claude 5.5 model (Opus, Sonnet, or Haiku 5.5) - A disposable sandbox, a Docker container or VM with Xvfb and the target desktop app installed; never point this at your primary machine - pip install anthropic pyautogui mss pillow - Comfort running a headless Linux container with a virtual display ## Step-by-step guide ### 1. Start a virtual display inside the sandbox Install Xvfb inside your container or VM and start it on a virtual display number, then point the DISPLAY environment variable at it so every GUI app, including the legacy accounting software, renders into that virtual framebuffer instead of a physical screen. This is the foundation everything else sits on: Claude never touches your real desktop, only this disposable one. ``` apt-get install -y xvfb x11-utils Xvfb :99 -screen 0 1024x768x24 & export DISPLAY=:99 pip install anthropic pyautogui mss pillow export ANTHROPIC_API_KEY="sk-ant-..." ``` **Expected result:** echo $DISPLAY prints :99, and launching any X11 app inside the container doesn't error with 'cannot open display.' ### 2. Capture real screenshots with mss Write a function that grabs the current virtual display and returns it as base64-encoded PNG bytes, the exact format a screenshot tool_result needs. mss is fast enough to call on every single step of the loop without adding noticeable latency, which matters since a computer-use agent takes a fresh screenshot after nearly every action. ``` import mss, io, base64 from PIL import Image def capture_screenshot_base64() -> str: with mss.mss() as sct: shot = sct.grab(sct.monitors[1]) img = Image.frombytes("RGB", shot.size, shot.bgra, "raw", "BGRX") buf = io.BytesIO() img.save(buf, format="PNG") return base64.standard_b64encode(buf.getvalue()).decode() ``` **Expected result:** Calling the function returns a long base64 string that decodes back into a real PNG of the virtual display's current contents. ### 3. Translate each member action into real input events The computer_toolset_20260801 tool sends you member actions like left_click, type, key, and scroll with coordinates and text, not instructions for how to perform them. Map each one to pyautogui calls: click at a coordinate, type text with a short interval so the legacy app's input handler doesn't drop keystrokes, and send key combinations as hotkeys. ``` import pyautogui def perform_action(name: str, input_data: dict) -> str: if name == "left_click": coord = input_data.get("coordinate") pyautogui.click(x=coord[0], y=coord[1]) if coord else pyautogui.click() return "OK" if name == "type": pyautogui.typewrite(input_data["text"], interval=0.01) return "OK" if name == "key": pyautogui.hotkey(*input_data["text"].lower().split("+")) return "OK" if name == "scroll": amt = input_data.get("scroll_amount", 3) sign = 1 if input_data.get("scroll_direction") == "up" else -1 pyautogui.scroll(sign * amt) return "OK" raise ValueError(f"Unhandled action: {name}") ``` **Expected result:** Each member action the model issues produces a real, visible effect on the virtual display, confirmed by diffing before/after screenshots. ### 4. Build the tool_result response and gate the save action Every tool_result for a computer action needs toolset_name set to 'computer', an image content block for screenshot or zoom and text for everything else, and is_error set when an action fails. Add one more rule on top: if the action is a key press matching your save shortcut, stop and ask a human before it runs. Everything up to that point in the invoice entry is harmless to redo; the save is not. ``` def requires_approval(name: str, input_data: dict) -> bool: return name == "key" and input_data.get("text", "").lower() == "ctrl+s" def handle_action(name: str, input_data: dict): if name in ("screenshot", "zoom"): return [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": capture_screenshot_base64()}}] return perform_action(name, input_data) ``` **Expected result:** A handler that performs every action except the save keystroke immediately, and routes that one keystroke through an approval check first. ### 5. Run the agent loop with the approval gate wired in Call messages.create with the computer_toolset_20260801 tool on every turn, process each tool_use block in order (stopping the batch at the first failure, as the toolset requires), and feed the results back as a tool_result message. When requires_approval fires, print what's about to happen and block on a real approval before letting the keystroke through. ``` def process_tool_calls(response): results, failed = [], False for block in response.content: if block.type != "tool_use" or getattr(block, "toolset_name", None) != "computer": continue result = {"type": "tool_result", "tool_use_id": block.id, "toolset_name": "computer"} if failed: result["content"], result["is_error"] = "Not executed: an earlier action failed.", True elif requires_approval(block.name, block.input): print("[APPROVAL NEEDED] Claude wants to save this invoice (Ctrl+S).") if input("Approve? (y/n): ").strip().lower() == "y": result["content"] = handle_action(block.name, block.input) else: result["content"], result["is_error"] = "Save blocked by operator.", True failed = True else: result["content"] = handle_action(block.name, block.input) results.append(result) return results ``` **Expected result:** The agent fills in invoice fields with no prompts, then stops and prints [APPROVAL NEEDED] the moment it reaches for Ctrl+S. ### 6. Test on one invoice with a before/after screenshot log Run the full loop against a single test invoice and save a screenshot before and after the approval decision, not just the final state. When someone asks six months from now why a particular invoice has the wrong total, that pair of screenshots is the audit trail that answers it, long after chat logs would have rotated out. ``` import time def log_screenshot(label: str): with open(f"audit_{label}_{int(time.time())}.png", "wb") as f: f.write(base64.standard_b64decode(capture_screenshot_base64())) ``` **Expected result:** Two PNG files per invoice run, one from right before the save approval and one from right after, saved alongside your other logs. ### 7. Harden it: network isolation and a hard step cap Run the sandbox with no outbound network access except to Anthropic's API, since a computer-use agent that can see and type into anything on screen is a bad thing to also give internet access to. Cap max_iterations so a confused agent clicking in circles fails loudly after a fixed number of steps instead of running (and billing) indefinitely. ``` MAX_ITERATIONS = 20 # In your container's network policy: allow only api.anthropic.com, deny all other egress. ``` **Expected result:** A sandbox that can reach Claude's API and nothing else, and an agent loop that gives up with a clear error after 20 steps instead of looping silently. ## Complete code example File: `invoice_entry_agent.py` ```python """Computer-use agent: enters a vendor invoice into a legacy desktop app, stopping for human approval before it saves. Run inside a disposable sandbox only. pip install anthropic pyautogui mss pillow; Xvfb running; ANTHROPIC_API_KEY set. """ import base64 import io import time import anthropic import mss import pyautogui from PIL import Image MAX_ITERATIONS = 20 client = anthropic.Anthropic() def capture_screenshot_base64() -> str: with mss.mss() as sct: shot = sct.grab(sct.monitors[1]) img = Image.frombytes("RGB", shot.size, shot.bgra, "raw", "BGRX") buf = io.BytesIO() img.save(buf, format="PNG") return base64.standard_b64encode(buf.getvalue()).decode() def log_screenshot(label: str) -> None: with open(f"audit_{label}_{int(time.time())}.png", "wb") as f: f.write(base64.standard_b64decode(capture_screenshot_base64())) def perform_action(name: str, input_data: dict) -> str: if name == "left_click": coord = input_data.get("coordinate") pyautogui.click(x=coord[0], y=coord[1]) if coord else pyautogui.click() return "OK" if name == "type": pyautogui.typewrite(input_data["text"], interval=0.01) return "OK" if name == "key": pyautogui.hotkey(*input_data["text"].lower().split("+")) return "OK" if name == "scroll": amt = input_data.get("scroll_amount", 3) sign = 1 if input_data.get("scroll_direction") == "up" else -1 pyautogui.scroll(sign * amt) return "OK" raise ValueError(f"Unhandled action: {name}") def handle_action(name: str, input_data: dict): if name in ("screenshot", "zoom"): return [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": capture_screenshot_base64()}}] return perform_action(name, input_data) def requires_approval(name: str, input_data: dict) -> bool: return name == "key" and input_data.get("text", "").lower() == "ctrl+s" def process_tool_calls(response) -> list[dict]: results, failed = [], False for block in response.content: if block.type != "tool_use" or getattr(block, "toolset_name", None) != "computer": continue result = {"type": "tool_result", "tool_use_id": block.id, "toolset_name": "computer"} if failed: result["content"], result["is_error"] = "Not executed: an earlier action failed.", True elif requires_approval(block.name, block.input): log_screenshot("before_save") print("\n[APPROVAL NEEDED] Claude wants to save this invoice (Ctrl+S).") if input("Approve? (y/n): ").strip().lower() == "y": result["content"] = handle_action(block.name, block.input) log_screenshot("after_save") else: result["content"], result["is_error"] = "Save blocked by operator.", True failed = True else: try: result["content"] = handle_action(block.name, block.input) except Exception as e: result["content"], result["is_error"] = f"Error: {e}", True failed = True results.append(result) return results def agent_loop(task: str) -> None: messages = [{"role": "user", "content": task}] for step in range(MAX_ITERATIONS): response = client.messages.create( model="claude-sonnet-5-5", max_tokens=2048, tools=[{"type": "computer_toolset_20260801"}], messages=messages, ) messages.append({"role": "assistant", "content": response.content}) tool_results = process_tool_calls(response) if not tool_results: print("Task complete.") return messages.append({"role": "user", "content": tool_results}) print(f"Stopped after {MAX_ITERATIONS} steps without completing.") if __name__ == "__main__": agent_loop( "Open the Invoices tab in the accounting app, enter a new invoice for " "vendor 'Acme Supplies', amount $482.50, dated today, then save it." ) ``` ## Common mistakes - **Running a computer-use agent against your actual desktop instead of a sandbox** — undefined Fix: Always run inside a disposable container or VM with a virtual display. The agent can click and type anywhere on screen; a sandbox limits the blast radius of a mistake to something you can throw away. - **Reaching for computer use when the target actually has a web UI or API** — undefined Fix: Check for a DOM or API first. A browser agent or a direct API call is faster, cheaper, and more reliable than screenshot-and-click for anything with a URL. - **Trying to infer which click is 'the destructive one' from coordinates alone** — undefined Fix: Gate on an explicit, named action, like a specific save keystroke, instead of guessing intent from pixel positions. Coordinates shift; the save shortcut doesn't. - **Letting the sandbox reach the open internet** — undefined Fix: Restrict egress to only the API endpoint you need. A desktop-controlling agent with open internet access is a bigger risk than the legacy app it's supposed to operate. - **Not capping max_iterations and letting a confused agent loop indefinitely** — undefined Fix: Set a hard step limit. Each step is a full screenshot round-trip; an uncapped loop fails by draining your budget, not by erroring cleanly. ## Best practices - Run every computer-use agent inside a sandbox with no outbound network access beyond the model API. - Match your virtual display resolution to what you tell Claude, so returned coordinates land where you expect. - Gate explicit, named destructive actions (a save shortcut, a submit keystroke) behind human approval rather than trying to infer intent from clicks. - Log a screenshot immediately before and after any approved action, not just the final state, for a real audit trail. - Cap max_iterations and watch for repeated identical screenshots, a sign the agent is stuck clicking the same spot. - Reserve computer use for targets with no DOM or API at all; use a browser agent for anything with a URL. ## Use cases ### undefined ### undefined ### undefined ## Frequently asked questions ### What's the real difference between a computer-use agent and a browser agent? A browser agent reads the DOM and accessibility tree inside a browser tab and acts on structured page content. A computer-use agent has none of that; it only sees a screenshot of the whole screen and clicks pixel coordinates, which is slower and more expensive per action but works against any application, including ones with no DOM or API at all. ### Does computer_toolset_20260801 need a beta header? No. It's a stable tool type as of late 2026, used the same way as any other tool definition in the Messages API, with no beta header required. ### Why gate the save action specifically instead of every click? Nearly every action before the save is harmless to redo: a misplaced click or a typo in a field costs you a retry, not a consequence. The save is the one action that commits the result, so it's the one worth a human looking at first. ### Can I run a computer-use agent without a sandbox, just on my own laptop? You can, but you shouldn't for anything beyond a quick manual test. The agent can click and type anywhere the virtual pointer and keyboard can reach, so a sandboxed, disposable environment is the difference between a mistake you can throw away and one you can't. ### How much does a computer-use agent cost to run compared to a browser agent? More, step for step. Each request adds roughly 4,500 input tokens of overhead for the toolset plus the tokens for the screenshot image itself, versus a browser agent reading much smaller structured text. Budget accordingly for any task that takes many steps. ### Can someone build this kind of desktop-automation agent for me? Yes. RapidDev builds production computer-use and browser agents, including sandboxing and approval gates like the one in this guide. A free 30-minute consultation is the fastest way to scope your specific legacy-app integration. --- Source: https://www.rapidevelopers.com/build-ai-agent/how-to-build-a-computer-use-agent © RapidDev — https://www.rapidevelopers.com/build-ai-agent/how-to-build-a-computer-use-agent