Skip to main content
RapidDev - Software Development Agency
build-ai-agent

How to Build a Browser Agent

A browser agent drives a real web browser, reading the page and clicking through it the way a person would, instead of calling a site's API. This guide builds one with the browser-use Python library and Claude that visits a list of competitor product pages, extracts price and stock as structured data, and flags anything that changed since the last run. Budget 45-60 minutes and a few cents per site checked.

What you'll learn

  • Drive a real browser with browser-use's Agent class instead of writing brittle CSS-selector scrapers
  • Get structured, typed output from a browser agent with a Pydantic output model instead of parsing free text
  • Run one agent per target site for reliability instead of one agent looping across many tabs
  • Compare each run's results against a stored baseline to flag only what actually changed
  • Harden a browser agent for unattended, scheduled runs instead of one-off interactive sessions
Book a free consultation
4.9Clutch rating ⭐
1,000+Happy partners
20+Countries served
200+Team members
Intermediate9 min read45-60 minPython 3.11+, browser-use, Playwright-managed Chromium, Claude API key, macOS/Linux/WindowsLast updated November 2026RapidDev Engineering Team
TL;DR

A browser agent drives a real web browser, reading the page and clicking through it the way a person would, instead of calling a site's API. This guide builds one with the browser-use Python library and Claude that visits a list of competitor product pages, extracts price and stock as structured data, and flags anything that changed since the last run. Budget 45-60 minutes and a few cents per site checked.

Quick facts about this guide
FactValue
ToolBrowser Use
DifficultyIntermediate
Time required45-60 min
CompatibilityPython 3.11+, browser-use, Playwright-managed Chromium, Claude API key, macOS/Linux/Windows
Last updatedNovember 2026

What you'll build

You'll build Price Watch: an agent that visits a short list of competitor product pages, reads the current price and stock status off each one, and reports which prices moved since the last time it ran. No API, because none of these competitors publish one, which is exactly the situation a browser agent exists for.

A browser agent controls a real browser through Playwright: it sees the rendered page, reads the DOM and accessibility tree, clicks elements and fills forms the way a person would, and extracts data straight out of the page rather than a fixed CSS selector that breaks the next time the site redesigns its layout. That's the whole value proposition over a traditional scraper: the agent re-plans its clicks each time based on what's actually on the page, so a reordered checkout flow or a renamed button doesn't take the whole script down.

This is a different tool than a computer-use agent, which controls an entire operating system by screenshotting the full screen and clicking pixel coordinates. A browser agent only ever operates inside a browser tab and reads structured page content; it's faster, cheaper per action, and the right choice for anything with a URL. Reach for computer use only when there's no DOM at all, like a legacy desktop app. See the computer-use agent guide for that case.

Prerequisites

  • Python 3.11+ and a terminal
  • An Anthropic API key (ANTHROPIC_API_KEY) with available credits
  • pip install browser-use, then playwright install chromium to fetch a managed browser binary
  • A short list of URLs you're allowed to automate against (check each site's terms before scraping it on a schedule)

Step-by-step guide

1

Install browser-use and its managed browser

Install the browser-use package, then run Playwright's install command to download a Chromium binary it controls directly; you don't need a system browser or a visible display for this to work. Export your Anthropic API key as an environment variable, since the LLM wrapper reads it automatically and you never want it hardcoded in a script that might end up in a scheduled job or a git repo.

typescript
1pip install browser-use
2playwright install chromium
3export ANTHROPIC_API_KEY="sk-ant-..."

Expected result: python -c "import browser_use" runs with no errors, and a Chromium binary is present under Playwright's cache directory.

2

Define a structured output schema with Pydantic

Instead of asking the agent to describe what it found in prose and then parsing that yourself, define the exact shape you want with a Pydantic model and pass it as the agent's output schema. The agent fills this schema from what it reads on the page, so your downstream code gets a typed object, not a paragraph you have to regex apart.

typescript
1from pydantic import BaseModel
2
3class PriceCheck(BaseModel):
4 title: str
5 price: float
6 in_stock: bool

Expected result: A Pydantic model ready to pass into browser-use's output_model_schema parameter.

3

Run one agent against a single product page

Create an Agent with a task description that names exactly what to extract, the LLM wrapper pointed at a current Claude model, and the output schema from the previous step. Run it with the single call below and read the structured result off history.final_result() or history.structured_output. Test on one URL before you loop over the full list; browser agents are much easier to debug one page at a time.

typescript
1from browser_use import Agent, ChatAnthropic
2
3llm = ChatAnthropic(model="claude-sonnet-5-5", temperature=0.0)
4
5async def check_one(url: str) -> PriceCheck:
6 agent = Agent(
7 task=f"Go to {url} and read the product title, current price, and "
8 f"whether it's in stock. Report exactly that.",
9 llm=llm,
10 output_model_schema=PriceCheck,
11 )
12 history = await agent.run()
13 return history.structured_output

Expected result: A visible (or headless) Chromium window navigates to the URL, and the function returns a PriceCheck object with real title, price, and stock values.

4

Loop over the full list of competitor URLs

Run one agent per URL in sequence rather than giving a single agent a task like 'check all five of these sites.' A fresh agent per target is more reliable: each run has a small, unambiguous task, and one site failing to load doesn't derail the rest of the batch the way a single long-running agent's context would.

typescript
1URLS = [
2 "https://competitor-a.example.com/product/widget",
3 "https://competitor-b.example.com/items/widget-pro",
4]
5
6async def check_all(urls: list[str]) -> dict[str, PriceCheck]:
7 results = {}
8 for url in urls:
9 results[url] = await check_one(url)
10 return results

Expected result: A dict mapping each URL to its own PriceCheck result after running sequentially.

5

Compare against a stored baseline and flag changes

Load the previous run's results from a JSON file, compare each current price to the stored one, and only report rows where the price actually moved. Without this step you'd have to eyeball every run's full output to notice a single competitor dropped their price 10%, which defeats the point of automating the check.

typescript
1import json
2from pathlib import Path
3
4BASELINE_FILE = Path("price_baseline.json")
5
6def flag_changes(current: dict, baseline: dict) -> list[str]:
7 alerts = []
8 for url, check in current.items():
9 old = baseline.get(url)
10 if old and abs(check.price - old["price"]) / old["price"] > 0.02:
11 alerts.append(f"{url}: {old['price']} -> {check.price}")
12 return alerts

Expected result: A short list of only the URLs whose price moved more than 2% since the last saved baseline.

6

Harden it: retries and headless mode for unattended runs

Check history.is_successful() after each run and retry once before giving up on a URL; browser agents occasionally fail a step on a slow-loading page, and a single retry recovers most of those. Run headless for any scheduled job, since there's no one watching the browser window, and log which sites failed outright so a dead competitor page doesn't silently drop out of your reports.

typescript
1async def check_with_retry(url: str, attempts: int = 2) -> PriceCheck | None:
2 for attempt in range(attempts):
3 agent = Agent(task=f"Go to {url} and read price and stock.", llm=llm,
4 output_model_schema=PriceCheck)
5 history = await agent.run()
6 if history.is_successful():
7 return history.structured_output
8 return None

Expected result: A function that silently retries once on failure and returns None (logged, not raised) only if both attempts fail.

Complete working example

price_watch.py
1"""Price Watch: a browser agent that checks competitor prices and flags changes.
2Run: python price_watch.py
3(pip install browser-use; playwright install chromium; set ANTHROPIC_API_KEY)
4"""
5import asyncio
6import json
7from pathlib import Path
8
9from browser_use import Agent, ChatAnthropic
10from pydantic import BaseModel
11
12BASELINE_FILE = Path("price_baseline.json")
13
14URLS = [
15 "https://competitor-a.example.com/product/widget",
16 "https://competitor-b.example.com/items/widget-pro",
17]
18
19llm = ChatAnthropic(model="claude-sonnet-5-5", temperature=0.0)
20
21
22class PriceCheck(BaseModel):
23 title: str
24 price: float
25 in_stock: bool
26
27
28async def check_with_retry(url: str, attempts: int = 2) -> PriceCheck | None:
29 for _ in range(attempts):
30 agent = Agent(
31 task=(
32 f"Go to {url} and read the product title, current price, and "
33 f"whether it's in stock. Report exactly that, nothing else."
34 ),
35 llm=llm,
36 output_model_schema=PriceCheck,
37 )
38 history = await agent.run()
39 if history.is_successful():
40 return history.structured_output
41 return None
42
43
44async def check_all(urls: list[str]) -> dict[str, PriceCheck]:
45 results: dict[str, PriceCheck] = {}
46 for url in urls:
47 result = await check_with_retry(url)
48 if result is not None:
49 results[url] = result
50 else:
51 print(f"[WARN] Failed to check {url} after retries")
52 return results
53
54
55def load_baseline() -> dict:
56 if BASELINE_FILE.exists():
57 return json.loads(BASELINE_FILE.read_text())
58 return {}
59
60
61def save_baseline(current: dict[str, PriceCheck]) -> None:
62 serializable = {url: check.model_dump() for url, check in current.items()}
63 BASELINE_FILE.write_text(json.dumps(serializable, indent=2))
64
65
66def flag_changes(current: dict[str, PriceCheck], baseline: dict) -> list[str]:
67 alerts = []
68 for url, check in current.items():
69 old = baseline.get(url)
70 if old and old["price"] and abs(check.price - old["price"]) / old["price"] > 0.02:
71 alerts.append(f"{url}: {old['price']} -> {check.price} ({check.title})")
72 return alerts
73
74
75async def main():
76 baseline = load_baseline()
77 current = await check_all(URLS)
78
79 alerts = flag_changes(current, baseline)
80 if alerts:
81 print("Price changes detected:")
82 for alert in alerts:
83 print(f" - {alert}")
84 else:
85 print("No price changes above threshold.")
86
87 save_baseline(current)
88
89
90if __name__ == "__main__":
91 asyncio.run(main())
92

Common mistakes

Why it's a problem: Giving one agent a task like 'check all five of these competitor sites' instead of one agent per URL

How to avoid: Scope each agent run to a single, unambiguous task. One site timing out or redesigning its layout then only costs you that one run, not the whole batch.

Why it's a problem: Parsing the agent's free-text summary with string matching or regex

How to avoid: Pass a Pydantic model as the output schema so the agent returns typed, structured data you can use directly, no parsing required.

Why it's a problem: Running every check without ever looking at history.is_successful()

How to avoid: A browser agent can finish a run without actually completing the task. Check is_successful() and retry before trusting the result.

Why it's a problem: Scraping a site on a schedule without checking its terms of service or robots.txt

How to avoid: Confirm you're allowed to automate against each target before you schedule repeated runs against it, and back off if a site starts blocking the traffic.

Why it's a problem: Running headful browsers on a server with no display for a scheduled job

How to avoid: Run headless for any unattended job. A visible browser window only helps while you're debugging interactively.

Best practices

  • Scope every agent run to one task and one target; chain multiple Agent calls instead of one agent with a long multi-site instruction.
  • Always define an output_model_schema for data you plan to use programmatically. Free text output is for debugging only.
  • Check is_successful() after every run and retry at least once before logging a failure.
  • Store a baseline of prior results and diff against it, so you only act on what actually changed.
  • Cache the Playwright browser binary in your deploy image; don't re-download it on every scheduled run.
  • Keep temperature at 0 for extraction tasks; you want consistent reading of the page, not creative variation.

Still stuck?

Copy one of these prompts to get a personalized, step-by-step explanation.

ChatGPT Prompt

I have this browser-use price-checking agent (below) that visits competitor URLs and extracts price and stock with a Pydantic schema. Add a second check that also extracts the shipping estimate shown on the page, and alert only if the total landed cost (price plus shipping) changes by more than 2%. [paste complete_code]

Build AI Agents Prompt

In Claude Code, open price_watch.py and ask: 'Add a Slack webhook call that posts the alerts list whenever flag_changes returns anything, using an environment variable for the webhook URL.'

Frequently asked questions

What's the difference between a browser agent and a traditional web scraper?

A traditional scraper relies on fixed CSS selectors or XPath that break the moment a site changes its layout. A browser agent reads the rendered page and accessibility tree each run and decides what to click or extract based on what's actually there, so it tolerates redesigns a selector-based script can't.

Can a browser agent fill out forms and click buttons, not just extract data?

Yes. The same Agent class that reads a price off a page can navigate multi-step flows, fill in form fields, and click through checkout or sign-up processes, since it's driving a real browser the same way a person would.

How much does running a browser agent cost per check?

Most of the cost is LLM tokens for reading page content and deciding each action, typically a few cents per page on a mid-tier model for a short, well-scoped task. A headless Chromium instance itself is free to run; you're paying for the model's reasoning over what it sees.

Does a browser agent need a visible browser window to work?

No. Playwright can run Chromium headless, with no display at all, which is what you want for any scheduled or server-side job. Run headful only while you're debugging a task interactively.

What happens if a site blocks automated browsers?

Respect it. Check the site's terms of service and robots.txt before automating against it, and back off immediately if you start seeing CAPTCHAs or blocks; that's the site telling you the traffic isn't welcome, not a technical problem to route around.

Can someone build and host this kind of browser agent for me?

Yes. RapidDev builds production browser agents, including the retry logic, scheduling, and alerting most tutorials skip. A free 30-minute consultation is the fastest way to scope what your version would need.

RapidDev

Talk to an Expert

Our team has built 1,000+ apps. Get personalized help with your project.

Book a free consultation
Matt Graham

Written by

Matt Graham · CEO & Founder, RapidDev

1,000+ client projects delivered. Columbia University & Harvard Business School alumnus, U.S. Navy veteran. About the author →

Learning is great. Shipping is faster with help.

Our engineers have built 1,000+ apps on your platform and the tools around it. If your project needs to be live sooner than your learning curve allows — book a free consultation.

Book a free consultation

We put the rapid in RapidDev

Need a dedicated strategic tech and growth partner? Discover what RapidDev can do for your business! Book a call with our team to schedule a free, no-obligation consultation. We'll discuss your project and provide a custom quote at no cost.