# How to Build Web scraping API with V0

- Tool: v0
- Difficulty: Advanced
- Compatibility: V0 Premium (Supabase integration)
- Last updated: April 2026

## TL;DR

Build a web scraping service with V0 featuring a REST API that accepts URLs and CSS selectors, extracts structured data with Cheerio (not Puppeteer — it runs on Vercel serverless), API key authentication, result caching in Supabase, and a job management dashboard. You'll handle Vercel's timeout limits with maxDuration and provide a ScrapingBee fallback for JavaScript-rendered pages — all in about 2-4 hours.

## Before you start

- A V0 account (Premium recommended for the project complexity)
- A Supabase project (free tier works — connect via V0's Connect panel)
- Optional: a ScrapingBee account for JavaScript-rendered pages (free tier: 1,000 requests)
- No other API keys or services needed

## Step-by-step guide

### 1. Set up the scrape jobs, results, and API keys schema

Open V0 and create a new project. Use the Connect panel to add Supabase. Create tables for scrape job definitions, extraction results, and API key management.

```
CREATE TABLE api_keys (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  owner_id uuid NOT NULL,
  key_hash text UNIQUE NOT NULL,
  key_prefix text NOT NULL,
  name text NOT NULL,
  is_active boolean DEFAULT true,
  created_at timestamptz DEFAULT now()
);

CREATE TABLE scrape_jobs (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  owner_id uuid NOT NULL,
  target_url text NOT NULL,
  selector_config jsonb NOT NULL DEFAULT '{}',
  schedule_cron text,
  status text DEFAULT 'pending'
    CHECK (status IN ('pending','running','completed','failed')),
  last_run_at timestamptz,
  created_at timestamptz DEFAULT now()
);

CREATE TABLE scrape_results (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  job_id uuid REFERENCES scrape_jobs(id) ON DELETE CASCADE,
  data jsonb NOT NULL,
  html_snapshot text,
  status_code int,
  duration_ms int,
  scraped_at timestamptz DEFAULT now()
);

CREATE INDEX idx_results_job_id ON scrape_results(job_id);
CREATE INDEX idx_results_scraped_at ON scrape_results(scraped_at DESC);

-- RLS: users only see their own data
ALTER TABLE scrape_jobs ENABLE ROW LEVEL SECURITY;
CREATE POLICY "Users manage own jobs" ON scrape_jobs
  FOR ALL USING (owner_id = auth.uid());

ALTER TABLE scrape_results ENABLE ROW LEVEL SECURITY;
CREATE POLICY "Users see own results" ON scrape_results
  FOR SELECT USING (
    job_id IN (SELECT id FROM scrape_jobs WHERE owner_id = auth.uid())
  );
```

> Pro tip: The selector_config is a JSON map like {"title": "h1", "price": ".price-tag", "image": "img.hero@src"}. The @src suffix tells the scraper to extract an attribute instead of text content.

**Expected result:** Three tables with RLS policies. API keys use hashed storage (only the prefix is visible). Scrape results are indexed by job and timestamp for fast queries.

### 2. Build the scraping API with Cheerio and API key auth

Create the public API endpoint that authenticates via X-API-Key header, fetches the target URL, parses it with Cheerio using the provided CSS selectors, and returns structured data.

```
import { NextRequest, NextResponse } from 'next/server'
import { createClient } from '@supabase/supabase-js'
import * as cheerio from 'cheerio'
import crypto from 'crypto'

export const maxDuration = 60

const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY!
)

export async function POST(req: NextRequest) {
  const apiKey = req.headers.get('x-api-key')
  if (!apiKey) {
    return NextResponse.json({ error: 'Missing API key' }, { status: 401 })
  }

  const keyHash = crypto.createHash('sha256').update(apiKey).digest('hex')
  const { data: keyRecord } = await supabase
    .from('api_keys')
    .select('owner_id, is_active')
    .eq('key_hash', keyHash)
    .single()

  if (!keyRecord?.is_active) {
    return NextResponse.json({ error: 'Invalid API key' }, { status: 401 })
  }

  const { url, selectors } = await req.json()
  const start = Date.now()

  const res = await fetch(url, {
    headers: { 'User-Agent': 'RapidScraper/1.0' },
    signal: AbortSignal.timeout(15000),
  })

  const html = await res.text()
  const $ = cheerio.load(html)
  const data: Record<string, string | string[]> = {}

  for (const [key, selector] of Object.entries(selectors as Record<string, string>)) {
    const [sel, attr] = selector.split('@')
    const elements = $(sel)
    if (elements.length > 1) {
      data[key] = elements.map((_, el) =>
        attr ? $(el).attr(attr) ?? '' : $(el).text().trim()
      ).get()
    } else {
      data[key] = attr
        ? elements.attr(attr) ?? ''
        : elements.text().trim()
    }
  }

  await supabase.from('scrape_results').insert({
    job_id: null,
    data,
    status_code: res.status,
    duration_ms: Date.now() - start,
  })

  return NextResponse.json({ data, status_code: res.status, duration_ms: Date.now() - start })
}
```

> Pro tip: Set maxDuration = 60 at the top of the route file. This extends the Vercel serverless timeout from the default 10 seconds (Hobby) to 60 seconds (Pro plan). Without it, slow websites will timeout.

**Expected result:** POST to /api/scrape with an X-API-Key header and JSON body containing url and selectors returns the extracted data, HTTP status code, and duration.

### 3. Build the API key generation system

Create a secure API key generation flow. Keys are generated as random strings, the full key is shown once to the user, and only the SHA-256 hash is stored in the database.

```
'use server'

import { createClient } from '@/lib/supabase/server'
import crypto from 'crypto'

export async function generateApiKey(name: string) {
  const supabase = await createClient()
  const user = (await supabase.auth.getUser()).data.user
  if (!user) return { error: 'Unauthorized' }

  const rawKey = `rsk_${crypto.randomBytes(32).toString('hex')}`
  const keyHash = crypto.createHash('sha256').update(rawKey).digest('hex')
  const keyPrefix = rawKey.slice(0, 8)

  const { error } = await supabase.from('api_keys').insert({
    owner_id: user.id,
    key_hash: keyHash,
    key_prefix: keyPrefix,
    name,
  })

  if (error) return { error: error.message }

  return { key: rawKey, prefix: keyPrefix }
}

export async function revokeApiKey(keyId: string) {
  const supabase = await createClient()
  await supabase
    .from('api_keys')
    .update({ is_active: false })
    .eq('id', keyId)
}
```

> Pro tip: The raw key is returned only once during generation. After that, only the prefix (rsk_xxxx) is visible in the dashboard. This follows the same pattern as Stripe and GitHub API keys.

**Expected result:** Generating an API key shows the full key in a Dialog with a copy button. The key is hashed before storage — the full key is never retrievable again.

### 4. Build the job management dashboard

Create a dashboard where users manage scrape jobs, view result history, and configure CSS selectors for each target URL.

```
// Paste this prompt into V0's AI chat:
// Build a scraping job dashboard at app/dashboard/page.tsx with:
// 1. Server Component fetching all scrape_jobs for the current user
// 2. shadcn/ui Table with columns: Target URL (truncated), Status Badge (pending=outline, running=secondary, completed=default, failed=destructive), Last Run, Results Count, Actions
// 3. Dialog for creating new jobs: Input for target URL, Textarea for JSON selector config with example placeholder ({"title": "h1", "price": ".price"}), optional Input for cron schedule
// 4. Per-job detail page at app/dashboard/jobs/[id]/page.tsx showing: result history Table, latest extracted data in a pre/code block, re-run Button
// 5. API Keys section with Table showing key prefix, name, status Badge, and Revoke Button with AlertDialog
// 6. Generate API Key Button opening Dialog with Input for key name, showing the generated key once with copy-to-clipboard
// 7. Summary Cards: total jobs, successful scrapes today, failed scrapes, active API keys
```

**Expected result:** A dashboard with job Table, status Badges, API key management, and per-job result history with extracted data preview.

### 5. Add result caching and ScrapingBee fallback

Add result caching to avoid re-scraping the same URL within a TTL period, and integrate ScrapingBee as a fallback for JavaScript-rendered pages that Cheerio cannot parse.

```
// Paste this prompt into V0's AI chat:
// Enhance the scraping API with:
// 1. Result caching: before scraping, check scrape_results for a result with the same URL scraped within the last hour. If found, return the cached data with a 'cached: true' flag.
// 2. ScrapingBee fallback: add a 'render_js' boolean parameter to the API. When true, call ScrapingBee's API (api.scrapingbee.com/api/v1?api_key=KEY&url=URL&render_js=true) instead of direct fetch. This handles JavaScript-rendered SPAs.
// 3. Set SCRAPINGBEE_API_KEY in V0's Vars tab (server-only).
// 4. Add a 'Test Scrape' Button on the job form that runs a single scrape and shows the result in a preview Card before saving the job.
// 5. Error handling: catch ETIMEDOUT, ECONNREFUSED, and HTTP 4xx/5xx with descriptive error messages.
// Update the scraping API route to support both modes.
```

**Expected result:** The API checks for cached results before scraping. JavaScript-heavy pages can be scraped via ScrapingBee fallback. A test button lets users preview extraction results before saving jobs.

## Complete code example

File: `app/api/scrape/route.ts`

```typescript
import { NextRequest, NextResponse } from 'next/server'
import { createClient } from '@supabase/supabase-js'
import * as cheerio from 'cheerio'
import crypto from 'crypto'

export const maxDuration = 60

const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY!
)

export async function POST(req: NextRequest) {
  const apiKey = req.headers.get('x-api-key')
  if (!apiKey) {
    return NextResponse.json({ error: 'Missing API key' }, { status: 401 })
  }

  const keyHash = crypto.createHash('sha256').update(apiKey).digest('hex')
  const { data: keyRecord } = await supabase
    .from('api_keys')
    .select('owner_id, is_active')
    .eq('key_hash', keyHash)
    .single()

  if (!keyRecord?.is_active) {
    return NextResponse.json({ error: 'Invalid API key' }, { status: 401 })
  }

  const { url, selectors } = await req.json()
  const start = Date.now()

  const res = await fetch(url, {
    headers: { 'User-Agent': 'RapidScraper/1.0' },
    signal: AbortSignal.timeout(15000),
  })

  const html = await res.text()
  const $ = cheerio.load(html)
  const data: Record<string, string | string[]> = {}

  for (const [key, selector] of Object.entries(
    selectors as Record<string, string>
  )) {
    const [sel, attr] = selector.split('@')
    const elements = $(sel)
    if (elements.length > 1) {
      data[key] = elements
        .map((_, el) =>
          attr ? $(el).attr(attr) ?? '' : $(el).text().trim()
        )
        .get()
    } else {
      data[key] = attr
        ? elements.attr(attr) ?? ''
        : elements.text().trim()
    }
  }

  return NextResponse.json({
    data,
    status_code: res.status,
    duration_ms: Date.now() - start,
  })
}
```

## Common mistakes

- **Using Puppeteer or Playwright for web scraping on Vercel** — Headless browsers require 200MB+ of Chromium, exceeding Vercel serverless function size limits. They also take 5-10 seconds just to launch, eating into timeout budgets. Fix: Use Cheerio for HTML parsing. It loads the HTML string and provides jQuery-like CSS selectors without a browser. For JavaScript-rendered pages, use a third-party service like ScrapingBee.
- **Not setting maxDuration in the route segment config** — Vercel Hobby plan defaults to 10 seconds. Slow websites or large pages will timeout before Cheerio can parse them, returning a 504 error. Fix: Export const maxDuration = 60 at the top of the route file. This extends the timeout to 60 seconds on Pro plans. On Hobby, the maximum is 10 seconds.
- **Storing API keys in plain text** — If someone gains read access to the database (SQL injection, leaked credentials, backup exposure), they get every user's API key. Fix: Hash API keys with SHA-256 before storing. Show the full key only once during generation. Store only the prefix (first 8 chars) for display in the dashboard.
- **Not setting a fetch timeout for target URLs** — Some websites take 30+ seconds to respond or hang indefinitely. Without a timeout, your serverless function waits until Vercel kills it. Fix: Use AbortSignal.timeout(15000) in the fetch call. This aborts the request after 15 seconds, leaving time for Cheerio parsing within the overall maxDuration limit.

## Best practices

- Use Cheerio instead of headless browsers for Vercel serverless — it parses HTML in milliseconds without Chromium overhead
- Set export const maxDuration = 60 in the route segment config to extend Vercel's serverless timeout
- Hash API keys with SHA-256 before storage and only show the full key once during generation
- Use AbortSignal.timeout() on fetch calls to prevent hanging requests from consuming your entire timeout budget
- Cache scrape results in Supabase with a TTL to avoid re-scraping unchanged pages repeatedly
- Use the @attr suffix convention in selectors (e.g., 'img.hero@src') to extract HTML attributes instead of text content
- Set SUPABASE_SERVICE_ROLE_KEY and SCRAPINGBEE_API_KEY in V0's Vars tab without NEXT_PUBLIC_ prefix

## Frequently asked questions

### Why Cheerio instead of Puppeteer?

Cheerio parses HTML strings using CSS selectors in milliseconds without needing a browser. Puppeteer requires 200MB+ of Chromium, exceeds Vercel serverless size limits, and takes 5-10 seconds just to launch. For static HTML pages, Cheerio is 100x faster and actually works on Vercel.

### What about JavaScript-rendered pages?

Cheerio only parses the initial HTML response. For SPAs that render content with JavaScript, integrate with ScrapingBee (1,000 free requests) which runs a headless browser on their infrastructure and returns the rendered HTML for Cheerio to parse.

### What is the Vercel timeout limit?

Hobby plan: 10 seconds. Pro plan: 60 seconds (with maxDuration = 60). Enterprise: 300 seconds. Set export const maxDuration = 60 in your route file to use the extended timeout on Pro.

### How do API keys work?

When a user generates a key, the server creates a random string (rsk_...), shows it once, and stores only the SHA-256 hash. On each API request, the provided key is hashed and compared against stored hashes. This means even database access does not reveal the actual keys.

### What V0 plan do I need?

V0 Premium ($20/month) is recommended. The scraping API involves multiple API routes, API key management, and a dashboard that require several prompt iterations.

### How do I deploy this?

Click Share then Publish in V0. Set SUPABASE_SERVICE_ROLE_KEY in V0's Vars tab (no NEXT_PUBLIC_ prefix). Optionally add SCRAPINGBEE_API_KEY for JavaScript rendering fallback. The scraping API is immediately accessible at your Vercel production URL.

### Can RapidDev help build a custom data extraction platform?

Yes. RapidDev has built 600+ apps including data pipeline platforms with scraping, transformation, and visualization. Book a free consultation to discuss your data extraction requirements.

---

Source: https://www.rapidevelopers.com/how-to-build-v0/web-scraping-api
© RapidDev — https://www.rapidevelopers.com/how-to-build-v0/web-scraping-api