# How to Build a Web Scraping API with Lovable

- Tool: How to Build with Lovable
- Difficulty: Advanced
- Compatibility: Lovable Pro or higher
- Last updated: April 2026

## TL;DR

Build a web scraping API in Lovable using Supabase Edge Functions with fetch and cheerio for HTML parsing. Features a scrape jobs queue with pg_cron scheduling, structured JSONB result storage, and Firecrawl as a fallback for anti-bot protected sites. Manage and monitor all scraping jobs from a dashboard.

## Before you start

- Lovable Pro account for multiple Edge Functions
- Supabase Pro plan for pg_cron (available on all plans but requires pg_cron extension enabled)
- Firecrawl API key from firecrawl.dev (free tier: 500 credits)
- Supabase service role key and Firecrawl API key saved to Cloud tab → Secrets

## Step-by-step guide

### 1. Create the job queue schema

Prompt Lovable to set up the scrape jobs and results tables. The queue design is the foundation — job status lifecycle and priority ordering determine scraping throughput.

```
Build a web scraping API. Create these Supabase tables:

- scrape_jobs: id, user_id, url (text), selector_config (jsonb, e.g. { fieldName: 'css-selector' }), priority (int default 5, 1=highest 10=lowest), status (pending|running|done|failed|retrying), attempt_count (int default 0), max_attempts (int default 3), use_firecrawl (bool default false), error_message (text), created_at, started_at, completed_at

- scrape_results: id, job_id (FK scrape_jobs UNIQUE), extracted_data (jsonb), raw_html_url (text, Supabase Storage path), page_title (text), scraped_at, response_status_code (int), response_time_ms (int)

RLS:
- scrape_jobs: user_id = auth.uid() for all operations
- scrape_results: accessible via job_id FK to user's jobs (check via EXISTS subquery)

Create indexes:
- CREATE INDEX idx_scrape_jobs_queue ON scrape_jobs(status, priority, created_at) WHERE status = 'pending'
- CREATE INDEX idx_scrape_jobs_user ON scrape_jobs(user_id, created_at DESC)

Enable pg_cron extension: it should already be enabled on your Supabase project. If not, ask in Supabase support chat.
```

> Pro tip: Ask Lovable to create a scrape_job_templates table where users can save common URL patterns and selector configs (e.g. 'E-commerce Product Page' template with selectors for title, price, availability). Templates speed up new job creation.

**Expected result:** Both tables are created with the queue index. The app loads with a job submission form and a queue DataTable.

### 2. Build the scrape worker Edge Function

Create the Edge Function that does the actual scraping. It fetches the page, applies CSS selectors, stores results, and handles Firecrawl fallback.

```
// supabase/functions/scrape-worker/index.ts
import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'
import { createClient } from 'https://esm.sh/@supabase/supabase-js@2'
import { load } from 'https://esm.sh/cheerio@1.0.0-rc.12'

const corsHeaders = { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization, apikey, content-type' }

serve(async (req: Request) => {
  if (req.method === 'OPTIONS') return new Response('ok', { headers: corsHeaders })

  const supabase = createClient(Deno.env.get('SUPABASE_URL') ?? '', Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? '')
  const { jobId } = await req.json()

  const { data: job } = await supabase.from('scrape_jobs').select('*').eq('id', jobId).single()
  if (!job) return new Response(JSON.stringify({ error: 'Job not found' }), { status: 404, headers: corsHeaders })

  await supabase.from('scrape_jobs').update({ status: 'running', started_at: new Date().toISOString(), attempt_count: job.attempt_count + 1 }).eq('id', jobId)

  const start = Date.now()
  try {
    let html = ''
    let statusCode = 200

    if (job.use_firecrawl) {
      html = await scrapeWithFirecrawl(job.url)
    } else {
      const res = await fetch(job.url, {
        headers: { 'User-Agent': 'Mozilla/5.0 (compatible; scraper/1.0)' },
        signal: AbortSignal.timeout(15000),
      })
      statusCode = res.status
      if (res.status === 403 || res.status === 429) {
        html = await scrapeWithFirecrawl(job.url)
      } else {
        html = await res.text()
      }
    }

    const $ = load(html)
    const extractedData: Record<string, string> = {}
    for (const [field, selector] of Object.entries(job.selector_config as Record<string, string>)) {
      extractedData[field] = $(selector).first().text().trim()
    }

    await supabase.from('scrape_results').upsert({
      job_id: jobId,
      extracted_data: extractedData,
      page_title: $('title').text().trim(),
      scraped_at: new Date().toISOString(),
      response_status_code: statusCode,
      response_time_ms: Date.now() - start,
    }, { onConflict: 'job_id' })

    await supabase.from('scrape_jobs').update({ status: 'done', completed_at: new Date().toISOString() }).eq('id', jobId)
    return new Response(JSON.stringify({ success: true, fields: Object.keys(extractedData).length }), { headers: corsHeaders })
  } catch (err) {
    const msg = err instanceof Error ? err.message : 'Scrape failed'
    const nextStatus = job.attempt_count + 1 >= job.max_attempts ? 'failed' : 'pending'
    await supabase.from('scrape_jobs').update({ status: nextStatus, error_message: msg, completed_at: nextStatus === 'failed' ? new Date().toISOString() : null }).eq('id', jobId)
    return new Response(JSON.stringify({ error: msg }), { status: 500, headers: corsHeaders })
  }
})

async function scrapeWithFirecrawl(url: string): Promise<string> {
  const res = await fetch('https://api.firecrawl.dev/v1/scrape', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${Deno.env.get('FIRECRAWL_API_KEY')}` },
    body: JSON.stringify({ url, formats: ['html'] }),
  })
  const data = await res.json()
  if (!res.ok) throw new Error(data.error ?? 'Firecrawl failed')
  return data.data?.html ?? data.data?.markdown ?? ''
}
```

**Expected result:** The scrape-worker Edge Function deploys. Calling it manually with a jobId fetches the URL, applies selectors, and stores results in scrape_results.

### 3. Set up the pg_cron queue processor

Create the pg_cron job that polls for pending scrape jobs and dispatches them to the scrape-worker Edge Function. This makes the queue fully automated.

```
Set up the automated queue processor:

1. Create an Edge Function at supabase/functions/process-queue/index.ts that:
   - Queries scrape_jobs WHERE status = 'pending' ORDER BY priority ASC, created_at ASC LIMIT 5
   - For each job: updates status to 'running' (to claim it), then calls the scrape-worker Edge Function via fetch
   - Uses Promise.allSettled to run up to 5 jobs concurrently
   - Returns a JSON summary: { dispatched: number, jobIds: string[] }

2. Register the pg_cron schedule in Supabase SQL editor:
SELECT cron.schedule(
  'process-scrape-queue',
  '* * * * *',
  $$
  SELECT net.http_post(
    url:='https://YOUR_PROJECT.supabase.co/functions/v1/process-queue',
    headers:=json_build_object('Authorization', 'Bearer YOUR_SERVICE_ROLE_KEY')::jsonb
  ) AS request_id;
  $$
);

3. Add a job intake Edge Function at supabase/functions/submit-scrape-job/index.ts that:
   - Accepts POST with { url, selectorConfig, priority?, useFirecrawl? }
   - Validates the URL is a valid HTTP/HTTPS URL
   - Inserts into scrape_jobs and returns the new job ID
   - Can be called without auth (add an API key check using the same pattern as the api-backend guide) or with Supabase Auth
```

> Pro tip: Add a concurrency control: before dispatching jobs, check how many are currently status='running'. If already at 5 running, skip this pg_cron tick. This prevents queue pile-up if jobs take longer than 1 minute.

**Expected result:** The pg_cron job runs every minute. Submitting a job via the intake Edge Function shows it as 'pending' in the dashboard, then transitions to 'running' and 'done' within 1-2 minutes.

### 4. Build the scraping dashboard

Create the management dashboard where users can submit new scrape jobs, see queue status, inspect results, and monitor error rates.

```
Build the scraping dashboard at src/pages/ScrapingDashboard.tsx:

1. Summary Cards at top: Jobs Today, Queue Depth (pending count), Success Rate %, Average Response Time

2. Job submission form (Card or Sheet):
   - URL Input (required, validated as URL format)
   - Selector Config builder: a key-value list where users add field name (e.g. 'price') + CSS selector (e.g. '.price-tag'). Show Add Row Button and remove buttons per row.
   - Priority Slider (1-10)
   - Use Firecrawl Checkbox with label 'Use Firecrawl for JavaScript-heavy sites (uses credits)'
   - Submit Button: calls the submit-scrape-job Edge Function

3. Job queue DataTable with columns: created_at (relative), URL (truncated with Tooltip for full URL), Priority Badge, Status Badge (pending=gray, running=blue with spinner, done=green, failed=red, retrying=yellow), attempt_count, Actions menu (View Results, Retry, Delete)

4. Results Sheet (opens when clicking View Results):
   - URL and scraped_at
   - Extracted data as a key-value table (field name → extracted text)
   - Response status code Badge and response time
   - Error message if failed

5. Recharts BarChart below the table: jobs per hour over the last 24 hours, colored by status (done=green, failed=red stacked)
```

**Expected result:** The dashboard shows queue stats. Submitting a job adds it to the DataTable. After 1-2 minutes, the status changes to 'done' and clicking 'View Results' shows the extracted data.

## Complete code example

File: `src/components/scraping/SelectorBuilder.tsx`

```typescript
import { useState } from 'react'
import { Button } from '@/components/ui/button'
import { Input } from '@/components/ui/input'
import { Label } from '@/components/ui/label'
import { Trash2, Plus } from 'lucide-react'

export interface SelectorRow {
  field: string
  selector: string
}

interface Props {
  value: SelectorRow[]
  onChange: (rows: SelectorRow[]) => void
}

export function SelectorBuilder({ value, onChange }: Props) {
  function addRow() {
    onChange([...value, { field: '', selector: '' }])
  }

  function updateRow(index: number, key: keyof SelectorRow, newValue: string) {
    const updated = value.map((row, i) => (i === index ? { ...row, [key]: newValue } : row))
    onChange(updated)
  }

  function removeRow(index: number) {
    onChange(value.filter((_, i) => i !== index))
  }

  return (
    <div className="space-y-2">
      {value.length > 0 && (
        <div className="grid grid-cols-[1fr_1fr_auto] gap-2 text-sm font-medium text-muted-foreground px-1">
          <span>Field Name</span>
          <span>CSS Selector</span>
          <span />
        </div>
      )}
      {value.map((row, i) => (
        <div key={i} className="grid grid-cols-[1fr_1fr_auto] gap-2 items-center">
          <Input
            placeholder="price"
            value={row.field}
            onChange={(e) => updateRow(i, 'field', e.target.value)}
          />
          <Input
            placeholder=".product-price"
            value={row.selector}
            onChange={(e) => updateRow(i, 'selector', e.target.value)}
            className="font-mono text-sm"
          />
          <Button variant="ghost" size="icon" onClick={() => removeRow(i)} className="text-destructive hover:text-destructive">
            <Trash2 className="h-4 w-4" />
          </Button>
        </div>
      ))}
      <Button variant="outline" size="sm" onClick={addRow} className="w-full">
        <Plus className="mr-2 h-4 w-4" />
        Add Field
      </Button>
    </div>
  )
}
```

## Common mistakes

- **Scraping sites without checking robots.txt** — Most sites have a /robots.txt file that specifies which paths may be crawled and which must not be. Ignoring robots.txt is against web etiquette and may be illegal depending on jurisdiction. Fix: Before scraping, fetch and parse the target site's /robots.txt. Check if the requested path is allowed for your User-Agent. Add a robots_check column to scrape_jobs and set it to false if robots.txt disallows the URL. Show a warning in the dashboard.
- **Using cheerio selectors that break when the page layout changes** — CSS selectors based on class names like '.product-price-sale-2024' are extremely brittle. The site's CSS class names change frequently, breaking all dependent selectors silently. Fix: Prefer semantic selectors: tag + attribute combinations like [itemprop='price'], structured data selectors, or data attribute selectors like [data-testid='price']. These change less frequently than generated CSS class names. Document why each selector was chosen.
- **Not setting a timeout on fetch() calls** — Some web servers accept the connection but never send a response body. Without a timeout, the Edge Function hangs until Supabase terminates it (up to 150 seconds), wasting execution time. Fix: Always use AbortSignal.timeout() with fetch: signal: AbortSignal.timeout(15000) for a 15-second timeout. Catch the timeout error specifically and log it as a distinct error type so you know how often pages are hanging.
- **Claiming jobs by status update without atomicity** — If the process-queue function reads 5 pending jobs and two pg_cron ticks overlap, both ticks may claim the same jobs, causing duplicate scraping. Fix: Use a PostgreSQL UPDATE ... RETURNING with a WHERE status = 'pending' LIMIT 5 and update to 'running' in a single atomic statement. This is an atomic claim operation that prevents duplicate processing. Ask Lovable to use supabase.rpc('claim_scrape_jobs', { count: 5 }) with a SECURITY DEFINER function.

## Best practices

- Treat web scraping as a privilege, not a right. Always check robots.txt, use reasonable request rates (no more than 1 request per second per domain), and add a descriptive User-Agent so site operators know who is accessing their site.
- Store the raw HTML in Supabase Storage alongside the extracted JSONB data. This lets you re-run selector extraction against historical HTML without fetching the page again — essential when you realize your CSS selector was wrong.
- Use Firecrawl or a similar managed scraping service as a fallback, not a primary scraper. Direct fetch is cheaper and faster. Fall back to Firecrawl only on 403, 429, or when the extracted data is clearly empty (which may indicate a JavaScript-rendered page).
- Add a domain-level rate limiter. Before dispatching a job, check how many other jobs for the same domain are currently running. Limit to 1-2 concurrent requests per domain to avoid being blocked.
- Never store scraped data that includes personal information (names, emails, phone numbers) without a clear legal basis. Design your selector configs to extract only publicly available, non-PII structured data like prices, product names, and availability.
- Test CSS selectors on a static snapshot before deploying. Save a local copy of the target page HTML and run your cheerio selectors against it to verify they return the expected data.

## Frequently asked questions

### Is web scraping legal?

Web scraping occupies a legally complex space. Scraping publicly accessible data is generally permitted, but scraping data behind authentication, ignoring robots.txt, circumventing technical measures, or scraping personal data in EU jurisdictions (GDPR) may create legal liability. Always check a site's Terms of Service before scraping. Use scraped data only as permitted. This guide is for building the infrastructure — responsibility for compliance rests with the operator.

### When should I use Firecrawl vs direct fetch?

Use direct fetch (Deno's built-in fetch()) first — it's free and faster. Fall back to Firecrawl when: the site returns 403 or 429, the extracted data is empty (suggesting JavaScript rendering), or the page content is loaded by client-side JavaScript that isn't present in the initial HTML. Firecrawl uses headless browsers and handles anti-bot measures, but uses paid credits. Keep Firecrawl as a fallback, not the default.

### Can I scrape sites that require login?

Yes, but it requires additional complexity. You'd need to store session cookies or auth tokens in Supabase Vault (using the same credential vault pattern as the integration hub guide) and include them in Edge Function requests. This crosses into territory where Terms of Service violations are more likely. Ensure you have explicit permission from the site operator before scraping authenticated content.

### What happens if the target site changes its HTML structure?

Your CSS selectors return empty strings silently. Add a validation step: after extracting data, check if required fields are empty. If a required field is empty, mark the job as 'failed' with an error message like 'Selector .price-tag returned empty result — the page structure may have changed.' This alerts you to re-check and update your selectors.

### How many scrape jobs can I run per day on the free plan?

Supabase Edge Functions on the Free plan allow 500,000 invocations per month and each invocation can run for up to 150 seconds. For scraping, assume each job takes 5-15 seconds average. At 5 concurrent jobs per minute via pg_cron, you can process roughly 7,200 jobs per day. The binding constraint is usually the target site's rate limits, not Supabase's. Firecrawl's free tier adds a limit of 500 credits per month.

### Can I use this to monitor competitor prices?

Technically yes, but with caveats. Price monitoring via scraping is common but operates in a legal gray area depending on the site's Terms of Service and jurisdiction. Many e-commerce sites explicitly prohibit automated price checking in their ToS. If you need price data reliably, consider official data partners or price intelligence APIs that provide structured product data through legitimate licensing agreements.

### How do I extract data from PDFs instead of HTML pages?

PDF extraction requires a different approach — cheerio cannot parse PDFs. For PDFs, the Edge Function should detect the content type from the response headers (application/pdf), upload the raw PDF to Supabase Storage, and then call a PDF parsing service API. Ask Lovable to add a pdf_extraction_mode flag to scrape_jobs that changes the worker logic to use a PDF parsing API instead of cheerio.

---

Source: https://www.rapidevelopers.com/how-to-build-lovable/web-scraping-api
© RapidDev — https://www.rapidevelopers.com/how-to-build-lovable/web-scraping-api