How to Integrate Screaming Frog with V0

The first step is generating the export file from Screaming Frog. The export format you choose determines how your V0 dashboard parses and displays the data. To export all crawl data as a single file in Screaming Frog: run your crawl by entering the URL in the 'Enter URL to spider' field and clicking 'Start'. Wait for the crawl to complete. Then go to File → Export in the menu bar and choose 'All Exports' to export all data as a CSV. Alternatively, go to Bulk Export → All Inlinks to export link data, or use the specific internal tabs (Response Codes, Page Titles, Meta Description, etc.) and use the Export button in the bottom-right corner of each tab to export just that data. For a more structured export, Screaming Frog supports JSON export via the API and command-line interface. However, for the browser-based upload workflow, CSV is simpler to work with since it does not require the Screaming Frog API configuration. The most useful exports for a custom dashboard are: - 'Internal HTML' tab → Export: gives you all HTML pages with title, meta description, H1, word count, response code, and size - Response Codes tab (4xx and 5xx filtered): gives you just the error pages - 'Bulk Export → Response Codes → Client Error (4xx) Inlinks': gives you broken internal links with source and anchor text For a comprehensive audit dashboard, export the 'Internal HTML' tab which includes the most metadata per page. The CSV headers match Screaming Frog's column names exactly — note these because your parser must reference the same column names.

Create a Next.js API route that accepts the uploaded Screaming Frog CSV file and parses it into structured data. This route uses Next.js FormData handling to receive the file upload and a CSV parser to extract the data. Create app/api/seo/parse-crawl/route.ts. This route handles a POST request with a multipart form containing the CSV file. Install the csv-parse package (npm install csv-parse) for reliable CSV parsing that handles quoted fields, special characters, and Screaming Frog's specific CSV dialect. Screaming Frog's CSV exports use standard comma separation with the first row as column headers. The exact column names depend on which export tab you used. For the 'Internal HTML' export, key columns include: 'Address' (the URL), 'Status Code', 'Title 1', 'Title 1 Length', 'Meta Description 1', 'Meta Description 1 Length', 'H1-1', 'H1-1 Length', 'Word Count', 'Indexability', 'Indexability Status', 'Inlinks', 'Size' (bytes), and 'Response Time' (milliseconds). The API route reads the uploaded file as text, passes it to csv-parse in synchronous mode to get an array of row objects, then transforms the rows into a clean data structure. Return the parsed data as JSON with summary statistics. For large crawls (10,000+ URLs), the CSV file can be several megabytes. Vercel's serverless functions support request bodies up to 4.5 MB. For larger crawls, consider parsing and storing the data in a database (Neon or Supabase) instead of keeping it in-memory per request.

1import { NextRequest, NextResponse } from 'next/server';
2import { parse } from 'csv-parse/sync';
3
4interface CrawlPage {
5  url: string;
6  statusCode: number;
7  title: string;
8  titleLength: number;
9  metaDescription: string;
10  metaDescriptionLength: number;
11  h1: string;
12  wordCount: number;
13  inLinks: number;
14  size: number;
15  responseTime: number;
16  indexability: string;
17}
18
19export async function POST(request: NextRequest) {
20  try {
21    const formData = await request.formData();
22    const file = formData.get('file') as File | null;
23
24    if (!file) {
25      return NextResponse.json({ error: 'No file uploaded' }, { status: 400 });
26    }
27
28    if (!file.name.endsWith('.csv')) {
29      return NextResponse.json(
30        { error: 'Please upload a CSV file from Screaming Frog' },
31        { status: 400 }
32      );
33    }
34
35    const csvText = await file.text();
36
37    // Parse CSV with column headers from first row
38    const records = parse(csvText, {
39      columns: true,         // Use first row as keys
40      skip_empty_lines: true,
41      trim: true,
42      relax_column_count: true,
43    }) as Record<string, string>[];
44
45    if (records.length === 0) {
46      return NextResponse.json({ error: 'CSV file is empty' }, { status: 400 });
47    }
48
49    // Detect column name format (Screaming Frog may vary by export type)
50    const firstRow = records[0];
51    const urlKey =
52      'Address' in firstRow ? 'Address'
53        : 'URL' in firstRow ? 'URL'
54        : Object.keys(firstRow)[0];
55
56    const pages: CrawlPage[] = records.map((row) => ({
57      url: row[urlKey] || '',
58      statusCode: parseInt(row['Status Code'] || '0', 10),
59      title: row['Title 1'] || row['Title'] || '',
60      titleLength: parseInt(row['Title 1 Length'] || row['Title Length'] || '0', 10),
61      metaDescription: row['Meta Description 1'] || row['Meta Description'] || '',
62      metaDescriptionLength: parseInt(
63        row['Meta Description 1 Length'] || row['Meta Description Length'] || '0',
64        10
65      ),
66      h1: row['H1-1'] || row['H1'] || '',
67      wordCount: parseInt(row['Word Count'] || '0', 10),
68      inLinks: parseInt(row['Inlinks'] || '0', 10),
69      size: parseInt(row['Size'] || '0', 10),
70      responseTime: parseInt(row['Response Time'] || '0', 10),
71      indexability: row['Indexability'] || '',
72    })).filter((p) => p.url.length > 0);
73
74    const summary = {
75      total: pages.length,
76      errorPages: pages.filter((p) => p.statusCode >= 400).length,
77      missingTitles: pages.filter((p) => p.titleLength === 0).length,
78      longTitles: pages.filter((p) => p.titleLength > 60).length,
79      missingDescriptions: pages.filter((p) => p.metaDescriptionLength === 0).length,
80      longDescriptions: pages.filter((p) => p.metaDescriptionLength > 160).length,
81      thinContent: pages.filter((p) => p.wordCount > 0 && p.wordCount < 300).length,
82      avgResponseTime: Math.round(
83        pages.reduce((sum, p) => sum + p.responseTime, 0) / pages.length
84      ),
85    };
86
87    return NextResponse.json({ pages, summary });
88  } catch (error) {
89    console.error('CSV parse error:', error);
90    return NextResponse.json(
91      { error: 'Failed to parse CSV file. Ensure it is a valid Screaming Frog export.' },
92      { status: 500 }
93    );
94  }
95}

With the parsing API route in place, prompt V0 to generate the dashboard interface. The dashboard needs two main areas: a file upload component for importing the Screaming Frog export, and the visualization components that render after the file is parsed. The upload component should be prominent and clear about what file format to upload. Tell V0 to accept only .csv files and to display a clear instruction: 'Export your crawl from Screaming Frog → File → Export → All Exports, then upload the CSV here.' After a successful upload, the dashboard transitions to showing the crawl data. Ask V0 to use React state to hold the parsed data — upload sends the file to /api/seo/parse-crawl, receives the parsed data as JSON, then stores it in useState. This avoids storing large crawl data on the server and keeps everything client-side after the initial parse. The visualization components should emphasize actionable information: the most common error types at the top (404s, missing titles, duplicate H1s), followed by filterable tables where the user can drill into specific issues. A page-level severity score (calculated from the count of issues per URL) helps prioritize which pages to fix first. Ask V0 to add a 'Download Report' button that generates a filtered CSV from the displayed data. This is useful for sharing specific issue sets with developers or content teams.

A summary table showing all pages is useful, but SEO professionals typically want to focus on one issue type at a time. Add drill-down views for the most common technical SEO issues: 4xx errors, pages with missing or problematic titles, and pages with thin content. For each issue type, ask V0 to generate a dedicated tab or section that shows only the affected pages with the relevant columns. For example, the 'Title Issues' view shows URL, current title, title length, and a recommendation column ('Missing — add a title' for 0-length, 'Too long — shorten by X characters' for titles over 60 characters, 'OK' for the rest). For the 4xx errors view, include a column showing the number of internal inlinks to each broken page. Pages with many inlinks pointing to them should be prioritized for fixing or redirecting — breaking a highly-linked page causes more crawl equity loss. Add an export function to each view: a button that downloads the current filtered view as a CSV. This is the most requested feature for SEO audit tools — the ability to export a list of URLs to fix and hand off to a developer or content editor. For SEO agencies who want to build a full-featured client reporting platform with automated Screaming Frog crawl scheduling, crawl comparison over time, and branded PDF export, RapidDev can help architect a more comprehensive system.