# How to Build a Content Moderation Tool with V0
- Tool: v0
- Difficulty: Advanced
- Compatibility: V0 Premium recommended, Supabase free tier, OpenAI API account
- Last updated: April 2026
## TL;DR
Build a content moderation tool that automatically scores user-generated content using the OpenAI Moderation API, queues flagged items for human review, and tracks enforcement actions with full audit trails. V0 generates the Next.js review queue and admin interface while Supabase database webhooks trigger the auto-moderation pipeline on new content.
## Before you start
- A V0 account (Premium recommended for multi-file generation)
- A Supabase project with database webhooks enabled
- An OpenAI API key for the Moderation API
- Basic familiarity with Next.js App Router and TypeScript
## Step-by-step guide
### 1. Create the Supabase Schema with Strike Trigger
*The three-strike trigger automates user bans at the database level, ensuring consistent enforcement regardless of which moderator takes the action or whether it is done through the UI or API.*
Set up tables for content items, moderation actions, moderation rules, and user strikes. Create a trigger on moderation_actions that increments strike_count when a warning is issued and automatically sets is_banned when strikes reach three.
```
-- Content items to moderate
create table content_items (
id uuid primary key default gen_random_uuid(),
author_id uuid references auth.users not null,
content_type text check (content_type in ('post','comment','image','profile')) not null,
content_text text,
content_url text,
status text check (status in ('pending','approved','rejected','escalated')) default 'pending',
auto_score numeric,
auto_categories text[] default '{}',
source_table text,
source_id uuid,
created_at timestamptz default now()
);
-- Moderation actions (audit trail)
create table moderation_actions (
id uuid primary key default gen_random_uuid(),
content_id uuid references content_items not null,
moderator_id uuid references auth.users not null,
action text check (action in ('approve','reject','escalate','warn','ban')) not null,
reason text,
created_at timestamptz default now()
);
-- Custom moderation rules
create table moderation_rules (
id uuid primary key default gen_random_uuid(),
name text not null,
pattern text not null,
action text check (action in ('flag','auto_reject')) not null,
is_active boolean default true,
created_at timestamptz default now()
);
-- User strike tracking
create table user_strikes (
user_id uuid primary key references auth.users,
strike_count int default 0,
is_banned boolean default false,
last_strike_at timestamptz
);
-- Trigger: increment strikes on warn/ban action
create or replace function handle_moderation_action()
returns trigger as $$
begin
if NEW.action in ('warn', 'ban') then
-- Get the content author
declare
v_author_id uuid;
begin
select author_id into v_author_id
from content_items where id = NEW.content_id;
-- Upsert user strikes
insert into user_strikes (user_id, strike_count, last_strike_at)
values (v_author_id, 1, now())
on conflict (user_id) do update set
strike_count = user_strikes.strike_count + 1,
last_strike_at = now();
-- Auto-ban at 3 strikes
update user_strikes
set is_banned = true
where user_id = v_author_id and strike_count >= 3;
end;
end if;
return NEW;
end;
$$ language plpgsql security definer;
create trigger on_moderation_action
after insert on moderation_actions
for each row execute function handle_moderation_action();
-- RLS
alter table content_items enable row level security;
alter table moderation_actions enable row level security;
alter table moderation_rules enable row level security;
alter table user_strikes enable row level security;
-- Only moderators can access (use a role check in practice)
create policy "Moderators read content" on content_items for select using (true);
create policy "Moderators update content" on content_items for update using (true);
create policy "Moderators read actions" on moderation_actions for select using (true);
create policy "Moderators create actions" on moderation_actions for insert with check (true);
create policy "Moderators manage rules" on moderation_rules for all using (true);
create policy "Moderators read strikes" on user_strikes for select using (true);
```
**Expected result:** Database tables created with a trigger that automatically tracks strikes and bans users after three warnings.
### 2. Build the Auto-Moderation API Pipeline
*Running the OpenAI Moderation API and custom regex rules in parallel speeds up the scoring pipeline. Storing the OPENAI_API_KEY without NEXT_PUBLIC_ prefix in V0's Vars tab ensures it is never exposed to the browser.*
Create an API route that receives content items, runs OpenAI Moderation API and custom regex rules concurrently, computes a combined score, and either auto-approves clean content or queues it for human review.
```
// app/api/moderate/auto/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { createClient } from '@supabase/supabase-js';
const supabase = createClient(
process.env.NEXT_PUBLIC_SUPABASE_URL!,
process.env.SUPABASE_SERVICE_ROLE_KEY!
);
const AUTO_APPROVE_THRESHOLD = 0.3;
const WEBHOOK_SECRET = process.env.MODERATION_WEBHOOK_SECRET!;
export async function POST(request: NextRequest) {
// Validate webhook secret
const authHeader = request.headers.get('authorization');
if (authHeader !== `Bearer ${WEBHOOK_SECRET}`) {
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
}
const { record } = await request.json();
const contentId = record.id;
const contentText = record.content_text || '';
// Run OpenAI Moderation and custom rules in parallel
const [aiResult, ruleResult] = await Promise.all([
moderateWithOpenAI(contentText),
moderateWithRules(contentText),
]);
// Combine scores (max of AI score and rule severity)
const combinedScore = Math.max(aiResult.score, ruleResult.score);
const categories = [...aiResult.categories, ...ruleResult.matchedRules];
// Update content item with scores
await supabase.from('content_items').update({
auto_score: combinedScore,
auto_categories: categories,
status: combinedScore < AUTO_APPROVE_THRESHOLD ? 'approved' : 'pending',
}).eq('id', contentId);
return NextResponse.json({
contentId,
score: combinedScore,
status: combinedScore < AUTO_APPROVE_THRESHOLD ? 'auto_approved' : 'queued_for_review',
});
}
async function moderateWithOpenAI(text: string) {
if (!text.trim()) return { score: 0, categories: [] };
const response = await fetch('https://api.openai.com/v1/moderations', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
},
body: JSON.stringify({ input: text }),
});
const data = await response.json();
const result = data.results[0];
// Get the highest category score
const scores = Object.values(result.category_scores) as number[];
const maxScore = Math.max(...scores);
// Get flagged categories
const flaggedCategories = Object.entries(result.categories)
.filter(([, flagged]) => flagged)
.map(([category]) => category);
return { score: maxScore, categories: flaggedCategories };
}
async function moderateWithRules(text: string) {
const { data: rules } = await supabase
.from('moderation_rules')
.select('*')
.eq('is_active', true);
const matchedRules: string[] = [];
let maxScore = 0;
for (const rule of rules ?? []) {
try {
const regex = new RegExp(rule.pattern, 'gi');
if (regex.test(text)) {
matchedRules.push(rule.name);
maxScore = Math.max(maxScore, rule.action === 'auto_reject' ? 1.0 : 0.7);
}
} catch {
// Skip invalid regex patterns
}
}
return { score: maxScore, matchedRules };
}
```
**Expected result:** The API route scores content using OpenAI and custom rules in parallel, auto-approving clean content and queuing flagged content for human review.
### 3. Build the Moderation Review Queue
*Moderators need a focused interface to quickly review flagged content, see the severity score, and take enforcement actions. Color-coded badges help prioritize the most severe items.*
Create the review queue page with filters for status, content type, and severity. Each item shows a preview, auto-score with color coding, and quick action buttons for approve, reject, escalate, and warn.
```
// app/moderation/page.tsx
import { createClient } from '@/lib/supabase/server';
import { Badge } from '@/components/ui/badge';
import { Button } from '@/components/ui/button';
import { Card, CardContent } from '@/components/ui/card';
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
import { ModerationActions } from './moderation-actions';
import { Shield, AlertTriangle, CheckCircle, XCircle } from 'lucide-react';
function getSeverityColor(score: number): string {
if (score >= 0.8) return 'bg-red-100 text-red-800';
if (score >= 0.5) return 'bg-orange-100 text-orange-800';
if (score >= 0.3) return 'bg-yellow-100 text-yellow-800';
return 'bg-green-100 text-green-800';
}
export default async function ModerationPage() {
const supabase = await createClient();
const { data: items } = await supabase
.from('content_items')
.select('*')
.in('status', ['pending', 'escalated'])
.order('auto_score', { ascending: false })
.order('created_at', { ascending: true });
const { data: stats } = await supabase
.from('content_items')
.select('status')
.in('status', ['pending', 'escalated']);
const pending = stats?.filter(s => s.status === 'pending').length ?? 0;
const escalated = stats?.filter(s => s.status === 'escalated').length ?? 0;
const allItems = items ?? [];
return (