Feature spec
AdvancedCategory
analytics-admin
Build with AI
6-12 hours with FlutterFlow + Firebase Remote Config
Custom build
2-4 weeks custom dev
Running cost
$0/mo up to 10,000 users; $25/mo at 10,000+ users on Supabase Pro
Works on
Everything it takes to ship A/B Testing — parts, prompts, and real costs.
Mobile A/B testing needs four pieces: Firebase Remote Config for percentage-based variant assignment with local caching, a Supabase table persisting the assignment so it is sticky across sessions and Remote Config cache misses, event logging for conversion tracking, and an admin query or Edge Function that calculates conversion rates per variant. With FlutterFlow this takes 6-10 hours for $0/month — Firebase Remote Config and Supabase free tiers cover everything up to 10,000 users.
What Mobile A/B Testing Actually Is
A/B testing (also called split testing) silently assigns each user to one of two variants — a control (A) and a variant (B) — on their first app open, then measures which variant drives more of a target conversion event (a purchase, a sign-up, a button tap). The product team sees conversion rates per variant and declares a winner without shipping a new app release. The core mechanics are: assignment must be random and percentage-controlled, the same user must always see the same variant across sessions, conversion events must be tagged with the experiment and variant, and the admin dashboard must calculate statistical significance rather than just raw counts. Firebase Remote Config handles the assignment and percentage rollout; Supabase handles the persistence and event logging.
What users consider table stakes in 2026
- Users are silently assigned to variant A or B on first app open with no visible UI disruption or loading delay
- Assignment is sticky — reopening the app, clearing the cache, or switching network shows the same variant every time
- Product team can view conversion rate per variant in a dashboard without running a SQL query manually
- Variants can be rolled out to a configurable percentage (e.g. 10% see variant B, 90% see control A) without a new app release
- Results display include statistical significance so the team knows when they have enough data to declare a winner
- Active tests can be stopped and a winner can be rolled out to 100% of users from the admin dashboard without rebuilding the app
Anatomy of the Feature
Six components. Firebase Remote Config handles variant assignment. Supabase handles persistence and analytics. FlutterFlow wires them together with custom action blocks — most of the complexity lives in those blocks.
Feature flag / variant assignment
ServiceFirebase Remote Config assigns users to variants via percentage rollout conditions. The Flutter firebase_remote_config package (firebase_remote_config: ^5.x) fetches the active experiment config on app start via fetchAndActivate() and caches it locally. The condition uses a user property 'randomization_id' (set from the first 8 characters of the user's ID) to ensure stable percentage splits.
Note: Set minimumFetchInterval to Duration(hours: 1) — the default is Duration(hours: 12) in production, which means config changes take up to 12 hours to reach users. The free tier allows 150 active user fetches per day per project; additional fetches are throttled, not billed.
Variant renderer
UIFlutter conditional widget that reads the experimentVariant value fetched from Remote Config and renders the appropriate UI: if (experimentVariant == 'B') return VariantBWidget() else return ControlWidget(). Variants are stored in a Map<String, Widget> keyed by experiment_id for multi-experiment support.
Note: Always define the control variant (A) as the default value in Remote Config — if the fetch fails or times out, the app shows the control experience, which is always the safe fallback.
Assignment persistence
DataSupabase user_experiments table stores user_id, experiment_id, variant, and assigned_at. Written on the user's first app open after Remote Config returns an assignment. On subsequent opens, if Remote Config returns the default value (cache miss), the app reads from Supabase instead — ensuring sticky assignment even when Remote Config is temporarily unavailable.
Note: This is the fallback that makes assignment truly sticky. Without it, a Remote Config cache expiry can show a user variant B one session and variant A the next — invalidating the experiment results.
Event tracking
BackendSupabase user_experiment_events table logs conversion events tagged with experiment_id, variant, event_type (e.g. 'purchase_completed', 'button_tapped'), and metadata JSONB. The analytics query to compare variants: SELECT variant, COUNT(*) as conversions, COUNT(DISTINCT user_id) as participants FROM user_experiment_events WHERE experiment_id = $1 GROUP BY variant.
Note: Define event type strings in a constants file (lib/constants/experiment_events.dart) and reference the same constants in both the Flutter logging call and the admin dashboard query — mismatched strings are the #1 cause of dashboards showing zero conversions for variant B.
Analytics dashboard (admin)
UIA React admin page (built in Lovable or v0 as a companion web tool) or a Supabase Studio custom query showing: participants per variant, conversions per variant, conversion rate (%), and a chi-squared p-value computed in a Supabase Edge Function (check_significance). A p-value below 0.05 indicates statistical significance.
Note: A simple chi-squared test requires only the 2x2 contingency table (variant A conversions/non-conversions vs variant B conversions/non-conversions) — implementable in 20 lines of JavaScript in a Supabase Edge Function.
Remote Config fetch on cold start
Backendfirebase_remote_config.fetchAndActivate() is called in main() before runApp() completes. Sets minimumFetchInterval: Duration(hours: 1) for development and Duration(hours: 1) for production (balancing freshness vs the 150 fetches/day free quota). The fetch is async and awaited before the first experiment-gated screen is shown.
Note: Do not rely on background fetch for experiment assignment — iOS Background App Refresh is frequently disabled by users or the OS. Always fetch in the foreground app open lifecycle.
The data model
Three tables: experiments catalogue, per-user variant assignments, and conversion events. Paste this into the Supabase SQL Editor and run it before building the FlutterFlow integration:
1create table public.experiments (2 id uuid primary key default gen_random_uuid(),3 slug text not null unique,4 description text,5 status text not null default 'active' check (status in ('active', 'paused', 'completed')),6 started_at timestamptz not null default now(),7 ended_at timestamptz8);910create table public.user_experiments (11 user_id uuid references auth.users(id) on delete cascade not null,12 experiment_id uuid references public.experiments(id) on delete cascade not null,13 variant text not null,14 assigned_at timestamptz not null default now(),15 primary key (user_id, experiment_id)16);1718create table public.user_experiment_events (19 id uuid primary key default gen_random_uuid(),20 user_id uuid references auth.users(id) on delete cascade not null,21 experiment_id uuid references public.experiments(id) on delete cascade not null,22 variant text not null,23 event_type text not null,24 metadata jsonb,25 created_at timestamptz not null default now()26);2728-- RLS29alter table public.user_experiments enable row level security;30alter table public.user_experiment_events enable row level security;3132create policy "Users insert own assignment"33 on public.user_experiments for insert34 with check (auth.uid() = user_id);3536create policy "Users read own assignment"37 on public.user_experiments for select38 using (auth.uid() = user_id);3940create policy "Users insert own events"41 on public.user_experiment_events for insert42 with check (auth.uid() = user_id);4344-- Admin reads all via service_role key in Edge Function45-- (no public RLS policy for admin reads — use service_role)4647-- Indexes for analytics queries48create index exp_events_experiment_idx49 on public.user_experiment_events (experiment_id, variant, event_type);5051create index exp_events_user_idx52 on public.user_experiment_events (user_id, experiment_id, created_at desc);Heads up: The admin analytics query (conversion rate per variant) should use the Supabase service_role key in a server-side Edge Function or Next.js Server Action — never expose service_role to the Flutter client. RLS allows users to insert only their own rows.
Build it — pick your path
Each tool fits this feature differently. Switch tabs to compare fit, time, the exact steps, and a copy-paste prompt.
The correct path for production-grade experimentation: full control over assignment algorithm, multi-variant (A/B/C) testing, statistical power calculations, and early stopping rules — the only path when experiments must meet a scientific or regulatory bar.
Step by step
- 1Build a server-side assignment service (Supabase Edge Function) that receives user_id and experiment_id, applies the percentage split using a deterministic hash (SHA-256 of user_id + experiment_salt), writes to user_experiments, and returns the assigned variant
- 2Replace Firebase Remote Config with the server-side assignment — the Flutter app calls the Edge Function on cold start and caches the response in SharedPreferences as a fallback
- 3Implement Bayesian significance calculation in the analytics Edge Function: compute posterior probability that variant B's true conversion rate exceeds A's, accounting for sample size
- 4Build an experiment management dashboard with early stopping rules: automatically pause an experiment and notify the team via Slack when the p-value drops below 0.05 with at least 100 participants per variant
Where this path bites
- Server-side assignment adds 100-300ms to app cold start — use aggressive SharedPreferences caching to avoid this on warm starts
- Bayesian significance calculation is non-trivial to implement correctly — use a well-tested statistics library rather than a hand-rolled formula
Third-party services you'll need
A/B testing requires Firebase Remote Config for variant assignment and Supabase for persistence and analytics. Google Analytics for Firebase is optional for richer funnel analysis:
| Service | What it does | Free tier | Paid from |
|---|---|---|---|
| Firebase Remote Config | Variant assignment via percentage rollout conditions; caches assignment locally on device; no SDK cost | Free up to 150 active user fetches per day per project (quota resets per project, not per user); additional fetches are throttled, not billed | Free (no paid tier for Remote Config) |
| Supabase | Experiment assignment persistence (sticky across cache misses), conversion event logging, admin analytics via SQL queries and Edge Functions | Free tier: 500MB DB, 2 projects | Pro $25/mo (8GB DB) |
| Google Analytics for Firebase (optional) | Funnel analysis per variant; event stream to see conversion paths for A vs B users | Free with no event volume limit for standard events | Free |
Swipe the table sideways to see pricing.
What it costs to run
Drag through the tiers to see how your monthly bill scales with users — no surprises later.
Estimated monthly running cost
Firebase Remote Config free tier covers 150 fetches/day — more than enough for 100 active users. Supabase free tier handles assignment persistence and event logging with room to spare.
Estimates use verified 2026 pricing and assume typical usage per user. Your real bill depends on activity, storage, and third-party plans.
What breaks when AI tools build this
The failures people actually hit on their first build — the symptom, why it happens, and the exact fix.
User sees variant B one session and variant A the next
Symptom: Firebase Remote Config cache expired between sessions and fetchAndActivate() returned the default value instead of the cached variant. Without a Supabase persistence layer as fallback, the user is assigned to whichever variant is the Remote Config default — typically 'A'. This contaminates the experiment with users who have seen both variants.
Fix: After fetchAndActivate(), check whether the returned value differs from the previous session's assignment (stored in SharedPreferences). If it matches the default and differs from the stored value, read the variant from Supabase user_experiments instead. Call fetchAndActivate() in main() with minimumFetchInterval: Duration(hours: 1) to minimise cache misses.
Conversion events double-count because of app backgrounding
Symptom: The event logging function fires in both the AppLifecycleState.resumed callback and the initial screen build. When the user opens the app from background (not cold start), both triggers fire — inserting two conversion event rows for the same session, inflating the conversion count for variant B or A.
Fix: Guard event logging with a session-scoped static flag: static bool _conversionLogged = false in the screen's State class. Set it to true immediately after the first insert call and check the flag before every subsequent logging attempt. Reset the flag only on actual cold starts (in initState when the screen is created fresh).
All users assigned to control (variant A)
Symptom: Firebase Remote Config percentage condition uses the user property 'randomization_id' to split users, but the randomization_id is never set in the Flutter app. Without the user property, every user falls into the default condition — which returns the default value 'A'. The variant B group always has zero participants.
Fix: Set the randomization_id user property early in the app flow, before the Remote Config fetch: await FirebaseAnalytics.instance.setUserProperty(name: 'randomization_id', value: currentUser.uid.substring(0, 8)). This must happen before or concurrently with fetchAndActivate() for the condition to evaluate correctly.
Admin dashboard shows 0 conversions for variant B
Symptom: The event_type string logged by the Flutter code is 'button_tap' but the dashboard query filters on 'button_tapped'. This kind of naming mismatch is invisible during development because both strings are syntactically valid — the insert succeeds but the dashboard query returns no rows for variant B.
Fix: Define a constants file in Flutter: class ExperimentEvents { static const String buttonTapped = 'button_tapped'; static const String purchaseCompleted = 'purchase_completed'; } Reference these constants in both the Supabase insert call and the admin dashboard SQL. Never hardcode event type strings in two separate places.
Remote Config fetch blocked on iOS in background
Symptom: iOS Background App Refresh is disabled by the user or the OS. Remote Config fetch in a background task silently fails, and the user opens the app with a stale or missing variant assignment. In the worst case, they see the wrong variant because the Supabase fallback was not implemented.
Fix: Do not rely on background fetches for experiment assignment. Always call fetchAndActivate() in the foreground app open lifecycle (in main() before runApp()). Handle the async call gracefully: await with a 5-second timeout, then fall back to the Supabase-persisted assignment if the fetch times out.
Best practices
Always write the variant assignment to Supabase immediately after Remote Config returns it — the Supabase row is your source of truth when the Remote Config cache expires
Set a randomization_id user property before fetchAndActivate() — without it, Remote Config percentage conditions cannot split users and everyone gets the default variant
Define all event type strings in a constants file and import from it in every location where events are logged — naming mismatches between the Flutter client and the admin SQL query are the single most common cause of missing conversion data
Call fetchAndActivate() in main() before runApp() and await it with a timeout — never defer experiment assignment to a later screen, because the user may reach the experiment-gated screen before the fetch completes
Set minimumFetchInterval to Duration(hours: 1) in both development and production — the Remote Config free tier allows 150 unique active user fetches per day, and hourly fetching from a 10,000-user app easily exceeds this if not cached
Guard conversion event logging with a session-scoped boolean flag to prevent double-counting from app backgrounding and resuming
Require at least 100 participants per variant before evaluating results — smaller samples produce statistically unreliable p-values and lead to false conclusions
Always define the control (A) as the Remote Config default value — if the fetch fails or returns empty, the user sees the safe, proven experience
When You Need Custom Development
FlutterFlow with Firebase Remote Config handles simple UI variant swaps cleanly. You outgrow it when the experimentation requirements become scientifically rigorous or legally regulated:
- Multi-variant testing (A/B/C/D) with statistical power calculations and early stopping rules — Firebase Remote Config supports multiple conditions but the analytics calculation requires a custom significance engine
- Experiment results must feed Mixpanel or Amplitude with variance-adjusted significance and sequential testing corrections — requires a custom analytics pipeline beyond Supabase event logging
- Legal or compliance requirements mandate an immutable audit log recording exactly which users were in which variant, when the assignment was made, and by which system — standard Supabase RLS + user_experiments is not sufficient for regulated industries
- Personalized variant assignment based on ML model predictions (assigning users more likely to convert to variant B) rather than random percentage — requires a custom assignment service with model inference
RapidDev builds these features for production
Real auth, database, and payments — fixed price, delivered in weeks. You own 100% of the code.
Frequently asked questions
How do I make sure the same user always sees the same variant?
Two layers: Firebase Remote Config caches the assignment locally and returns the same value within the cache TTL (set minimumFetchInterval to Duration(hours: 1)). Supabase user_experiments is the permanent fallback — on each app open, after fetchAndActivate(), write the returned variant to Supabase if no row exists yet; on cache misses, read from Supabase instead of the Remote Config default.
How do I know when I have enough data to declare a winner?
Run a chi-squared test on the 2x2 contingency table: variant A (conversions, non-conversions) vs variant B (conversions, non-conversions). A p-value below 0.05 with at least 100 participants per variant indicates statistical significance. Implement this in a Supabase Edge Function (check_significance) that the admin dashboard calls — never declare a winner on raw conversion counts alone.
Can I test more than two variants at once?
Firebase Remote Config supports multiple conditions per parameter — you can create 'A', 'B', and 'C' conditions for a 3-way split. Add additional Flutter conditional cases for the third variant. The analytics query groups by variant so it handles N variants without changes. Be aware that multi-variant tests require proportionally more participants for statistical significance.
How do I avoid showing the wrong variant during a slow Remote Config fetch?
Await fetchAndActivate() in main() before runApp() completes — the app displays a splash screen or loading indicator while the fetch runs. Set a 5-second timeout on the fetch using Future.timeout(). If the fetch times out, fall back to the Supabase-persisted assignment. This ensures users never see a variant flash before the correct one loads.
Can I run multiple A/B tests at the same time without interference?
Yes — use separate Remote Config parameter keys per experiment (e.g. 'experiment_onboarding_headline', 'experiment_checkout_button_color'). The user_experiments and user_experiment_events tables both carry experiment_id, so analytics are fully separated. Be cautious about interaction effects: if two experiments affect the same user flow, their results may be correlated.
How do I roll back a bad variant without a new app release?
In Firebase Remote Config console, change the percentage condition for variant B to 0% (all users in control). The change propagates within 1 hour (the minimum fetch interval). Users who cached variant B will revert to control on their next fetch. This is why Remote Config exists — zero-release rollback is its primary advantage over hardcoded feature flags.
Do I need to disclose A/B testing to users?
For UX experiments that do not involve pricing, medical advice, or personally sensitive content, disclosure is generally not legally required. GDPR and CCPA apply if the experiment collects personal data — user_experiment_events logs user_id and event data, which is personal data under both frameworks. Include A/B testing in your privacy policy under 'how we improve our service' and ensure user_id is pseudonymized if needed.
Need this feature production-ready?
RapidDev builds a/b testing into real apps — auth, database, payments — at $13K–$25K.
Book a free consultation30-min call. No commitment.