# How to Add A/B Testing to Your App (Copy-Paste Prompts Included)

- Tool: App Features
- Last updated: July 2026

## TL;DR

Mobile A/B testing needs four pieces: Firebase Remote Config for percentage-based variant assignment with local caching, a Supabase table persisting the assignment so it is sticky across sessions and Remote Config cache misses, event logging for conversion tracking, and an admin query or Edge Function that calculates conversion rates per variant. With FlutterFlow this takes 6-10 hours for $0/month — Firebase Remote Config and Supabase free tiers cover everything up to 10,000 users.

## What Mobile A/B Testing Actually Is

A/B testing (also called split testing) silently assigns each user to one of two variants — a control (A) and a variant (B) — on their first app open, then measures which variant drives more of a target conversion event (a purchase, a sign-up, a button tap). The product team sees conversion rates per variant and declares a winner without shipping a new app release. The core mechanics are: assignment must be random and percentage-controlled, the same user must always see the same variant across sessions, conversion events must be tagged with the experiment and variant, and the admin dashboard must calculate statistical significance rather than just raw counts. Firebase Remote Config handles the assignment and percentage rollout; Supabase handles the persistence and event logging.

## Anatomy of the Feature

Six components. Firebase Remote Config handles variant assignment. Supabase handles persistence and analytics. FlutterFlow wires them together with custom action blocks — most of the complexity lives in those blocks.

- **Feature flag / variant assignment** (service): Firebase Remote Config assigns users to variants via percentage rollout conditions. The Flutter firebase_remote_config package (firebase_remote_config: ^5.x) fetches the active experiment config on app start via fetchAndActivate() and caches it locally. The condition uses a user property 'randomization_id' (set from the first 8 characters of the user's ID) to ensure stable percentage splits.
- **Variant renderer** (ui): Flutter conditional widget that reads the experimentVariant value fetched from Remote Config and renders the appropriate UI: if (experimentVariant == 'B') return VariantBWidget() else return ControlWidget(). Variants are stored in a Map<String, Widget> keyed by experiment_id for multi-experiment support.
- **Assignment persistence** (data): Supabase user_experiments table stores user_id, experiment_id, variant, and assigned_at. Written on the user's first app open after Remote Config returns an assignment. On subsequent opens, if Remote Config returns the default value (cache miss), the app reads from Supabase instead — ensuring sticky assignment even when Remote Config is temporarily unavailable.
- **Event tracking** (backend): Supabase user_experiment_events table logs conversion events tagged with experiment_id, variant, event_type (e.g. 'purchase_completed', 'button_tapped'), and metadata JSONB. The analytics query to compare variants: SELECT variant, COUNT(*) as conversions, COUNT(DISTINCT user_id) as participants FROM user_experiment_events WHERE experiment_id = $1 GROUP BY variant.
- **Analytics dashboard (admin)** (ui): A React admin page (built in Lovable or v0 as a companion web tool) or a Supabase Studio custom query showing: participants per variant, conversions per variant, conversion rate (%), and a chi-squared p-value computed in a Supabase Edge Function (check_significance). A p-value below 0.05 indicates statistical significance.
- **Remote Config fetch on cold start** (backend): firebase_remote_config.fetchAndActivate() is called in main() before runApp() completes. Sets minimumFetchInterval: Duration(hours: 1) for development and Duration(hours: 1) for production (balancing freshness vs the 150 fetches/day free quota). The fetch is async and awaited before the first experiment-gated screen is shown.

## Data model

Three tables: experiments catalogue, per-user variant assignments, and conversion events. Paste this into the Supabase SQL Editor and run it before building the FlutterFlow integration:

```sql
create table public.experiments (
  id uuid primary key default gen_random_uuid(),
  slug text not null unique,
  description text,
  status text not null default 'active' check (status in ('active', 'paused', 'completed')),
  started_at timestamptz not null default now(),
  ended_at timestamptz
);

create table public.user_experiments (
  user_id uuid references auth.users(id) on delete cascade not null,
  experiment_id uuid references public.experiments(id) on delete cascade not null,
  variant text not null,
  assigned_at timestamptz not null default now(),
  primary key (user_id, experiment_id)
);

create table public.user_experiment_events (
  id uuid primary key default gen_random_uuid(),
  user_id uuid references auth.users(id) on delete cascade not null,
  experiment_id uuid references public.experiments(id) on delete cascade not null,
  variant text not null,
  event_type text not null,
  metadata jsonb,
  created_at timestamptz not null default now()
);

-- RLS
alter table public.user_experiments enable row level security;
alter table public.user_experiment_events enable row level security;

create policy "Users insert own assignment"
  on public.user_experiments for insert
  with check (auth.uid() = user_id);

create policy "Users read own assignment"
  on public.user_experiments for select
  using (auth.uid() = user_id);

create policy "Users insert own events"
  on public.user_experiment_events for insert
  with check (auth.uid() = user_id);

-- Admin reads all via service_role key in Edge Function
-- (no public RLS policy for admin reads — use service_role)

-- Indexes for analytics queries
create index exp_events_experiment_idx
  on public.user_experiment_events (experiment_id, variant, event_type);

create index exp_events_user_idx
  on public.user_experiment_events (user_id, experiment_id, created_at desc);
```

The admin analytics query (conversion rate per variant) should use the Supabase service_role key in a server-side Edge Function or Next.js Server Action — never expose service_role to the Flutter client. RLS allows users to insert only their own rows.

## Build paths

### Flutterflow — fit 3/10, 6-10 hours

The primary path for mobile A/B testing: Firebase Remote Config percentage rollout is wired via a custom action block, variant rendering uses FlutterFlow's Conditional Widget, and Supabase handles assignment persistence and event logging.

1. Create a FlutterFlow project with Firebase and Supabase both connected; run the SQL schema from this page in the Supabase SQL Editor
2. Set a randomization_id user property early in the app flow: add a Custom Action in the app initialization (before the first A/B-gated screen) that calls FirebaseAnalytics.instance.setUserProperty(name: 'randomization_id', value: currentUser.uid.substring(0, 8))
3. Add a Custom Action for fetchRemoteConfig: call await FirebaseRemoteConfig.instance.setConfigSettings(RemoteConfigSettings(minimumFetchInterval: Duration(hours: 1))); await FirebaseRemoteConfig.instance.fetchAndActivate(); return FirebaseRemoteConfig.instance.getString('experiment_onboarding_step2')
4. In the experiment-gated screen, add a Conditional Widget: if the page state variable experimentVariant equals 'B', show VariantBColumn; else show ControlColumn
5. Add another Custom Action for persistAssignment that upserts into Supabase user_experiments; add a logConversionEvent Custom Action that inserts into user_experiment_events with event_type, experiment_id, and variant
6. Add a Custom Action for fallback: after fetchRemoteConfig, if the returned value equals the Remote Config default ('A'), query Supabase user_experiments for an existing assignment; use the stored variant if found

Limitations:

- Complex variant logic — multi-step flow branching, navigation changes between variants — requires significant Dart custom action code that pushes FlutterFlow to its visual-builder limits
- The analytics dashboard must be built separately as a web app (Lovable or v0) or queried directly in Supabase Studio — FlutterFlow is not suited for tabular admin analytics
- FlutterFlow Pro ($70/mo) is required for code export if the custom action blocks are extensive; free plan runs in the FlutterFlow preview app only

### Lovable — fit 1/10, not recommended

Lovable generates web React apps and cannot build or test a mobile Flutter A/B test. A Lovable-built web admin dashboard showing experiment results is a valid companion tool, but the mobile experiment itself must be built in FlutterFlow or custom Flutter.

1. Build the experiment analytics admin dashboard in Lovable: a React page that calls a Supabase Edge Function returning conversion rates per variant for each active experiment
2. Lovable can generate the chi-squared significance indicator and a table showing participant counts, conversion rates, and p-values per variant
3. Connect Lovable Cloud to the same Supabase project used by the FlutterFlow app — the admin dashboard reads from the same user_experiment_events table

Starter prompt:

```
Build an experiment analytics admin dashboard. Connect to Supabase using the service_role key (store it in Lovable Cloud Secrets, never in client code). Fetch from user_experiment_events grouped by experiment_id and variant: SELECT variant, COUNT(*) AS conversions, COUNT(DISTINCT user_id) AS participants FROM user_experiment_events WHERE experiment_id = :id GROUP BY variant. Display a table per experiment showing: experiment slug, variant A conversion rate (%), variant B conversion rate (%), participant counts per variant, and a chi-squared p-value (p < 0.05 = statistically significant). Add a dropdown to select the active experiment from the experiments table (status = 'active'). Show a 'Declare Winner' button that calls a Supabase Edge Function (declare_winner) setting experiments.status = 'completed'. Handle loading state, empty state when no events exist yet, and error state when Supabase returns an error.
```

Limitations:

- Lovable cannot build the mobile Flutter app — it generates web React code only
- The mobile experiment assignment, Remote Config fetch, and conversion event logging must all be built in FlutterFlow or custom Flutter code
- Using Lovable as the admin dashboard and FlutterFlow as the mobile client is a valid split-tool architecture but requires careful coordination of experiment_id values between the two codebases

### Custom — fit 5/10, 2-4 weeks

The correct path for production-grade experimentation: full control over assignment algorithm, multi-variant (A/B/C) testing, statistical power calculations, and early stopping rules — the only path when experiments must meet a scientific or regulatory bar.

1. Build a server-side assignment service (Supabase Edge Function) that receives user_id and experiment_id, applies the percentage split using a deterministic hash (SHA-256 of user_id + experiment_salt), writes to user_experiments, and returns the assigned variant
2. Replace Firebase Remote Config with the server-side assignment — the Flutter app calls the Edge Function on cold start and caches the response in SharedPreferences as a fallback
3. Implement Bayesian significance calculation in the analytics Edge Function: compute posterior probability that variant B's true conversion rate exceeds A's, accounting for sample size
4. Build an experiment management dashboard with early stopping rules: automatically pause an experiment and notify the team via Slack when the p-value drops below 0.05 with at least 100 participants per variant

Limitations:

- Server-side assignment adds 100-300ms to app cold start — use aggressive SharedPreferences caching to avoid this on warm starts
- Bayesian significance calculation is non-trivial to implement correctly — use a well-tested statistics library rather than a hand-rolled formula

## Gotchas

- **User sees variant B one session and variant A the next** — Firebase Remote Config cache expired between sessions and fetchAndActivate() returned the default value instead of the cached variant. Without a Supabase persistence layer as fallback, the user is assigned to whichever variant is the Remote Config default — typically 'A'. This contaminates the experiment with users who have seen both variants. Fix: After fetchAndActivate(), check whether the returned value differs from the previous session's assignment (stored in SharedPreferences). If it matches the default and differs from the stored value, read the variant from Supabase user_experiments instead. Call fetchAndActivate() in main() with minimumFetchInterval: Duration(hours: 1) to minimise cache misses.
- **Conversion events double-count because of app backgrounding** — The event logging function fires in both the AppLifecycleState.resumed callback and the initial screen build. When the user opens the app from background (not cold start), both triggers fire — inserting two conversion event rows for the same session, inflating the conversion count for variant B or A. Fix: Guard event logging with a session-scoped static flag: static bool _conversionLogged = false in the screen's State class. Set it to true immediately after the first insert call and check the flag before every subsequent logging attempt. Reset the flag only on actual cold starts (in initState when the screen is created fresh).
- **All users assigned to control (variant A)** — Firebase Remote Config percentage condition uses the user property 'randomization_id' to split users, but the randomization_id is never set in the Flutter app. Without the user property, every user falls into the default condition — which returns the default value 'A'. The variant B group always has zero participants. Fix: Set the randomization_id user property early in the app flow, before the Remote Config fetch: await FirebaseAnalytics.instance.setUserProperty(name: 'randomization_id', value: currentUser.uid.substring(0, 8)). This must happen before or concurrently with fetchAndActivate() for the condition to evaluate correctly.
- **Admin dashboard shows 0 conversions for variant B** — The event_type string logged by the Flutter code is 'button_tap' but the dashboard query filters on 'button_tapped'. This kind of naming mismatch is invisible during development because both strings are syntactically valid — the insert succeeds but the dashboard query returns no rows for variant B. Fix: Define a constants file in Flutter: class ExperimentEvents { static const String buttonTapped = 'button_tapped'; static const String purchaseCompleted = 'purchase_completed'; } Reference these constants in both the Supabase insert call and the admin dashboard SQL. Never hardcode event type strings in two separate places.
- **Remote Config fetch blocked on iOS in background** — iOS Background App Refresh is disabled by the user or the OS. Remote Config fetch in a background task silently fails, and the user opens the app with a stale or missing variant assignment. In the worst case, they see the wrong variant because the Supabase fallback was not implemented. Fix: Do not rely on background fetches for experiment assignment. Always call fetchAndActivate() in the foreground app open lifecycle (in main() before runApp()). Handle the async call gracefully: await with a 5-second timeout, then fall back to the Supabase-persisted assignment if the fetch times out.

## Best practices

- Always write the variant assignment to Supabase immediately after Remote Config returns it — the Supabase row is your source of truth when the Remote Config cache expires
- Set a randomization_id user property before fetchAndActivate() — without it, Remote Config percentage conditions cannot split users and everyone gets the default variant
- Define all event type strings in a constants file and import from it in every location where events are logged — naming mismatches between the Flutter client and the admin SQL query are the single most common cause of missing conversion data
- Call fetchAndActivate() in main() before runApp() and await it with a timeout — never defer experiment assignment to a later screen, because the user may reach the experiment-gated screen before the fetch completes
- Set minimumFetchInterval to Duration(hours: 1) in both development and production — the Remote Config free tier allows 150 unique active user fetches per day, and hourly fetching from a 10,000-user app easily exceeds this if not cached
- Guard conversion event logging with a session-scoped boolean flag to prevent double-counting from app backgrounding and resuming
- Require at least 100 participants per variant before evaluating results — smaller samples produce statistically unreliable p-values and lead to false conclusions
- Always define the control (A) as the Remote Config default value — if the fetch fails or returns empty, the user sees the safe, proven experience

## Frequently asked questions

### How do I make sure the same user always sees the same variant?

Two layers: Firebase Remote Config caches the assignment locally and returns the same value within the cache TTL (set minimumFetchInterval to Duration(hours: 1)). Supabase user_experiments is the permanent fallback — on each app open, after fetchAndActivate(), write the returned variant to Supabase if no row exists yet; on cache misses, read from Supabase instead of the Remote Config default.

### How do I know when I have enough data to declare a winner?

Run a chi-squared test on the 2x2 contingency table: variant A (conversions, non-conversions) vs variant B (conversions, non-conversions). A p-value below 0.05 with at least 100 participants per variant indicates statistical significance. Implement this in a Supabase Edge Function (check_significance) that the admin dashboard calls — never declare a winner on raw conversion counts alone.

### Can I test more than two variants at once?

Firebase Remote Config supports multiple conditions per parameter — you can create 'A', 'B', and 'C' conditions for a 3-way split. Add additional Flutter conditional cases for the third variant. The analytics query groups by variant so it handles N variants without changes. Be aware that multi-variant tests require proportionally more participants for statistical significance.

### How do I avoid showing the wrong variant during a slow Remote Config fetch?

Await fetchAndActivate() in main() before runApp() completes — the app displays a splash screen or loading indicator while the fetch runs. Set a 5-second timeout on the fetch using Future.timeout(). If the fetch times out, fall back to the Supabase-persisted assignment. This ensures users never see a variant flash before the correct one loads.

### Can I run multiple A/B tests at the same time without interference?

Yes — use separate Remote Config parameter keys per experiment (e.g. 'experiment_onboarding_headline', 'experiment_checkout_button_color'). The user_experiments and user_experiment_events tables both carry experiment_id, so analytics are fully separated. Be cautious about interaction effects: if two experiments affect the same user flow, their results may be correlated.

### How do I roll back a bad variant without a new app release?

In Firebase Remote Config console, change the percentage condition for variant B to 0% (all users in control). The change propagates within 1 hour (the minimum fetch interval). Users who cached variant B will revert to control on their next fetch. This is why Remote Config exists — zero-release rollback is its primary advantage over hardcoded feature flags.

### Do I need to disclose A/B testing to users?

For UX experiments that do not involve pricing, medical advice, or personally sensitive content, disclosure is generally not legally required. GDPR and CCPA apply if the experiment collects personal data — user_experiment_events logs user_id and event data, which is personal data under both frameworks. Include A/B testing in your privacy policy under 'how we improve our service' and ensure user_id is pseudonymized if needed.

---

Source: https://www.rapidevelopers.com/app-features/a-b-testing
© RapidDev — https://www.rapidevelopers.com/app-features/a-b-testing