# How to use Bubble.io to run backend data processing jobs: Step-by-Step Guide

- Tool: Bubble
- Difficulty: Beginner
- Time required: 20-25 min
- Compatibility: All Bubble plans (Growth plan+ for backend workflows)
- Last updated: March 2026

## TL;DR

Bubble's backend workflows let you process large datasets in the background without blocking the user interface. Using Schedule API Workflow on a List, you can iterate through thousands of records, perform calculations or updates, and avoid timeout errors by chunking work into manageable batches. This tutorial covers setting up batch processing, recursive workflows, and monitoring job progress.

## Overview: Running Backend Data Processing Jobs in Bubble

When you need to process hundreds or thousands of database records — recalculating scores, sending bulk emails, cleaning data, or generating reports — you cannot do it in a frontend workflow without timing out. This tutorial shows how to use Bubble's backend workflows to process data in the background, chunk large operations into batches, and track progress.

## Before you start

- A Bubble app with data that needs batch processing
- Understanding of backend workflows (Settings → API tab)
- Familiarity with Data Types and Workflows
- Growth plan or higher for scheduled backend workflows

## Step-by-step guide

### 1. Enable and create a backend workflow

Go to Settings → API tab and check Enable Workflow API. Then go to the Workflow tab, click the pages dropdown, and select Backend workflows. Click New API Workflow. Name it process-single-record. Add a parameter called record_id of type text (or the specific data type you are processing). Inside the workflow, add actions to process a single record: for example, Make changes to a thing with calculated field updates, or call an external API for each record.

> Pro tip: Design backend workflows to process ONE record at a time. Use Schedule API Workflow on a List to call this workflow for each record in the batch.

**Expected result:** A backend workflow exists that can process a single record when called with its ID.

### 2. Set up batch processing with Schedule API Workflow on a List

Create a frontend workflow trigger (a button or scheduled event). In this workflow, do a search for the records you want to process (e.g., Do a search for Users where needs_update = yes). Add the action Schedule API Workflow on a List. Select your process-single-record workflow. Set the list to your search results. Map the record_id parameter to the current item. Bubble will queue one workflow execution per item in the list, processing them sequentially in the background.

**Expected result:** A batch job is triggered that schedules individual processing for each record in the list.

### 3. Implement chunking for very large datasets

For datasets larger than 10,000 records, process in chunks to avoid memory issues. Create a backend workflow called process-chunk with a parameter offset (number). Search for records with a :items from offset :items until offset+100 constraint. Process this chunk using Schedule API Workflow on a List. At the end, check if there are more records: if Do a search count > offset + 100, schedule process-chunk again with offset = offset + 100. Add a termination condition: if the search returns empty results, do not reschedule.

**Expected result:** Large datasets are processed in manageable chunks of 100 records each.

### 4. Add progress tracking

Create a ProcessingJob data type with fields: name (text), total_records (number), processed_count (number), status (text: Queued, Running, Completed, Failed), started_at (date), completed_at (date), error_message (text). Before starting the batch, create a ProcessingJob record with the total count. In your process-single-record workflow, increment the processed_count by 1 (Make changes to ProcessingJob). Display progress on an admin page: processed_count / total_records * 100 as a percentage bar.

**Expected result:** Job progress is tracked in real time and visible on an admin dashboard.

### 5. Handle errors and implement retry logic

In your process-single-record workflow, use the Include errors in response option in API Connector calls. If an action fails, log the error by creating an ErrorLog record with the record ID, error message, and timestamp. For retries, create a separate backend workflow called retry-failed that searches for ErrorLog records and re-processes them. Schedule this to run after the main batch completes. For complex data processing pipelines with error recovery and monitoring, RapidDev can help design robust backend architectures.

**Expected result:** Errors are logged rather than stopping the entire batch, and failed records can be retried.

## Complete code example

File: `Workflow summary`

```text
BACKEND DATA PROCESSING WORKFLOW SUMMARY
==========================================

DATA TYPES:
  ProcessingJob: name, total_records, processed_count,
    status, started_at, completed_at, error_message
  ErrorLog: job (ProcessingJob), record_id, error_message,
    timestamp, is_retried

BACKEND WORKFLOW: process-single-record
  Parameters: record_id (text), job_id (text)
  1. Find the record by ID
  2. Perform processing (calculations, API calls, updates)
  3. Increment ProcessingJob's processed_count +1
  4. On error: Create ErrorLog, continue

FRONTEND TRIGGER:
  1. Create ProcessingJob (status=Running, total=count)
  2. Search for records to process
  3. Schedule API Workflow on a List
     Workflow: process-single-record
     List: search results
     Map: record_id = Current item's unique id
  4. Display progress bar on admin page

CHUNKED PROCESSING:
  Backend: process-chunk (offset parameter)
  1. Search records :items from offset :items until offset+100
  2. Schedule process-single-record for each
  3. If more records exist: Schedule process-chunk (offset+100)
  4. If no more records: Update job status=Completed

RETRY LOGIC:
  Backend: retry-failed
  1. Search ErrorLog where is_retried = no
  2. For each: re-run process-single-record
  3. Update ErrorLog is_retried = yes
```

## Common mistakes

- **Processing all records in a single frontend workflow** — Frontend workflows have a timeout limit. Processing thousands of records will cause the workflow to fail mid-execution. Fix: Always use backend workflows for batch processing. They run server-side and are not affected by browser timeouts.
- **Not adding a termination condition to recursive workflows** — Without a termination condition, recursive workflows run infinitely, consuming workload units and potentially taking the app offline Fix: Always check if there are remaining records before scheduling the next iteration. Stop when the search returns empty results.
- **Searching the full dataset in every chunk iteration** — Without offset/pagination, each chunk re-processes records that were already handled Fix: Use offset-based pagination or mark records as processed (add a is_processed flag) so each chunk only handles new records.

## Best practices

- Design backend workflows to process one record at a time for reliability
- Use Schedule API Workflow on a List for batch processing
- Chunk large datasets (10,000+) into batches of 100-500 records
- Always include a termination condition in recursive workflows
- Track job progress with a ProcessingJob data type
- Log errors instead of stopping the entire batch
- Implement retry logic for failed records
- Monitor workload unit consumption for batch jobs in Settings → Metrics

## Frequently asked questions

### How many records can Schedule API Workflow on a List handle?

It works reliably with lists up to about 10,000 items. For larger lists, use chunking with offset-based pagination to process in batches.

### How fast does Bubble process backend workflows?

Bubble processes approximately 100 database rows per second. A batch of 10,000 records takes roughly 100 seconds. Performance varies based on workflow complexity.

### Will batch processing consume workload units?

Yes. Each workflow execution and database operation costs workload units. A batch processing 10,000 records might use 5,000-20,000 WUs depending on complexity. Monitor usage in Settings → Metrics.

### Can I cancel a running batch job?

There is no built-in cancel button. As a workaround, add a check at the start of each process-single-record workflow: if the ProcessingJob's status is Cancelled, skip processing. Then set the status to Cancelled from the admin page.

### Can I run multiple batch jobs simultaneously?

Yes, but be cautious. Multiple simultaneous jobs compete for the same workload unit allocation and may slow each other down. Process one large job at a time for best performance.

### Can RapidDev help with complex data processing?

Yes. RapidDev can help design efficient data processing pipelines, optimize batch operations for workload units, and implement advanced patterns like parallel processing and external compute offloading.

---

Source: https://www.rapidevelopers.com/bubble-tutorial/use-bubble-run-backend-data-processing-jobs
© RapidDev — https://www.rapidevelopers.com/bubble-tutorial/use-bubble-run-backend-data-processing-jobs