# How to Integrate Retool with TensorFlow Serving

- Tool: Retool
- Difficulty: Advanced
- Time required: 30 minutes
- Last updated: April 2026

## TL;DR

Connect Retool to TensorFlow Serving using a REST API Resource targeting your self-hosted TensorFlow Serving instance's HTTP endpoints. Build model management panels that submit inference requests, view model metadata and version status, compare prediction outputs across versions, and monitor serving performance — giving your ML team an internal tool for testing and managing deployed TensorFlow models.

## Build a TensorFlow Model Management and Inference Panel in Retool

TensorFlow Serving makes it straightforward to deploy trained TensorFlow models with a production-grade serving infrastructure, but it provides no native UI for testing models, comparing versions, or monitoring which model version is serving requests. ML teams typically use command-line curl requests or custom scripts to test deployed models — a tedious process that becomes a bottleneck when data scientists, QA engineers, and product managers all need to test the same model. A Retool app connected to TensorFlow Serving's REST API provides an accessible internal tool for the entire team.

TensorFlow Serving's REST API runs on port 8501 by default and exposes endpoints for three operations: checking model status and version metadata, running predictions using the predict endpoint, and running classification or regression requests for models that use the TensorFlow Serving SignatureDef format. The API accepts JSON-formatted tensor inputs and returns JSON-formatted prediction outputs. Retool's REST API Resource handles the HTTP communication, and JavaScript transformers reshape the raw tensor output into human-readable results that non-ML team members can interpret in Tables and Charts.

This integration is most valuable for teams in the model validation and monitoring phase of the ML lifecycle. Data scientists can test new model versions against known inputs directly from the Retool panel without writing code. QA engineers can submit batches of test cases and verify expected outputs. Product managers can see model confidence scores and prediction distributions for business-relevant input examples. For teams running multiple model versions simultaneously (A/B testing or canary deployments), Retool can submit identical inputs to multiple serving endpoints and display results side by side for direct comparison.

## Before you start

- A running TensorFlow Serving instance with at least one model loaded (accessible from Retool's network on port 8501 for REST)
- Knowledge of your model's name as registered in TensorFlow Serving and its expected input tensor names and shapes
- For self-hosted Retool: network connectivity between Retool and TensorFlow Serving (same VPC or whitelist Retool IPs)
- For Retool Cloud: TensorFlow Serving endpoint must be publicly accessible or exposed via a VPN/tunnel
- Basic understanding of TensorFlow model input/output tensor format and the TensorFlow Serving REST API structure

## Step-by-step guide

### 1. Verify TensorFlow Serving REST API is accessible

Before configuring Retool, confirm your TensorFlow Serving instance is running and the REST API is accessible. TensorFlow Serving exposes its REST API on port 8501 by default when launched with the --rest_api_port=8501 flag. The gRPC API runs on port 8500 — Retool uses REST, not gRPC, so port 8501 is the relevant one. To verify accessibility, you should be able to reach your server at http://your-tf-serving-host:8501/v1/models/your_model_name and receive a JSON response showing the model's version and state. Common deployment scenarios include TensorFlow Serving running as a Docker container (docker run -p 8501:8501 tensorflow/serving:latest), as a Kubernetes service with a LoadBalancer or NodePort, or as a direct process on a GPU-equipped VM. For Retool Cloud to reach a self-hosted TensorFlow Serving instance, the serving endpoint must be either publicly accessible (with appropriate security measures), reachable via a VPN tunnel, or accessible through a reverse proxy that adds authentication headers. For self-hosted Retool deployments in the same VPC as TensorFlow Serving, you can use private IP addresses and there is no need to expose the serving endpoint to the internet — this is the recommended architecture for production deployments. Document your TensorFlow Serving URL, model name, and default serving version before proceeding.

**Expected result:** Your TensorFlow Serving endpoint responds at http://your-host:8501/v1/models/your_model_name with a JSON object showing model version_status. Retool can reach this URL from its network.

### 2. Create a REST API Resource pointing to TensorFlow Serving

In Retool, navigate to the Resources tab from your organization's homepage sidebar. Click '+ Create new', scroll the connector list, and select 'REST API'. On the configuration screen, set the Name to 'TensorFlow Serving' or a more specific name like 'TF Serving (Production)' if you have multiple environments. Set the Base URL to your TensorFlow Serving REST API URL — for example, http://tf-serving.internal:8501 for an internal deployment, or https://tf-serving.yourcompany.com:8501 if accessed through a reverse proxy with TLS. Leave Authentication as 'No authentication' unless your TensorFlow Serving deployment sits behind a reverse proxy (like nginx or Envoy) that requires an API key or Bearer token — if so, add the appropriate authentication headers here. Add a header with key Content-Type and value application/json, as all TensorFlow Serving REST requests must send JSON bodies. If your reverse proxy adds authentication, configure it in the Headers section or select the appropriate auth type. Click 'Test connection' — if the base URL is reachable, it should succeed even without a valid model path in the URL. Click 'Save resource'. You are now ready to build queries for model status checks and inference requests. For deployments with multiple TensorFlow Serving instances (staging and production), create separate resources with distinct base URLs so your Retool app can target either environment.

**Expected result:** A REST API Resource named 'TensorFlow Serving' with the correct base URL is saved in Retool. The resource is available in the query editor for building inference and status queries.

### 3. Query model status and version metadata

Create a query to check the status and available versions of a deployed TensorFlow model. In the Code panel, click '+ New query', select your TensorFlow Serving resource, and name the query getModelStatus. Set the Method to GET and the URL path to /v1/models/{{ modelNameSelect.value || 'my_model' }} — replace 'my_model' with your default model name, or reference a Select component that lets users choose which model to inspect. Click 'Run query'. TensorFlow Serving returns a JSON object with a model_version_status array, where each entry has a version (integer), state ('AVAILABLE', 'LOADING', 'UNLOADING', or 'START'), and a status object with an error code and error message. For a healthy model with multiple versions, you might see version 1 in state AVAILABLE and version 2 in state AVAILABLE with the default serving version being the highest-numbered one. Create a transformer that flattens the version status array into a clean Table format and adds a derived 'is_default' field marking the highest available version. Drag a Table component and bind it to {{ getModelStatus.data }}. To get full model metadata including the model's signature definition (input and output tensor names and shapes), create a second query getModelMetadata with path /v1/models/{{ modelNameSelect.value }}/metadata. The metadata endpoint returns the full SignatureDef, which documents exactly which tensor names and data types your inference requests need to use.

```
// Transformer: flatten TF Serving model version status
const versions = data.model_version_status || [];
const maxVersion = Math.max(...versions.map(v => parseInt(v.version)));

return versions.map(v => ({
  version: v.version,
  state: v.state,
  is_default: parseInt(v.version) === maxVersion,
  error_code: v.status?.error_code || 'OK',
  error_message: v.status?.error_message || ''
}));
```

**Expected result:** The getModelStatus query returns version status information. The Table shows each model version with its availability state and a 'Default' badge on the highest available version.

### 4. Build an inference request query

The core capability of the integration is submitting inference requests from Retool to TensorFlow Serving. Create a query named runInference. Select your TensorFlow Serving resource, set the Method to POST, and set the URL path to /v1/models/{{ modelNameSelect.value }}:predict — or to target a specific version, use /v1/models/{{ modelNameSelect.value }}/versions/{{ versionSelect.value }}:predict. In the Body section, select JSON and enter the inference request body. TensorFlow Serving's predict API requires the inputs to be formatted as a 'instances' array (for row-based format) or as 'inputs' with named tensor arrays (for column-based format). The format depends on how your model was exported. For the row-based format, each element in 'instances' is a single inference request. For image models, this might be a 2D array of pixel values. For text models, this might be a string or tokenized integer array. Reference a TextInput or TextArea component for the input values — users enter their test inputs in the Retool UI and the query formats them into the correct tensor structure. Add a tip to run the query on button click (not automatically) to prevent accidental or excessive inference calls. Configure the query to trigger on a Button component's Click event. Add a Success event handler that stores the result in a state variable for display.

```
{
  "instances": [
    {{ JSON.parse(inputTextArea.value || '[[0.0, 0.0, 0.0]]') }}
  ]
}
```

**Expected result:** Clicking the 'Run Inference' button sends a POST request to the predict endpoint. The query returns a JSON response with a 'predictions' array containing the model's output for the submitted input.

### 5. Transform and display prediction outputs

Raw TensorFlow Serving prediction outputs are JSON arrays that need context to be interpretable. For classification models, predictions are typically arrays of class probabilities — you need to map the highest probability to its class label. For regression models, predictions are numeric values that may need scaling or unit annotation. Create a JavaScript transformer on your runInference query that interprets the raw predictions array and formats it for display. For a classification model, build a label array matching your model's output classes, find the index of the maximum probability, and format each class with its probability as a percentage. Display the top prediction prominently using a Text component and show the full probability distribution in a Chart component (type: Bar, with class labels on the x-axis and probability percentages on the y-axis). For model version comparison, duplicate the inference query and point it at a second version URL, run both queries in parallel when the user clicks 'Compare', and display results side by side using two Stat components showing the top prediction from each version and a Table showing the full probability distributions for both versions with a percentage difference column. For teams managing complex TensorFlow Serving deployments with multiple model types, canary routing, and automated accuracy monitoring, RapidDev's team can help architect the complete Retool MLOps dashboard.

```
// Transformer: interpret classification model predictions
const predictions = data.predictions || [];

// Define your model's class labels — update to match your model
const CLASS_LABELS = [
  'Class A', 'Class B', 'Class C', 'Class D'
];

if (predictions.length === 0) return [];

const probabilities = predictions[0]; // First instance
const maxIndex = probabilities.indexOf(Math.max(...probabilities));

const result = CLASS_LABELS.map((label, i) => ({
  rank: i + 1,
  class: label,
  probability: probabilities[i],
  probability_pct: `${(probabilities[i] * 100).toFixed(2)}%`,
  is_top_prediction: i === maxIndex
}));

// Sort by probability descending
return result.sort((a, b) => b.probability - a.probability).map((r, i) => ({
  ...r,
  rank: i + 1
}));
```

**Expected result:** After running an inference request, the prediction transformer outputs a ranked list of classes with probability percentages. A Bar Chart shows the full probability distribution. A Stat component highlights the top predicted class and its confidence score.

## Best practices

- Use TensorFlow Serving's /v1/models/{model}/metadata endpoint to document your model's input tensor names, shapes, and data types in your Retool app's UI — display this as a schema reference panel so team members know how to format inference inputs correctly.
- Never run inference queries automatically (on page load or on component change) — configure them to trigger only on explicit button clicks to avoid accidental repeated inference calls that generate unnecessary GPU compute costs.
- For production model testing, create separate Retool Resources pointing to staging and production TensorFlow Serving instances, and add a prominent environment indicator (red 'PRODUCTION' badge) to prevent accidental production requests during testing.
- Store inference test cases (inputs and expected outputs) in a Retool Database table or your internal database, so QA engineers can run regression tests by selecting from saved test cases rather than manually re-entering inputs each session.
- For classification models, store class label arrays in your Retool app's configuration or a database table rather than hard-coding them in transformers — this allows updating labels without modifying transformer code when model classes change between versions.
- Use Retool Workflows to run scheduled batch inference jobs on holdout datasets and track accuracy metrics over time in a database — this creates a model monitoring capability alongside the interactive testing panel.
- For self-hosted TensorFlow Serving deployments, use Retool's self-hosted deployment in the same VPC rather than exposing the serving endpoint to the internet — this eliminates network latency, avoids authentication complexity, and keeps inference requests entirely within your private network.

## Use cases

### Build an inference testing panel

Your data science team deploys a new version of a text classification model to TensorFlow Serving and needs to test it with sample inputs before promoting it to production traffic. A Retool app with a text input field for entering model inputs, a query that formats and sends the inference request to TensorFlow Serving's predict endpoint, and a results display showing the predicted class and confidence scores — gives the entire team a shareable testing tool without individual curl commands.

Prompt example:

```
Build a model testing panel with a TextArea for entering inference input (as JSON), a 'Run Inference' button that POSTs to TensorFlow Serving's /v1/models/my_model:predict endpoint, and a results section showing the top predicted class and confidence score. Include a Table showing all class probabilities sorted by confidence.
```

### Create a model version management dashboard

Your MLOps team manages five different TensorFlow models, each with multiple active versions. They need a panel showing which version of each model is currently serving (the default version), when it was deployed, and what versions are available for testing. The Retool app queries TensorFlow Serving's model status endpoint for each model, displays version metadata, and provides buttons to run quick inference tests against any specific version.

Prompt example:

```
Build a model inventory dashboard that queries the status endpoint for each of your TensorFlow Serving models (/v1/models/{model_name}), displays model name, available versions, serving state, and created timestamp for each. Add a model selector dropdown and a version picker to run inference against any specific model version.
```

### Build a batch inference and evaluation tool

Your QA team needs to submit a batch of 50 test inputs to a deployed classification model and verify that predicted labels match expected outputs. A Retool app with a file upload for test CSV data, a JavaScript query that splits it into individual inference requests and sends them to TensorFlow Serving in sequence, and a results Table showing input, expected label, predicted label, confidence, and pass/fail status — automates the model validation process that previously required a custom Python script.

Prompt example:

```
Build a batch evaluation panel with a file upload for a CSV of test cases (columns: input_text, expected_label). Run inference for each row against a TensorFlow Serving predict endpoint and display results in a Table showing input, expected, predicted, confidence, and a pass/fail indicator. Show overall accuracy as a Stat component.
```

## Troubleshooting

### Query returns connection refused or network error

Cause: Retool cannot reach the TensorFlow Serving REST API endpoint. The most common causes are the serving process not running, the wrong port (8500 is gRPC, 8501 is REST), or network/firewall rules blocking the connection from Retool's IP addresses.

Solution: Verify TensorFlow Serving is running with --rest_api_port=8501. For Retool Cloud, ensure port 8501 is open in your security group/firewall rules and that your serving host is publicly accessible or reachable via VPN. For self-hosted Retool, verify the Resource base URL uses the private IP of the serving host within the same VPC. Run a test request from your local machine using curl to confirm the endpoint is responsive before testing from Retool.

### Predict endpoint returns 400 Bad Request with 'Could not decode JSON input'

Cause: The inference request body JSON is malformed, uses the wrong format (instances vs inputs), uses incorrect tensor names, or contains tensor shapes that do not match the model's expected input shape.

Solution: First check your model's expected input format by querying the /v1/models/{model_name}/metadata endpoint, which documents the exact input tensor names and dtypes in the signature_def. Use the 'instances' key for row-based format (single JSON array per instance) or the 'inputs' key with named tensors for column-based format. Verify tensor shapes match the model's expected dimensions — a shape mismatch returns a clear error message in the response body describing the expected shape.

```
// Row-based format (most common):
{"instances": [[1.0, 2.0, 3.0]]}

// Column-based format (for named inputs):
{"inputs": {"input_tensor_name": [[1.0, 2.0, 3.0]]}}
```

### Model status returns 404 Not Found

Cause: The model name in the URL path does not match the model name registered in TensorFlow Serving when it was started. TensorFlow Serving model names are case-sensitive and must exactly match the --model_name flag or model_config_list entry used at startup.

Solution: Verify the exact model name by checking the TensorFlow Serving startup command or model configuration file. The model name in Retool's query path (/v1/models/{model_name}) must exactly match the name provided to TensorFlow Serving with --model_name=your_model_name. Check server logs for 'Aspiring servable' log entries that show loaded model names.

### Inference requests succeed but predictions array is all zeros or nonsensical

Cause: The model is loaded and serving, but the input tensor format is incorrect — tensor values are the wrong data type, the wrong shape, or in the wrong normalization range expected by the model.

Solution: Check the model's expected input dtype from the metadata endpoint (int32, float32, string, etc.). Ensure input values are within the expected normalization range (e.g., 0-1 for normalized images, not 0-255). For text models, verify whether the model expects raw strings, integer token IDs, or pre-computed embeddings. Test with a known-good input that produced correct results during training to verify the serving setup before debugging the Retool formatting.

## Frequently asked questions

### Does TensorFlow Serving have built-in authentication?

No. TensorFlow Serving has no built-in authentication mechanism. To secure your serving endpoint, place it behind a reverse proxy (nginx, Envoy, or a cloud load balancer) that enforces authentication — API keys, Bearer tokens, or IP-based access control. Configure the authentication headers in your Retool REST API Resource so all queries include the required credentials.

### Can Retool connect to TensorFlow Serving via gRPC?

No. Retool's query system supports HTTP-based APIs. TensorFlow Serving's REST API (port 8501) is the correct interface for Retool integration. TensorFlow Serving's gRPC interface (port 8500) is not compatible with Retool's Resource system, though you could build a thin HTTP proxy that translates REST requests to gRPC if needed for latency-sensitive workloads.

### How do I test multiple model versions side by side in Retool?

Create two inference queries — one using the path /v1/models/{model}/versions/1:predict and another using /v1/models/{model}/versions/2:predict. Set both queries to trigger on the same Button click using multiple event handlers. Use a Retool Container component split into two columns, showing the prediction results from each query side by side. A comparison transformer can compute the difference between confidence scores for the top prediction across both versions.

### What is the maximum input size I can send to TensorFlow Serving through Retool?

TensorFlow Serving's default maximum request size is 4 MB for REST API requests. Retool's Resource queries have a 100 MB response size limit and a configurable timeout (default 10 seconds, up to 10 minutes for Cloud). For large batch inference requests or high-dimensional input tensors, consider sending smaller batches (10-20 instances per request) and chaining multiple queries using a JavaScript query with Promise.all() rather than sending a single very large request.

---

Source: https://www.rapidevelopers.com/retool-integrations/tensorflow
© RapidDev — https://www.rapidevelopers.com/retool-integrations/tensorflow