Testing

Mobile Testing CI/CD: Firebase Test Lab for Android and iOS

HelpMeTest

HelpMeTest

— 7 min read
Mobile Testing CI/CD: Firebase Test Lab for Android and iOS

Firebase Test Lab runs your Android and iOS tests on real physical devices and emulators in Google's data centers. It's the standard solution for CI/CD mobile testing when you need real devices without managing device labs. This guide covers setup, test matrix configuration, result analysis, and strategies to keep costs under control.

Key Takeaways

Test on both physical devices and emulators. Physical devices catch issues that emulators miss: hardware-specific rendering, actual touch latency, real cellular/WiFi behavior. Emulators are cheaper for regression detection; physical devices are for pre-release validation.

Configure a test matrix to cover device diversity. One test run can cover multiple device models and OS versions. Don't just test on the latest — test on the OS versions your users actually have.

Robo test finds crashes without writing a line of test code. Firebase's Robo test crawler explores your app automatically, finding crashes you didn't write tests for. Run it on every release build.

Use test sharding to parallelize long test suites. A 30-minute test suite can run in 5 minutes with 6 shards. Sharding across devices reduces total CI time significantly.

Review artifacts on every failure. Firebase Test Lab generates screenshots, videos, logcat output, and performance metrics. A failing test with no artifact review is a missed debugging opportunity.

What Firebase Test Lab Provides

Firebase Test Lab is a cloud-based device testing infrastructure. You upload your app APK (Android) or IPA (iOS), your test APK, and a configuration file. Firebase runs your tests on physical devices and emulators in their data centers and returns results.

What you get:

  • Test results (pass/fail per test)
  • Screenshots at key points during execution
  • Video of the full test run
  • Logcat output (Android) or syslog (iOS)
  • Performance metrics (CPU, memory)
  • Crash reports with stack traces

Supported test types:

  • Android Instrumentation tests (Espresso, UIAutomator)
  • Android Robo test (automated crawler, no test code needed)
  • Android Game Loop test
  • iOS XCUITest

Setup

Prerequisites

  1. A Firebase project (free to create at console.firebase.google.com)
  2. Google Cloud SDK with gcloud CLI
  3. Your app built and signed for testing

Install and Configure gcloud CLI

# Install gcloud
<span class="hljs-comment"># macOS with Homebrew
brew install google-cloud-sdk

<span class="hljs-comment"># Or download directly from Google
curl https://sdk.cloud.google.com <span class="hljs-pipe">| bash

<span class="hljs-comment"># Authenticate
gcloud auth login
gcloud config <span class="hljs-built_in">set project YOUR_FIREBASE_PROJECT_ID

Enable Test Lab API

gcloud services enable testing.googleapis.com
gcloud services <span class="hljs-built_in">enable toolresults.googleapis.com

Running Your First Android Test

Build your test APK

# From your project root
./gradlew assembleDebug assembleDebugAndroidTest

This produces:

  • app/build/outputs/apk/debug/app-debug.apk (your app)
  • app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk (your tests)

Run on Firebase Test Lab

gcloud firebase test android run \
  --<span class="hljs-built_in">type instrumentation \
  --app app/build/outputs/apk/debug/app-debug.apk \
  --<span class="hljs-built_in">test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
  --device model=Pixel6,version=33,locale=en,orientation=portrait \
  --<span class="hljs-built_in">timeout 10m

Results appear in your terminal and in the Firebase console.

Running Robo Test (No Test Code Required)

gcloud firebase test android run \
  --<span class="hljs-built_in">type robo \
  --app app/build/outputs/apk/debug/app-debug.apk \
  --device model=Pixel6,version=33 \
  --robo-directives click:login_button=,text:email_input=<span class="hljs-built_in">test@example.com \
  --<span class="hljs-built_in">timeout 3m

--robo-directives guides the Robo crawler: tell it which button to click, what text to enter in fields. This prevents it from getting stuck on a login screen.

Test Matrix Configuration

A test matrix runs your tests against multiple device/OS combinations in parallel:

gcloud firebase test android run \
  --<span class="hljs-built_in">type instrumentation \
  --app app-debug.apk \
  --<span class="hljs-built_in">test app-debug-androidTest.apk \
  --device model=Pixel6,version=33,locale=en,orientation=portrait \
  --device model=Pixel4a,version=31,locale=en,orientation=portrait \
  --device model=SamsungGalaxyS21,version=32,locale=en,orientation=portrait \
  --device model=Pixel6,version=33,locale=en,orientation=landscape \
  --<span class="hljs-built_in">timeout 15m

This runs all tests on 4 device/orientation combinations in parallel. Total time is approximately the time for a single run.

Device Selection Strategy

Don't test on every available device — that's expensive and slow. Test on a representative matrix:

Tier 1 (every commit):

  • Latest Pixel (latest OS) — your development target
  • Most popular Samsung model on current OS

Tier 2 (before release):

  • Older OS versions (2 versions back)
  • Different screen sizes (tablet vs phone)
  • Popular manufacturer variants (Samsung, OnePlus)
# Tier 1 — fast, runs on every PR
TIER1_DEVICES=<span class="hljs-string">"--device model=Pixel7,version=33 --device model=OnePlus9Pro,version=31"

<span class="hljs-comment"># Tier 2 — full matrix, runs before release
TIER2_DEVICES=<span class="hljs-string">"
  --device model=Pixel7,version=33
  --device model=Pixel5,version=31
  --device model=SamsungGalaxyS22,version=32
  --device model=MotorolaEdge30,version=31
  --device model=Pixel4,version=30"

iOS Testing

For iOS XCUITest:

# Build for testing (requires Xcode)
xcodebuild build-for-testing \
  -scheme YourApp \
  -destination <span class="hljs-string">"generic/platform=iOS Simulator" \
  -derivedDataPath DerivedData

<span class="hljs-comment"># Package the test bundle
<span class="hljs-built_in">cd DerivedData/Build/Products/Debug-iphonesimulator
zip -r YourApp.zip YourApp.app YourAppUITests-Runner.app

<span class="hljs-comment"># Upload to Firebase Test Lab
gcloud firebase <span class="hljs-built_in">test ios run \
  --<span class="hljs-built_in">test YourApp.zip \
  --device model=iphone14pro,version=16.6,locale=en_US,orientation=portrait \
  --<span class="hljs-built_in">timeout 10m

Integration with GitHub Actions

# .github/workflows/mobile-tests.yml
name: Mobile Tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  android-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-java@v3
        with:
          java-version: 17
          distribution: temurin
      
      - name: Build APKs
        run: ./gradlew assembleDebug assembleDebugAndroidTest
      
      - name: Authenticate with Google Cloud
        uses: google-github-actions/auth@v1
        with:
          credentials_json: ${{ secrets.FIREBASE_SERVICE_ACCOUNT }}
      
      - name: Set up Google Cloud SDK
        uses: google-github-actions/setup-gcloud@v1
      
      - name: Run tests on Firebase Test Lab
        run: |
          gcloud firebase test android run \
            --type instrumentation \
            --app app/build/outputs/apk/debug/app-debug.apk \
            --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
            --device model=Pixel6,version=33 \
            --timeout 15m \
            --results-bucket gs://your-project-test-results \
            --results-dir "github-actions-${GITHUB_RUN_ID}"
      
      - name: Download test results
        if: always()
        run: |
          gsutil -m cp -r \
            gs://your-project-test-results/github-actions-${GITHUB_RUN_ID} \
            test-results/
      
      - name: Upload test artifacts
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: firebase-test-results
          path: test-results/

Service Account Setup

Create a service account with the Firebase Test Admin role:

# Create service account
gcloud iam service-accounts create github-test-runner \
  --display-name <span class="hljs-string">"GitHub Actions Test Runner"

<span class="hljs-comment"># Grant Firebase Test Admin role
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member serviceAccount:github-test-runner@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role roles/firebase.analyticsAdmin

<span class="hljs-comment"># Also need these:
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member serviceAccount:github-test-runner@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role roles/cloudtestservice.testAdmin

<span class="hljs-comment"># Download key
gcloud iam service-accounts keys create firebase-key.json \
  --iam-account github-test-runner@YOUR_PROJECT_ID.iam.gserviceaccount.com

Store firebase-key.json contents as FIREBASE_SERVICE_ACCOUNT in GitHub Secrets.

Test Sharding

For large test suites, sharding runs tests in parallel across multiple devices:

# Split tests into 4 shards (run on 4 devices simultaneously)
gcloud firebase <span class="hljs-built_in">test android run \
  --<span class="hljs-built_in">type instrumentation \
  --app app-debug.apk \
  --<span class="hljs-built_in">test app-debug-androidTest.apk \
  --device model=Pixel6,version=33 \
  --num-uniform-shards 4 \
  --<span class="hljs-built_in">timeout 15m

With 4 shards, a 40-minute test suite completes in ~10 minutes. The cost is roughly the same (4 device-hours instead of 1), but CI time is dramatically reduced.

For non-uniform sharding by test class:

gcloud firebase test android run \
  --<span class="hljs-built_in">type instrumentation \
  --app app-debug.apk \
  --<span class="hljs-built_in">test app-debug-androidTest.apk \
  --device model=Pixel6,version=33 \
  --test-targets <span class="hljs-string">"class com.example.LoginTests" \
  --test-targets <span class="hljs-string">"class com.example.CheckoutTests"

Analyzing Results

In the Firebase Console

The Firebase console shows a matrix view: each row is a test, each column is a device. Green = pass, red = fail, yellow = flaky (sometimes passes, sometimes fails).

Click any failing test to see:

  • Screenshot sequence
  • Video recording
  • Logcat with the exact crash
  • Stack trace for crashes

From the Command Line

# Get result bucket path from the test run output
<span class="hljs-comment"># Then download artifacts
gsutil <span class="hljs-built_in">ls gs://your-project-test-results/matrix-id/

<span class="hljs-comment"># Download specific device results
gsutil <span class="hljs-built_in">cp -r gs://your-project-test-results/matrix-id/Pixel6-33-en-portrait/logcat .
gsutil <span class="hljs-built_in">cp -r gs://your-project-test-results/matrix-id/Pixel6-33-en-portrait/test_result_1.xml .

Parse the JUnit XML for CI integration:

# Check if any tests failed
python3 -c <span class="hljs-string">"
import xml.etree.ElementTree as ET
import sys

tree = ET.parse('test_result_1.xml')
root = tree.getroot()
failures = len(root.findall('.//failure'))
errors = len(root.findall('.//error'))
print(f'Failures: {failures}, Errors: {errors}')
sys.exit(1 if failures + errors > 0 else 0)
"

Cost Optimization

Firebase Test Lab pricing is per device-minute. Managing costs matters for frequent CI runs.

Strategies:

Separate fast and slow tests. Run a small set of smoke tests (5-10 critical flows) on every commit. Run the full suite only on PRs to main or nightly.

Use virtual devices for fast feedback. Virtual (emulator) devices cost less than physical devices on Firebase Test Lab. Use virtual for frequent runs, physical for pre-release.

# Virtual device (cheaper)
--device model=Nexus6P,version=27  <span class="hljs-comment"># Virtual device

<span class="hljs-comment"># Physical device (more expensive, more accurate)
--device model=Pixel6,version=33  <span class="hljs-comment"># Physical device

Reduce timeout. The --timeout value is the maximum time per device. Set it to the minimum needed (test suite time + 20% buffer), not the maximum allowed.

Tag tests for selective runs:

@Test
@Category(SmokeTest.class)
public void loginFlowWorks() { ... }

@Test
@Category(RegressionTest.class)
public void complexCartScenario() { ... }
# Run only smoke tests
gcloud firebase <span class="hljs-built_in">test android run \
  --test-targets <span class="hljs-string">"annotation com.example.SmokeTest" \
  ...

<span class="hljs-comment"># Run full suite
gcloud firebase <span class="hljs-built_in">test android run \
  --test-targets <span class="hljs-string">"class com.example.AllTests" \
  ...

Summary

Firebase Test Lab solves the "run mobile tests in CI" problem without managing a device lab. Real devices catch real issues; the cloud infrastructure handles the device management.

The setup investment is real (~2-4 hours for initial configuration and CI integration), but the ROI is clear: mobile tests that actually run on every commit, on real devices, without maintaining hardware.

Start with Robo tests (zero test code, catches crashes) and a single-device instrumentation run, then expand to a full matrix as your test suite matures.

Read more