Testing

Testing Apps on Fly.io: Health Checks, Multi-Region, and CI Integration

HelpMeTest

HelpMeTest

— 7 min read
Testing Apps on Fly.io: Health Checks, Multi-Region, and CI Integration

Fly.io has become a go-to platform for teams that want Heroku-style simplicity with genuine multi-region distribution. You push code, and Fly.io runs it close to your users in 30+ regions worldwide. But with that power comes a real testing challenge: how do you verify that your app is healthy not just in one region but across all the regions you've deployed to? How do you validate rolling deployments without taking down production? And how do you wire all of this into a CI pipeline that gives you confidence before traffic hits real users?

This guide covers practical testing strategies for Fly.io deployments — from configuring health checks in fly.toml to writing smoke tests that run automatically after every deploy.

Understanding Fly.io's Deployment Model

Before writing tests, it helps to understand what you're testing. Fly.io runs your app as Machines — lightweight Firecracker microVMs that boot in milliseconds. When you deploy, Fly.io performs a rolling update: new Machines start, health checks run, and only if checks pass does Fly.io route traffic to them.

This means your health check configuration is your first line of defense. Get it wrong, and either bad deployments slip through or good ones get rolled back.

Configuring Health Checks in fly.toml

Your fly.toml is the foundation of your deployment configuration. Here's a production-ready health check setup:

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 1

  [[http_service.checks]]
    grace_period = "10s"
    interval = "15s"
    method = "GET"
    path = "/health"
    protocol = "http"
    timeout = "5s"
    tls_skip_verify = false

    [http_service.checks.headers]
      X-Internal-Check = "fly-health"

[checks]
  [checks.db_connection]
    grace_period = "20s"
    interval = "30s"
    method = "GET"
    path = "/health/db"
    protocol = "http"
    timeout = "10s"
    port = 8080

The /health endpoint should be lightweight — just enough to confirm the process is alive. The /health/db check is more thorough. Here's what these endpoints should look like in a Node.js app:

// health.js
app.get('/health', (req, res) => {
  res.json({
    status: 'ok',
    region: process.env.FLY_REGION,
    app: process.env.FLY_APP_NAME,
    machine: process.env.FLY_MACHINE_ID,
    timestamp: new Date().toISOString()
  });
});

app.get('/health/db', async (req, res) => {
  try {
    await db.query('SELECT 1');
    res.json({ status: 'ok', db: 'connected' });
  } catch (err) {
    res.status(503).json({ status: 'error', db: err.message });
  }
});

Including FLY_REGION in your health response is useful because it lets you verify which region a request is being served from — critical for multi-region debugging.

Testing Multi-Region Deployments

Fly.io lets you run Machines in multiple regions simultaneously. Testing multi-region behavior means verifying that:

  1. Each region is actually serving traffic
  2. Data replication is working (if you use LiteFS or Fly Postgres with read replicas)
  3. Region-specific configuration is correct

Here's a bash script that validates all your configured regions:

#!/bin/bash
# test-regions.sh — validate all Fly.io regions are healthy

APP_NAME="your-app-name"
REGIONS=("iad" "lhr" "sin" "syd")
FAILED=0

for region in "${REGIONS[@]}"; do
  echo "Testing region: $region"

  # Use the Fly.io anycast address with region header
  RESPONSE=$(curl -s -o /tmp/response.json -w "%{http_code}" \
    -H "fly-prefer-region: $region" \
    "https://${APP_NAME}.fly.dev/health")

  if [ "$RESPONSE" != "200" ]; then
    echo "  FAIL: HTTP $RESPONSE from $region"
    FAILED=$((FAILED + 1))
    continue
  fi

  SERVING_REGION=$(jq -r '.region' /tmp/response.json)
  echo "  OK: Serving from $SERVING_REGION (requested $region)"

  if [ "$SERVING_REGION" != "$region" ]; then
    echo "  WARN: Region mismatch — possibly no machine in $region, routed to $SERVING_REGION"
  fi
done

if [ $FAILED -gt 0 ]; then
  echo "Region tests failed: $FAILED/${#REGIONS[@]}"
  exit 1
fi

echo "All regions healthy"

Using the Machines API for Deployment Validation

The Fly.io Machines API gives you programmatic control over your deployment. You can use it to verify Machine states after a deploy:

#!/bin/bash
# check-machines.sh — verify all machines are in expected state after deploy

APP_NAME="your-app-name"
FLY_API_TOKEN="${FLY_API_TOKEN}"
EXPECTED_STATE="started"

MACHINES=$(curl -s \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  "https://api.machines.dev/v1/apps/${APP_NAME}/machines")

TOTAL=$(echo "$MACHINES" | jq length)
STARTED=$(echo "$MACHINES" | jq "[.[] | select(.state == \"${EXPECTED_STATE}\")] | length")

echo "Machines total: $TOTAL, started: $STARTED"

if [ "$STARTED" -ne "$TOTAL" ]; then
  echo "ERROR: Not all machines are in state '${EXPECTED_STATE}'"
  echo "$MACHINES" | jq '.[] | {id, region, state}'
  exit 1
fi

# Check version consistency — all machines should run the same image
VERSIONS=$(echo "$MACHINES" | jq -r '.[].image_ref.tag' | sort -u)
VERSION_COUNT=$(echo "$VERSIONS" | wc -l | tr -d ' ')

if [ "$VERSION_COUNT" -gt 1 ]; then
  echo "WARN: Multiple versions running:"
  echo "$VERSIONS"
fi

echo "Deployment verified: $TOTAL machines running version $(echo "$VERSIONS" | head -1)"

flyctl in CI Pipelines

The most reliable pattern for Fly.io CI integration is to deploy, wait for health checks, then run smoke tests. Here's a complete GitHub Actions workflow:

# .github/workflows/deploy.yml
name: Deploy and Test

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup flyctl
        uses: superfly/flyctl-actions/setup-flyctl@master

      - name: Deploy to Fly.io
        run: flyctl deploy --remote-only --wait-timeout 300
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

      - name: Wait for deployment to stabilize
        run: |
          echo "Waiting for health checks to pass..."
          sleep 15

          # Poll until app responds consistently
          for i in {1..10}; do
            STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://your-app.fly.dev/health)
            if [ "$STATUS" = "200" ]; then
              echo "Health check passed on attempt $i"
              break
            fi
            echo "Attempt $i: got HTTP $STATUS, retrying..."
            sleep 5
          done

      - name: Run smoke tests
        run: ./scripts/smoke-tests.sh
        env:
          APP_URL: https://your-app.fly.dev

      - name: Verify machine states
        run: ./scripts/check-machines.sh
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Writing Effective Smoke Tests

Smoke tests after a Fly.io deploy should cover the critical paths your users depend on. Here's a structured approach using bash and curl:

#!/bin/bash
# smoke-tests.sh

APP_URL="${APP_URL:-https://your-app.fly.dev}"
FAILED=0

run_test() {
  local name="$1"
  local result="$2"
  local expected="$3"

  if [ "$result" = "$expected" ]; then
    echo "  PASS: $name"
  else
    echo "  FAIL: $name (expected: $expected, got: $result)"
    FAILED=$((FAILED + 1))
  fi
}

echo "=== Smoke Tests: $APP_URL ==="

# Test 1: Basic health check
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$APP_URL/health")
run_test "Health endpoint returns 200" "$STATUS" "200"

# Test 2: Database connectivity
DB_STATUS=$(curl -s "$APP_URL/health/db" | jq -r '.status')
run_test "Database connection is healthy" "$DB_STATUS" "ok"

# Test 3: API authentication endpoint exists
AUTH_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"wrong"}' \
  "$APP_URL/api/auth/login")
run_test "Auth endpoint reachable (returns 401 for bad creds)" "$AUTH_STATUS" "401"

# Test 4: Static assets load
ASSET_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$APP_URL/favicon.ico")
run_test "Static assets accessible" "$ASSET_STATUS" "200"

# Test 5: Response time is acceptable
RESPONSE_TIME=$(curl -s -o /dev/null -w "%{time_total}" "$APP_URL/health")
RESPONSE_MS=$(echo "$RESPONSE_TIME * 1000" | bc | cut -d. -f1)
if [ "$RESPONSE_MS" -lt 2000 ]; then
  echo "  PASS: Response time acceptable (${RESPONSE_MS}ms)"
else
  echo "  FAIL: Response time too slow (${RESPONSE_MS}ms)"
  FAILED=$((FAILED + 1))
fi

echo ""
if [ $FAILED -gt 0 ]; then
  echo "Smoke tests FAILED: $FAILED tests failed"
  exit 1
fi
echo "All smoke tests passed"

Testing Rolling Deployments

Fly.io performs rolling deployments by default — old Machines keep running while new ones start. You can verify this behavior with a script that monitors availability during a deploy:

#!/bin/bash
# monitor-during-deploy.sh
# Run this in one terminal while deploying in another

APP_URL="https://your-app.fly.dev"
INTERVAL=2
DOWNTIME=0
REQUESTS=0

echo "Monitoring $APP_URL during deployment..."
echo "Press Ctrl+C to stop"

while true; do
  START=$(date +%s%N)
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$APP_URL/health")
  END=$(date +%s%N)
  LATENCY=$(( (END - START) / 1000000 ))

  REQUESTS=$((REQUESTS + 1))
  TIMESTAMP=$(date +%H:%M:%S)

  if [ "$STATUS" != "200" ]; then
    DOWNTIME=$((DOWNTIME + 1))
    echo "$TIMESTAMP  FAIL  HTTP $STATUS  ${LATENCY}ms  [downtime: $DOWNTIME/$REQUESTS]"
  else
    echo "$TIMESTAMP  OK    HTTP $STATUS  ${LATENCY}ms"
  fi

  sleep $INTERVAL
done

Continuous Monitoring with HelpMeTest

Running smoke tests after each deploy catches regressions, but it doesn't tell you about problems that develop over time — memory leaks, database connection exhaustion, or third-party API degradation. For ongoing confidence, tools like HelpMeTest complement your deploy pipeline by running full browser-based tests on a schedule.

HelpMeTest's Robot Framework and Playwright-based tests can simulate real user journeys across your Fly.io regions, with AI-powered test generation that creates tests from plain English descriptions. With usage-based pricing ($0.003/run, no base fee) covering unlimited tests and parallel execution, you can run comprehensive multi-region checks every few minutes without worrying about test infrastructure costs.

A typical setup pairs the CI smoke tests above with HelpMeTest monitoring runs every 5 minutes, giving you both immediate post-deploy confidence and ongoing production health visibility.

Validating fly.toml Configuration Changes

When you change fly.toml — especially health check settings or machine sizing — validate the configuration before deploying:

# Validate fly.toml syntax
flyctl config validate

# Show planned changes without deploying
flyctl deploy --dry-run

# Check current health check status
flyctl checks list

# Watch health check results in real time
flyctl checks list --watch

After any configuration change, verify that your health checks are firing correctly:

# Tail logs to see health check requests
flyctl logs --app your-app-name | grep "health"

# Check machine status
flyctl status --app your-app-name

# SSH into a machine to debug health check behavior
flyctl ssh console --app your-app-name

Testing Fly.io Postgres

If you're using Fly Postgres, add a dedicated test for your database cluster health:

#!/bin/bash
# test-fly-postgres.sh

PG_APP="your-app-db"

echo "Testing Fly Postgres cluster: $PG_APP"

# Check cluster status
flyctl postgres status --app "$PG_APP"

# Verify leader election
MACHINES=$(flyctl machines list --app "$PG_APP" --json)
LEADER=$(echo "$MACHINES" | jq -r '[.[] | select(.checks[]?.status == "passing")] | length')
echo "Healthy machines: $LEADER"

# Test connection string works
flyctl postgres connect --app "$PG_APP" -c "SELECT version();"

Putting It All Together

Here's the complete testing pipeline for a Fly.io deployment:

  1. Pre-deploy: Run unit and integration tests in CI
  2. Deploy: flyctl deploy --remote-only --wait-timeout 300
  3. Post-deploy: Wait for Fly.io health checks, then run smoke tests
  4. Validate machines: Confirm all Machines are in started state and on the same version
  5. Multi-region check: Hit each configured region and verify responses
  6. Continuous monitoring: HelpMeTest runs full E2E tests on a schedule

The key insight is that Fly.io's built-in health checks and rolling deploy behavior give you a safety net, but they only check what you configure. Smoke tests and continuous monitoring catch what health checks miss — subtle regressions, performance degradation, and user-facing issues that only appear under real conditions.

With this setup, you can deploy to Fly.io with confidence, knowing that bad deploys will be caught before they reach users and that any post-deploy issues will surface immediately.

Read more

Start now free