Testing

Testing Railway Deployments: Preview Environments and Service Networking

HelpMeTest

HelpMeTest

— 7 min read
Testing Railway Deployments: Preview Environments and Service Networking

Railway has quietly become one of the most developer-friendly deployment platforms available. Its combination of instant deploys, automatic preview environments, and built-in service networking makes it compelling for teams that want to move fast. But "move fast" only works if you have a testing strategy that keeps up — and Railway's architecture introduces some testing challenges that aren't obvious until you've run into them.

This guide covers how to test Railway deployments effectively: from validating ephemeral preview environments in pull requests to testing inter-service communication over Railway's private network.

Railway's Deployment Model

Railway organizes applications into projects and services. A project is your logical application boundary; services are the individual components (web app, database, background worker, etc.) that run inside it. Each service has its own environment variables, deploy settings, and health checks.

When you open a pull request, Railway can automatically spin up a complete copy of your project — every service, every database, every worker — with its own URLs and isolated state. These are preview environments, and they're Railway's killer feature for testing workflows.

Setting Up Preview Environments

Preview environments require a railway.toml configuration in your repository root:

[build]
builder = "NIXPACKS"
buildCommand = "npm run build"

[deploy]
startCommand = "npm run start"
healthcheckPath = "/health"
healthcheckTimeout = 300
restartPolicyType = "ON_FAILURE"
restartPolicyMaxRetries = 3

Enable preview environments in your Railway project settings. Once enabled, each PR gets its own environment with URLs following the pattern <service>-<branch>-<project>.up.railway.app.

Here's a GitHub Actions workflow that tests your preview environment on every PR:

# .github/workflows/preview-tests.yml
name: Preview Environment Tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  wait-for-preview:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.get-url.outputs.url }}
    steps:
      - name: Install Railway CLI
        run: npm install -g @railway/cli

      - name: Wait for preview deployment
        id: get-url
        run: |
          # Railway CLI needs auth token
          export RAILWAY_TOKEN="${{ secrets.RAILWAY_TOKEN }}"

          # Poll until deployment is healthy
          MAX_ATTEMPTS=30
          for i in $(seq 1 $MAX_ATTEMPTS); do
            STATUS=$(railway status --json 2>/dev/null | jq -r '.status // "unknown"')
            echo "Attempt $i/$MAX_ATTEMPTS: deployment status = $STATUS"

            if [ "$STATUS" = "SUCCESS" ]; then
              URL=$(railway domain --json | jq -r '.url')
              echo "url=$URL" >> $GITHUB_OUTPUT
              echo "Preview URL: $URL"
              break
            fi

            if [ "$STATUS" = "FAILED" ]; then
              echo "Deployment failed"
              exit 1
            fi

            sleep 10
          done
        env:
          RAILWAY_PROJECT_ID: ${{ secrets.RAILWAY_PROJECT_ID }}
          RAILWAY_ENVIRONMENT: pr-${{ github.event.pull_request.number }}

  test-preview:
    needs: wait-for-preview
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run smoke tests against preview
        run: ./scripts/smoke-tests.sh
        env:
          APP_URL: ${{ needs.wait-for-preview.outputs.preview_url }}

      - name: Run integration tests
        run: npm test -- --testPathPattern=integration
        env:
          TEST_BASE_URL: ${{ needs.wait-for-preview.outputs.preview_url }}

Testing Service Networking (service.internal)

Railway's private network lets services communicate with each other using .railway.internal DNS names. This is Railway's equivalent of Docker Compose service networking — services talk to each other over a private network without exposing ports to the internet.

Testing service-to-service communication is often overlooked, but it's where subtle bugs hide. Your API might work fine on its own, but fail when it tries to reach the database or Redis service over the internal network.

Service discovery on Railway uses the pattern <service-name>.railway.internal on the service's private port. Here's how to validate internal networking in your application:

// internal-health.js — a dedicated endpoint for verifying internal connectivity
const services = {
  database: {
    url: `http://${process.env.DATABASE_SERVICE_HOST || 'postgres.railway.internal'}:5432`,
    check: async () => {
      // For Postgres, use pg client
      const { Pool } = require('pg');
      const pool = new Pool({ connectionString: process.env.DATABASE_URL });
      await pool.query('SELECT 1');
      await pool.end();
      return { status: 'ok' };
    }
  },
  redis: {
    url: `redis://${process.env.REDIS_SERVICE_HOST || 'redis.railway.internal'}:6379`,
    check: async () => {
      const redis = require('redis');
      const client = redis.createClient({ url: process.env.REDIS_URL });
      await client.connect();
      await client.ping();
      await client.quit();
      return { status: 'ok' };
    }
  },
  worker: {
    url: `http://worker.railway.internal:3001`,
    check: async () => {
      const response = await fetch(`http://worker.railway.internal:3001/health`);
      if (!response.ok) throw new Error(`HTTP ${response.status}`);
      return { status: 'ok' };
    }
  }
};

app.get('/health/internal', async (req, res) => {
  const results = {};
  let allHealthy = true;

  for (const [name, service] of Object.entries(services)) {
    try {
      results[name] = await Promise.race([
        service.check(),
        new Promise((_, reject) =>
          setTimeout(() => reject(new Error('timeout')), 5000)
        )
      ]);
    } catch (err) {
      results[name] = { status: 'error', message: err.message };
      allHealthy = false;
    }
  }

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'ok' : 'degraded',
    services: results,
    environment: process.env.RAILWAY_ENVIRONMENT_NAME
  });
});

Then test this endpoint in CI:

#!/bin/bash
# test-internal-networking.sh

APP_URL="${APP_URL}"
echo "Testing internal service connectivity at $APP_URL"

RESPONSE=$(curl -s "$APP_URL/health/internal")
STATUS=$(echo "$RESPONSE" | jq -r '.status')

if [ "$STATUS" != "ok" ]; then
  echo "FAIL: Internal networking check failed"
  echo "$RESPONSE" | jq '.'
  exit 1
fi

echo "PASS: All internal services reachable"
echo "$RESPONSE" | jq '.services'

Railway CLI in CI Pipelines

The Railway CLI is the most direct way to interact with Railway from CI. Install it, authenticate with a service account token, and you can deploy, check status, and query logs.

# Install Railway CLI
npm install -g @railway/cli

# Authenticate using a token (never interactive in CI)
export RAILWAY_TOKEN="your-service-account-token"

# Deploy to a specific environment
railway up --environment production --service api

# Check deployment status
railway status

# View recent logs (useful for debugging CI failures)
railway logs --lines 100

# Run a one-off command in the deployed environment
railway run -- npm run db:migrate

Here's a complete deployment and test script for production releases:

#!/bin/bash
# deploy-and-test.sh

set -e

ENVIRONMENT="${RAILWAY_ENVIRONMENT:-production}"
SERVICE="${RAILWAY_SERVICE:-api}"

echo "=== Deploying to Railway ($ENVIRONMENT/$SERVICE) ==="

# Deploy
railway up \
  --environment "$ENVIRONMENT" \
  --service "$SERVICE" \
  --detach  # Return immediately, don't tail logs

# Wait for deployment to complete
echo "Waiting for deployment..."
TIMEOUT=300
ELAPSED=0

while [ $ELAPSED -lt $TIMEOUT ]; do
  STATUS=$(railway status --json | jq -r '.deploymentStatus // "unknown"')

  case "$STATUS" in
    "SUCCESS")
      echo "Deployment succeeded"
      break
      ;;
    "FAILED"|"CRASHED")
      echo "Deployment failed with status: $STATUS"
      railway logs --lines 50
      exit 1
      ;;
    *)
      echo "  Status: $STATUS (${ELAPSED}s elapsed)"
      sleep 10
      ELAPSED=$((ELAPSED + 10))
      ;;
  esac
done

if [ $ELAPSED -ge $TIMEOUT ]; then
  echo "Timeout waiting for deployment"
  exit 1
fi

# Get the deployed URL
APP_URL=$(railway domain --json | jq -r '.url')
echo "Deployed to: $APP_URL"

# Run smoke tests
echo "=== Running smoke tests ==="
APP_URL="$APP_URL" ./scripts/smoke-tests.sh

echo "=== Deploy and test complete ==="

Testing Environment Variables

One of the trickiest parts of Railway testing is environment variables. Railway manages variables per-environment, and misconfigured variables are a common cause of deployment failures that only manifest at runtime.

Here's a validation approach — an endpoint that reports which expected variables are present (without exposing their values):

// env-check.js
const REQUIRED_VARS = [
  'DATABASE_URL',
  'REDIS_URL',
  'SECRET_KEY',
  'STRIPE_SECRET_KEY',
  'SENDGRID_API_KEY'
];

const OPTIONAL_VARS = [
  'SENTRY_DSN',
  'ANALYTICS_KEY',
  'FEATURE_FLAGS'
];

app.get('/health/env', (req, res) => {
  // Only expose this endpoint in non-production, or behind auth
  if (process.env.NODE_ENV === 'production' && !req.headers['x-internal-check']) {
    return res.status(403).json({ error: 'Forbidden' });
  }

  const required = {};
  let allPresent = true;

  for (const varName of REQUIRED_VARS) {
    const present = !!process.env[varName];
    required[varName] = present ? 'SET' : 'MISSING';
    if (!present) allPresent = false;
  }

  const optional = {};
  for (const varName of OPTIONAL_VARS) {
    optional[varName] = process.env[varName] ? 'SET' : 'NOT SET';
  }

  res.status(allPresent ? 200 : 503).json({
    status: allPresent ? 'ok' : 'missing_vars',
    environment: process.env.RAILWAY_ENVIRONMENT_NAME,
    required,
    optional
  });
});

Test it in CI:

ENV_STATUS=$(curl -s -H "X-Internal-Check: ci" "$APP_URL/health/env")
MISSING=$(echo "$ENV_STATUS" | jq -r '.required | to_entries[] | select(.value == "MISSING") | .key')

if [ -n "$MISSING" ]; then
  echo "FAIL: Missing required environment variables:"
  echo "$MISSING"
  exit 1
fi

Multi-Service Integration Testing

Railway's project model makes it natural to run multi-service integration tests. When all your services deploy together in a preview environment, you can test flows that cross service boundaries.

Here's a test structure for a multi-service Railway project:

// tests/integration/multi-service.test.js
const BASE_URL = process.env.TEST_BASE_URL;

describe('Multi-service integration', () => {
  test('API can reach background worker via job queue', async () => {
    // Trigger a background job through the API
    const jobResponse = await fetch(`${BASE_URL}/api/jobs`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ type: 'send-email', payload: { to: 'test@example.com' } })
    });

    expect(jobResponse.status).toBe(202);
    const { jobId } = await jobResponse.json();

    // Poll until the worker processes the job
    let jobStatus;
    for (let i = 0; i < 20; i++) {
      await new Promise(r => setTimeout(r, 1000));
      const statusResponse = await fetch(`${BASE_URL}/api/jobs/${jobId}`);
      const data = await statusResponse.json();
      jobStatus = data.status;
      if (['completed', 'failed'].includes(jobStatus)) break;
    }

    expect(jobStatus).toBe('completed');
  });

  test('File upload service stores to shared storage', async () => {
    const formData = new FormData();
    formData.append('file', new Blob(['test content']), 'test.txt');

    const uploadResponse = await fetch(`${BASE_URL}/api/uploads`, {
      method: 'POST',
      body: formData
    });

    expect(uploadResponse.status).toBe(201);
    const { fileUrl } = await uploadResponse.json();

    // Verify the file is accessible
    const fileResponse = await fetch(fileUrl);
    expect(fileResponse.status).toBe(200);
    expect(await fileResponse.text()).toBe('test content');
  });

  test('Internal service health check passes', async () => {
    const response = await fetch(`${BASE_URL}/health/internal`);
    const data = await response.json();
    expect(data.status).toBe('ok');
  });
});

Using Deployment Hooks for Test Triggers

Railway supports deploy hooks — webhooks that fire when a deployment completes. You can use these to trigger external test runs from HelpMeTest or your own test infrastructure.

Configure a webhook in Railway's project settings, then handle it in your CI or test runner:

// webhook-handler.js — receives Railway deployment events
app.post('/webhooks/railway', async (req, res) => {
  const { type, status, environmentName, serviceUrl } = req.body;

  // Acknowledge immediately
  res.json({ received: true });

  if (type === 'DEPLOY' && status === 'SUCCESS') {
    console.log(`Deploy succeeded for ${environmentName}: ${serviceUrl}`);

    // Trigger HelpMeTest run or your test suite
    await triggerTestSuite({
      environment: environmentName,
      url: serviceUrl
    });
  }
});

HelpMeTest can also be configured to run automatically after Railway deployments complete. Its AI-generated tests can be set to run against your preview environment URL, giving you browser-level end-to-end confidence on every pull request — not just curl-based smoke tests.

Continuous Monitoring on Railway

Railway deployments can be stable for days, then fail due to a memory leak, a database connection pool exhaustion, or a change in a third-party API response. Smoke tests only run at deploy time; they won't catch issues that emerge gradually.

For continuous production health monitoring on Railway:

#!/bin/bash
# continuous-health-check.sh
# Run this as a Railway cron job service

APP_URL="${MONITORED_APP_URL}"
ALERT_WEBHOOK="${SLACK_WEBHOOK_URL}"

check_health() {
  local endpoint="$1"
  local expected_status="${2:-200}"

  ACTUAL=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "${APP_URL}${endpoint}")

  if [ "$ACTUAL" != "$expected_status" ]; then
    echo "FAIL: $endpoint returned $ACTUAL (expected $expected_status)"
    return 1
  fi
  return 0
}

FAILED=0
check_health "/health" || FAILED=$((FAILED + 1))
check_health "/health/db" || FAILED=$((FAILED + 1))
check_health "/health/internal" || FAILED=$((FAILED + 1))
check_health "/api/status" || FAILED=$((FAILED + 1))

if [ $FAILED -gt 0 ]; then
  curl -s -X POST "$ALERT_WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{\"text\": \"Railway health check failed: $FAILED endpoint(s) unhealthy on $(date)\"}"
  exit 1
fi

echo "$(date): All health checks passed"

Add this as a Railway service that runs on a cron schedule in your railway.toml:

[deploy]
cronSchedule = "*/5 * * * *"
startCommand = "bash /app/continuous-health-check.sh"

Summary

Railway's architecture rewards teams that invest in testing infrastructure. The key principles:

  • Preview environments are your best tool — test every PR against a real isolated copy of your infrastructure
  • Internal networking tests catch the service-to-service failures that smoke tests miss
  • Railway CLI in CI gives you deployment control and status visibility without relying on UI interactions
  • Deploy hooks bridge Railway's deployment events to your test runners
  • Continuous monitoring catches production drift that only shows up after deploy

With these patterns in place, Railway's developer experience stays fast while your reliability confidence stays high.

Read more

Start now free