Testing

Testing Railway Deployments: Preview Environments and Service Networking

HelpMeTest

HelpMeTest

— 11 min read
Testing Railway Deployments: Preview Environments and Service Networking

Railway has quietly become one of the most developer-friendly deployment platforms available. Its combination of instant deploys, automatic preview environments, and built-in service networking makes it compelling for teams that want to move fast. But "move fast" only works if you have a testing strategy that keeps up — and Railway's architecture introduces some testing challenges that aren't obvious until you've run into them.

This guide covers how to test Railway deployments effectively: from validating ephemeral preview environments in pull requests to testing inter-service communication over Railway's private network.

Railway's Deployment Model

Railway organizes applications into projects and services. A project is your logical application boundary; services are the individual components (web app, database, background worker, etc.) that run inside it. Each service has its own environment variables, deploy settings, and health checks.

When you open a pull request, Railway can automatically spin up a complete copy of your project — every service, every database, every worker — with its own URLs and isolated state. These are preview environments, and they're Railway's killer feature for testing workflows.

Setting Up Preview Environments

Preview environments require a railway.toml configuration in your repository root:

[build]
builder = "NIXPACKS"
buildCommand = "npm run build"

[deploy]
startCommand = "npm run start"
healthcheckPath = "/health"
healthcheckTimeout = 300
restartPolicyType = "ON_FAILURE"
restartPolicyMaxRetries = 3

Enable preview environments in your Railway project settings. Once enabled, each PR gets its own environment with URLs following the pattern <service>-<branch>-<project>.up.railway.app.

Here's a GitHub Actions workflow that tests your preview environment on every PR:

# .github/workflows/preview-tests.yml
name: Preview Environment Tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  wait-for-preview:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.get-url.outputs.url }}
    steps:
      - name: Install Railway CLI
        run: npm install -g @railway/cli

      - name: Wait for preview deployment
        id: get-url
        run: |
          # Railway CLI needs auth token
          export RAILWAY_TOKEN="${{ secrets.RAILWAY_TOKEN }}"

          # Poll until deployment is healthy
          MAX_ATTEMPTS=30
          for i in $(seq 1 $MAX_ATTEMPTS); do
            STATUS=$(railway status --json 2>/dev/null | jq -r '.status // "unknown"')
            echo "Attempt $i/$MAX_ATTEMPTS: deployment status = $STATUS"

            if [ "$STATUS" = "SUCCESS" ]; then
              URL=$(railway domain --json | jq -r '.url')
              echo "url=$URL" >> $GITHUB_OUTPUT
              echo "Preview URL: $URL"
              break
            fi

            if [ "$STATUS" = "FAILED" ]; then
              echo "Deployment failed"
              exit 1
            fi

            sleep 10
          done
        env:
          RAILWAY_PROJECT_ID: ${{ secrets.RAILWAY_PROJECT_ID }}
          RAILWAY_ENVIRONMENT: pr-${{ github.event.pull_request.number }}

  test-preview:
    needs: wait-for-preview
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run smoke tests against preview
        run: ./scripts/smoke-tests.sh
        env:
          APP_URL: ${{ needs.wait-for-preview.outputs.preview_url }}

      - name: Run integration tests
        run: npm test -- --testPathPattern=integration
        env:
          TEST_BASE_URL: ${{ needs.wait-for-preview.outputs.preview_url }}

Testing Service Networking (service.internal)

Railway's private network lets services communicate with each other using .railway.internal DNS names. This is Railway's equivalent of Docker Compose service networking — services talk to each other over a private network without exposing ports to the internet.

Testing service-to-service communication is often overlooked, but it's where subtle bugs hide. Your API might work fine on its own, but fail when it tries to reach the database or Redis service over the internal network.

Service discovery on Railway uses the pattern <service-name>.railway.internal on the service's private port. Here's how to validate internal networking in your application:

// internal-health.js — a dedicated endpoint for verifying internal connectivity
const services = {
  database: {
    url: `http://${process.env.DATABASE_SERVICE_HOST || 'postgres.railway.internal'}:5432`,
    check: async () => {
      // For Postgres, use pg client
      const { Pool } = require('pg');
      const pool = new Pool({ connectionString: process.env.DATABASE_URL });
      await pool.query('SELECT 1');
      await pool.end();
      return { status: 'ok' };
    }
  },
  redis: {
    url: `redis://${process.env.REDIS_SERVICE_HOST || 'redis.railway.internal'}:6379`,
    check: async () => {
      const redis = require('redis');
      const client = redis.createClient({ url: process.env.REDIS_URL });
      await client.connect();
      await client.ping();
      await client.quit();
      return { status: 'ok' };
    }
  },
  worker: {
    url: `http://worker.railway.internal:3001`,
    check: async () => {
      const response = await fetch(`http://worker.railway.internal:3001/health`);
      if (!response.ok) throw new Error(`HTTP ${response.status}`);
      return { status: 'ok' };
    }
  }
};

app.get('/health/internal', async (req, res) => {
  const results = {};
  let allHealthy = true;

  for (const [name, service] of Object.entries(services)) {
    try {
      results[name] = await Promise.race([
        service.check(),
        new Promise((_, reject) =>
          setTimeout(() => reject(new Error('timeout')), 5000)
        )
      ]);
    } catch (err) {
      results[name] = { status: 'error', message: err.message };
      allHealthy = false;
    }
  }

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'ok' : 'degraded',
    services: results,
    environment: process.env.RAILWAY_ENVIRONMENT_NAME
  });
});

Then test this endpoint in CI:

#!/bin/bash
<span class="hljs-comment"># test-internal-networking.sh

APP_URL=<span class="hljs-string">"${APP_URL}"
<span class="hljs-built_in">echo <span class="hljs-string">"Testing internal service connectivity at $APP_URL"

RESPONSE=$(curl -s <span class="hljs-string">"$APP_URL/health/internal")
STATUS=$(<span class="hljs-built_in">echo <span class="hljs-string">"$RESPONSE" <span class="hljs-pipe">| jq -r <span class="hljs-string">'.status')

<span class="hljs-keyword">if [ <span class="hljs-string">"$STATUS" != <span class="hljs-string">"ok" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Internal networking check failed"
  <span class="hljs-built_in">echo <span class="hljs-string">"$RESPONSE" <span class="hljs-pipe">| jq <span class="hljs-string">'.'
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

<span class="hljs-built_in">echo <span class="hljs-string">"PASS: All internal services reachable"
<span class="hljs-built_in">echo <span class="hljs-string">"$RESPONSE" <span class="hljs-pipe">| jq <span class="hljs-string">'.services'

Railway CLI in CI Pipelines

The Railway CLI is the most direct way to interact with Railway from CI. Install it, authenticate with a service account token, and you can deploy, check status, and query logs.

# Install Railway CLI
npm install -g @railway/cli

<span class="hljs-comment"># Authenticate using a token (never interactive in CI)
<span class="hljs-built_in">export RAILWAY_TOKEN=<span class="hljs-string">"your-service-account-token"

<span class="hljs-comment"># Deploy to a specific environment
railway up --environment production --service api

<span class="hljs-comment"># Check deployment status
railway status

<span class="hljs-comment"># View recent logs (useful for debugging CI failures)
railway logs --lines 100

<span class="hljs-comment"># Run a one-off command in the deployed environment
railway run -- npm run db:migrate

Here's a complete deployment and test script for production releases:

#!/bin/bash
<span class="hljs-comment"># deploy-and-test.sh

<span class="hljs-built_in">set -e

ENVIRONMENT=<span class="hljs-string">"${RAILWAY_ENVIRONMENT:-production}"
SERVICE=<span class="hljs-string">"${RAILWAY_SERVICE:-api}"

<span class="hljs-built_in">echo <span class="hljs-string">"=== Deploying to Railway ($ENVIRONMENT/<span class="hljs-variable">$SERVICE) ==="

<span class="hljs-comment"># Deploy
railway up \
  --environment <span class="hljs-string">"$ENVIRONMENT" \
  --service <span class="hljs-string">"$SERVICE" \
  --detach  <span class="hljs-comment"># Return immediately, don't tail logs

<span class="hljs-comment"># Wait for deployment to complete
<span class="hljs-built_in">echo <span class="hljs-string">"Waiting for deployment..."
TIMEOUT=300
ELAPSED=0

<span class="hljs-keyword">while [ <span class="hljs-variable">$ELAPSED -lt <span class="hljs-variable">$TIMEOUT ]; <span class="hljs-keyword">do
  STATUS=$(railway status --json <span class="hljs-pipe">| jq -r <span class="hljs-string">'.deploymentStatus // "unknown"')

  <span class="hljs-keyword">case <span class="hljs-string">"$STATUS" <span class="hljs-keyword">in
    <span class="hljs-string">"SUCCESS")
      <span class="hljs-built_in">echo <span class="hljs-string">"Deployment succeeded"
      <span class="hljs-built_in">break
      <span class="hljs-pipe">;;
    <span class="hljs-string">"FAILED"|<span class="hljs-string">"CRASHED")
      <span class="hljs-built_in">echo <span class="hljs-string">"Deployment failed with status: $STATUS"
      railway logs --lines 50
      <span class="hljs-built_in">exit 1
      <span class="hljs-pipe">;;
    *)
      <span class="hljs-built_in">echo <span class="hljs-string">"  Status: $STATUS (<span class="hljs-variable">${ELAPSED}s elapsed)"
      <span class="hljs-built_in">sleep 10
      ELAPSED=$((ELAPSED + <span class="hljs-number">10))
      <span class="hljs-pipe">;;
  <span class="hljs-keyword">esac
<span class="hljs-keyword">done

<span class="hljs-keyword">if [ <span class="hljs-variable">$ELAPSED -ge <span class="hljs-variable">$TIMEOUT ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"Timeout waiting for deployment"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

<span class="hljs-comment"># Get the deployed URL
APP_URL=$(railway domain --json <span class="hljs-pipe">| jq -r <span class="hljs-string">'.url')
<span class="hljs-built_in">echo <span class="hljs-string">"Deployed to: $APP_URL"

<span class="hljs-comment"># Run smoke tests
<span class="hljs-built_in">echo <span class="hljs-string">"=== Running smoke tests ==="
APP_URL=<span class="hljs-string">"$APP_URL" ./scripts/smoke-tests.sh

<span class="hljs-built_in">echo <span class="hljs-string">"=== Deploy and test complete ==="

Testing Environment Variables

One of the trickiest parts of Railway testing is environment variables. Railway manages variables per-environment, and misconfigured variables are a common cause of deployment failures that only manifest at runtime.

Here's a validation approach — an endpoint that reports which expected variables are present (without exposing their values):

// env-check.js
const REQUIRED_VARS = [
  'DATABASE_URL',
  'REDIS_URL',
  'SECRET_KEY',
  'STRIPE_SECRET_KEY',
  'SENDGRID_API_KEY'
];

const OPTIONAL_VARS = [
  'SENTRY_DSN',
  'ANALYTICS_KEY',
  'FEATURE_FLAGS'
];

app.get('/health/env', (req, res) => {
  // Only expose this endpoint in non-production, or behind auth
  if (process.env.NODE_ENV === 'production' && !req.headers['x-internal-check']) {
    return res.status(403).json({ error: 'Forbidden' });
  }

  const required = {};
  let allPresent = true;

  for (const varName of REQUIRED_VARS) {
    const present = !!process.env[varName];
    required[varName] = present ? 'SET' : 'MISSING';
    if (!present) allPresent = false;
  }

  const optional = {};
  for (const varName of OPTIONAL_VARS) {
    optional[varName] = process.env[varName] ? 'SET' : 'NOT SET';
  }

  res.status(allPresent ? 200 : 503).json({
    status: allPresent ? 'ok' : 'missing_vars',
    environment: process.env.RAILWAY_ENVIRONMENT_NAME,
    required,
    optional
  });
});

Test it in CI:

ENV_STATUS=$(curl -s -H "X-Internal-Check: ci" <span class="hljs-string">"$APP_URL/health/env")
MISSING=$(<span class="hljs-built_in">echo <span class="hljs-string">"$ENV_STATUS" <span class="hljs-pipe">| jq -r <span class="hljs-string">'.required | to_entries[] <span class="hljs-pipe">| select(.value == "MISSING") <span class="hljs-pipe">| .key')

<span class="hljs-keyword">if [ -n <span class="hljs-string">"$MISSING" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Missing required environment variables:"
  <span class="hljs-built_in">echo <span class="hljs-string">"$MISSING"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

Multi-Service Integration Testing

Railway's project model makes it natural to run multi-service integration tests. When all your services deploy together in a preview environment, you can test flows that cross service boundaries.

Here's a test structure for a multi-service Railway project:

// tests/integration/multi-service.test.js
const BASE_URL = process.env.TEST_BASE_URL;

describe('Multi-service integration', () => {
  test('API can reach background worker via job queue', async () => {
    // Trigger a background job through the API
    const jobResponse = await fetch(`${BASE_URL}/api/jobs`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ type: 'send-email', payload: { to: 'test@example.com' } })
    });

    expect(jobResponse.status).toBe(202);
    const { jobId } = await jobResponse.json();

    // Poll until the worker processes the job
    let jobStatus;
    for (let i = 0; i < 20; i++) {
      await new Promise(r => setTimeout(r, 1000));
      const statusResponse = await fetch(`${BASE_URL}/api/jobs/${jobId}`);
      const data = await statusResponse.json();
      jobStatus = data.status;
      if (['completed', 'failed'].includes(jobStatus)) break;
    }

    expect(jobStatus).toBe('completed');
  });

  test('File upload service stores to shared storage', async () => {
    const formData = new FormData();
    formData.append('file', new Blob(['test content']), 'test.txt');

    const uploadResponse = await fetch(`${BASE_URL}/api/uploads`, {
      method: 'POST',
      body: formData
    });

    expect(uploadResponse.status).toBe(201);
    const { fileUrl } = await uploadResponse.json();

    // Verify the file is accessible
    const fileResponse = await fetch(fileUrl);
    expect(fileResponse.status).toBe(200);
    expect(await fileResponse.text()).toBe('test content');
  });

  test('Internal service health check passes', async () => {
    const response = await fetch(`${BASE_URL}/health/internal`);
    const data = await response.json();
    expect(data.status).toBe('ok');
  });
});

Using Deployment Hooks for Test Triggers

Railway supports deploy hooks — webhooks that fire when a deployment completes. You can use these to trigger external test runs from HelpMeTest or your own test infrastructure.

Configure a webhook in Railway's project settings, then handle it in your CI or test runner:

// webhook-handler.js — receives Railway deployment events
app.post('/webhooks/railway', async (req, res) => {
  const { type, status, environmentName, serviceUrl } = req.body;

  // Acknowledge immediately
  res.json({ received: true });

  if (type === 'DEPLOY' && status === 'SUCCESS') {
    console.log(`Deploy succeeded for ${environmentName}: ${serviceUrl}`);

    // Trigger HelpMeTest run or your test suite
    await triggerTestSuite({
      environment: environmentName,
      url: serviceUrl
    });
  }
});

HelpMeTest can also be configured to run automatically after Railway deployments complete. Its AI-generated tests can be set to run against your preview environment URL, giving you browser-level end-to-end confidence on every pull request — not just curl-based smoke tests.

Continuous Monitoring on Railway

Railway deployments can be stable for days, then fail due to a memory leak, a database connection pool exhaustion, or a change in a third-party API response. Smoke tests only run at deploy time; they won't catch issues that emerge gradually.

For continuous production health monitoring on Railway:

#!/bin/bash
<span class="hljs-comment"># continuous-health-check.sh
<span class="hljs-comment"># Run this as a Railway cron job service

APP_URL=<span class="hljs-string">"${MONITORED_APP_URL}"
ALERT_WEBHOOK=<span class="hljs-string">"${SLACK_WEBHOOK_URL}"

<span class="hljs-function">check_health() {
  <span class="hljs-built_in">local endpoint=<span class="hljs-string">"$1"
  <span class="hljs-built_in">local expected_status=<span class="hljs-string">"${2:-200}"

  ACTUAL=$(curl -s -o /dev/null -w <span class="hljs-string">"%{http_code}" --max-time 10 <span class="hljs-string">"${APP_URL}<span class="hljs-variable">${endpoint}")

  <span class="hljs-keyword">if [ <span class="hljs-string">"$ACTUAL" != <span class="hljs-string">"$expected_status" ]; <span class="hljs-keyword">then
    <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: $endpoint returned <span class="hljs-variable">$ACTUAL (expected <span class="hljs-variable">$expected_status)"
    <span class="hljs-built_in">return 1
  <span class="hljs-keyword">fi
  <span class="hljs-built_in">return 0
}

FAILED=0
check_health <span class="hljs-string">"/health" <span class="hljs-pipe">|| FAILED=$((FAILED + <span class="hljs-number">1))
check_health <span class="hljs-string">"/health/db" <span class="hljs-pipe">|| FAILED=$((FAILED + <span class="hljs-number">1))
check_health <span class="hljs-string">"/health/internal" <span class="hljs-pipe">|| FAILED=$((FAILED + <span class="hljs-number">1))
check_health <span class="hljs-string">"/api/status" <span class="hljs-pipe">|| FAILED=$((FAILED + <span class="hljs-number">1))

<span class="hljs-keyword">if [ <span class="hljs-variable">$FAILED -gt 0 ]; <span class="hljs-keyword">then
  curl -s -X POST <span class="hljs-string">"$ALERT_WEBHOOK" \
    -H <span class="hljs-string">"Content-Type: application/json" \
    -d <span class="hljs-string">"{\"text\": \"Railway health check failed: $FAILED endpoint(s) unhealthy on <span class="hljs-subst">$(date)\"}"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi

<span class="hljs-built_in">echo <span class="hljs-string">"$(date): All health checks passed"

Add this as a Railway service that runs on a cron schedule in your railway.toml:

[deploy]
cronSchedule = "*/5 * * * *"
startCommand = "bash /app/continuous-health-check.sh"

Summary

Railway's architecture rewards teams that invest in testing infrastructure. The key principles:

  • Preview environments are your best tool — test every PR against a real isolated copy of your infrastructure
  • Internal networking tests catch the service-to-service failures that smoke tests miss
  • Railway CLI in CI gives you deployment control and status visibility without relying on UI interactions
  • Deploy hooks bridge Railway's deployment events to your test runners
  • Continuous monitoring catches production drift that only shows up after deploy

With these patterns in place, Railway's developer experience stays fast while your reliability confidence stays high.

Read more