Deployment

Blue-Green Deployment Testing: Smoke Tests, Traffic Cutover & Rollback Validation

HelpMeTest

HelpMeTest

— 7 min read
Blue-Green Deployment Testing: Smoke Tests, Traffic Cutover & Rollback Validation

Blue-green deployment reduces downtime risk by running two identical production environments — blue (live) and green (new). The switch happens at the router level, and rollback is instant: flip traffic back to blue. The testing challenge is making the cutover decision confidently: is green actually ready?

The Blue-Green Testing Problem

A blue-green cutover is irreversible in the moment — users are immediately on the new version. The window to catch issues is before the cutover, not after. Testing must answer:

  • Does the green deployment start correctly and pass health checks?
  • Does green produce the same responses as blue for critical paths?
  • Does green handle the expected load?
  • If something goes wrong, does rollback actually restore green to working state?

Health Check Validation

Before any traffic reaches green, validate it's genuinely healthy:

#!/bin/bash
# validate-green.sh

GREEN_URL="${1:-http://green.internal.example.com}"
MAX_RETRIES=30
RETRY_INTERVAL=5

echo "Validating green deployment at $GREEN_URL"

# Phase 1: Basic health check with retries
echo "Phase 1: Health endpoint"
for i in $(seq 1 $MAX_RETRIES); do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$GREEN_URL/health")
  if [ "$STATUS" = "200" ]; then
    echo "  Health check passed on attempt $i"
    break
  fi
  echo "  Attempt $i: got $STATUS, retrying in ${RETRY_INTERVAL}s..."
  sleep $RETRY_INTERVAL
  [ $i -eq $MAX_RETRIES ] && { echo "FAIL: Health check never passed"; exit 1; }
done

# Phase 2: Deep health — dependencies reachable
echo "Phase 2: Dependency health"
DEEP_HEALTH=$(curl -s "$GREEN_URL/health/deep")
DB_STATUS=$(echo "$DEEP_HEALTH" | jq -r '.database.status')
CACHE_STATUS=$(echo "$DEEP_HEALTH" | jq -r '.cache.status')
QUEUE_STATUS=$(echo "$DEEP_HEALTH" | jq -r '.queue.status')

[ "$DB_STATUS" = "healthy" ] || { echo "FAIL: Database unhealthy: $DB_STATUS"; exit 1; }
[ "$CACHE_STATUS" = "healthy" ] || { echo "FAIL: Cache unhealthy: $CACHE_STATUS"; exit 1; }
[ "$QUEUE_STATUS" = "healthy" ] || { echo "FAIL: Queue unhealthy: $QUEUE_STATUS"; exit 1; }
echo "  All dependencies healthy"

# Phase 3: Version check — is the right version deployed?
echo "Phase 3: Version validation"
DEPLOYED_VERSION=$(curl -s "$GREEN_URL/version" | jq -r '.version')
EXPECTED_VERSION="${EXPECTED_VERSION:-$(git rev-parse --short HEAD)}"
[ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ] || {
  echo "FAIL: Wrong version deployed. Expected $EXPECTED_VERSION, got $DEPLOYED_VERSION"
  exit 1
}
echo "  Version $DEPLOYED_VERSION confirmed"

echo "Green deployment validation PASSED"

Automated Smoke Tests

Smoke tests verify critical paths work on green before the cutover. They're minimal, fast, and focused on the highest-value flows:

// smoke-tests/smoke.test.js
const axios = require('axios');

const GREEN_URL = process.env.GREEN_URL;
const BLUE_URL = process.env.BLUE_URL;

// Auth credentials for smoke test user (created in test environment)
const TEST_EMAIL = process.env.SMOKE_TEST_EMAIL;
const TEST_PASSWORD = process.env.SMOKE_TEST_PASSWORD;

let authToken;

// Setup: authenticate against GREEN
beforeAll(async () => {
  const response = await axios.post(`${GREEN_URL}/api/auth/login`, {
    email: TEST_EMAIL,
    password: TEST_PASSWORD
  });
  authToken = response.data.token;
});

const authHeaders = () => ({ Authorization: `Bearer ${authToken}` });

describe('Critical path smoke tests', () => {
  it('home page loads', async () => {
    const response = await axios.get(`${GREEN_URL}/`);
    expect(response.status).toBe(200);
    expect(response.headers['content-type']).toContain('text/html');
  });

  it('authentication works', async () => {
    // Already tested in beforeAll — verify token is valid
    const response = await axios.get(`${GREEN_URL}/api/me`, {
      headers: authHeaders()
    });
    expect(response.status).toBe(200);
    expect(response.data.email).toBe(TEST_EMAIL);
  });

  it('primary API endpoints respond', async () => {
    const endpoints = [
      '/api/products',
      '/api/categories',
      '/api/user/preferences'
    ];

    for (const endpoint of endpoints) {
      const response = await axios.get(`${GREEN_URL}${endpoint}`, {
        headers: authHeaders()
      });
      expect(response.status).toBe(200);
    }
  });

  it('creates and retrieves a resource (data layer is working)', async () => {
    // Create
    const createResponse = await axios.post(
      `${GREEN_URL}/api/items`,
      { name: `smoke-test-${Date.now()}`, category: 'test' },
      { headers: authHeaders() }
    );
    expect(createResponse.status).toBe(201);
    const itemId = createResponse.data.id;

    // Retrieve
    const getResponse = await axios.get(
      `${GREEN_URL}/api/items/${itemId}`,
      { headers: authHeaders() }
    );
    expect(getResponse.status).toBe(200);
    expect(getResponse.data.id).toBe(itemId);

    // Cleanup
    await axios.delete(
      `${GREEN_URL}/api/items/${itemId}`,
      { headers: authHeaders() }
    );
  });

  it('static assets are served', async () => {
    const response = await axios.get(`${GREEN_URL}/static/main.js`);
    expect(response.status).toBe(200);
    expect(response.headers['content-type']).toContain('javascript');
    // Verify cache headers are present
    expect(response.headers['cache-control']).toBeDefined();
  });
});

describe('Blue/green response parity', () => {
  it('product listing returns same schema', async () => {
    const [blueResponse, greenResponse] = await Promise.all([
      axios.get(`${BLUE_URL}/api/products`, { headers: authHeaders() }),
      axios.get(`${GREEN_URL}/api/products`, { headers: authHeaders() })
    ]);

    expect(blueResponse.status).toBe(greenResponse.status);
    // Same top-level keys in response
    const blueKeys = Object.keys(blueResponse.data).sort();
    const greenKeys = Object.keys(greenResponse.data).sort();
    expect(greenKeys).toEqual(blueKeys);
  });

  it('response time within acceptable range', async () => {
    const measurements = [];
    for (let i = 0; i < 5; i++) {
      const start = Date.now();
      await axios.get(`${GREEN_URL}/api/products`, { headers: authHeaders() });
      measurements.push(Date.now() - start);
    }
    const avg = measurements.reduce((a, b) => a + b) / measurements.length;
    // Green must respond within 200ms average (adjust for your SLA)
    expect(avg).toBeLessThan(200);
  });
});

Traffic Cutover Testing

Test the cutover mechanism itself. This is rarely tested — teams assume the router switch works — until it doesn't.

AWS ALB Target Group Swap

#!/bin/bash
# test-cutover.sh

ALB_ARN="arn:aws:elasticloadbalancing:us-east-1:123456789:loadbalancer/app/my-alb/abc123"
LISTENER_ARN="arn:aws:elasticloadbalancing:us-east-1:123456789:listener/app/my-alb/abc123/def456"
BLUE_TG="arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/blue-tg/111"
GREEN_TG="arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/green-tg/222"
PUBLIC_URL="https://api.example.com"

# Step 1: Send continuous requests during cutover (should see zero failures)
echo "Starting continuous request monitor..."
cat > /tmp/monitor.sh << 'MONITOR'
#!/bin/bash
PASS=0; FAIL=0
while true; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$1/health")
  [ "$STATUS" = "200" ] && ((PASS++)) || { ((FAIL++)); echo "$(date): FAIL $STATUS"; }
  sleep 0.1
done
MONITOR
chmod +x /tmp/monitor.sh
bash /tmp/monitor.sh "$PUBLIC_URL" &
MONITOR_PID=$!

# Step 2: Perform cutover
echo "Switching traffic to green..."
aws elbv2 modify-listener \
  --listener-arn "$LISTENER_ARN" \
  --default-actions Type=forward,TargetGroupArn="$GREEN_TG"

echo "Waiting 30s to observe post-cutover behavior..."
sleep 30

# Step 3: Verify traffic is on green
ACTIVE_VERSION=$(curl -s "$PUBLIC_URL/version" | jq -r '.version')
echo "Active version after cutover: $ACTIVE_VERSION"

# Step 4: Stop monitor and check results
kill $MONITOR_PID
wait $MONITOR_PID 2>/dev/null

echo "Done. Check /tmp/monitor output for failures during cutover."

Weighted Traffic Shift Testing (Canary within Blue-Green)

Before full cutover, shift a percentage of traffic:

# 10% to green, 90% stays on blue
aws elbv2 modify-listener \
  --listener-arn "$LISTENER_ARN" \
  --default-actions "Type=forward,ForwardConfig={
    TargetGroups=[
      {TargetGroupArn=$BLUE_TG,Weight=90},
      {TargetGroupArn=$GREEN_TG,Weight=10}
    ]
  }"

# Monitor error rate for 5 minutes
start_time=$(date +%s)
while true; do
  elapsed=$(( $(date +%s) - start_time ))
  [ $elapsed -gt 300 ] && break

  # Check green target group 4xx/5xx counts
  ERROR_COUNT=$(aws cloudwatch get-metric-statistics \
    --namespace AWS/ApplicationELB \
    --metric-name HTTPCode_Target_5XX_Count \
    --dimensions Name=TargetGroup,Value="${GREEN_TG##*/}" \
    --start-time "$(date -u -d '1 minute ago' +%Y-%m-%dT%H:%M:%S)" \
    --end-time "$(date -u +%Y-%m-%dT%H:%M:%S)" \
    --period 60 --statistics Sum \
    | jq '.Datapoints[0].Sum // 0')

  echo "$(date): Green error count last minute: $ERROR_COUNT"

  if [ "$(echo "$ERROR_COUNT > 5" | bc)" -eq 1 ]; then
    echo "ERROR THRESHOLD EXCEEDED — rolling back"
    # Rollback
    aws elbv2 modify-listener \
      --listener-arn "$LISTENER_ARN" \
      --default-actions "Type=forward,TargetGroupArn=$BLUE_TG"
    exit 1
  fi

  sleep 30
done

echo "Canary phase stable — proceed to full cutover"

Rollback Verification

A rollback that has never been tested will fail when you need it most.

#!/bin/bash
# test-rollback.sh — run in staging periodically

BLUE_URL="http://blue.internal.example.com"
GREEN_URL="http://green.internal.example.com"
PUBLIC_URL="https://staging.example.com"
LISTENER_ARN="${LISTENER_ARN}"
BLUE_TG="${BLUE_TG_ARN}"
GREEN_TG="${GREEN_TG_ARN}"

echo "=== Rollback Test ==="

# 1. Verify we know current state
INITIAL_ACTIVE=$(curl -s "$PUBLIC_URL/version" | jq -r '.version')
echo "Initial active version: $INITIAL_ACTIVE"

# 2. Switch to green (simulating a deployment)
echo "Simulating deployment: switching to green..."
aws elbv2 modify-listener \
  --listener-arn "$LISTENER_ARN" \
  --default-actions "Type=forward,TargetGroupArn=$GREEN_TG"

sleep 5

GREEN_VERSION=$(curl -s "$PUBLIC_URL/version" | jq -r '.version')
echo "Green version active: $GREEN_VERSION"

# 3. Perform rollback
echo "Performing rollback..."
ROLLBACK_START=$(date +%s%3N)
aws elbv2 modify-listener \
  --listener-arn "$LISTENER_ARN" \
  --default-actions "Type=forward,TargetGroupArn=$BLUE_TG"
ROLLBACK_END=$(date +%s%3N)
ROLLBACK_TIME=$((ROLLBACK_END - ROLLBACK_START))

echo "Rollback completed in ${ROLLBACK_TIME}ms"

# 4. Verify rollback restored correct version
sleep 2
RESTORED_VERSION=$(curl -s "$PUBLIC_URL/version" | jq -r '.version')

if [ "$RESTORED_VERSION" = "$INITIAL_ACTIVE" ]; then
  echo "PASS: Rollback successful. Version restored: $RESTORED_VERSION"
else
  echo "FAIL: Rollback failed. Expected $INITIAL_ACTIVE, got $RESTORED_VERSION"
  exit 1
fi

# 5. Verify functionality after rollback
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$PUBLIC_URL/health")
[ "$STATUS" = "200" ] || { echo "FAIL: Health check failed after rollback"; exit 1; }
echo "PASS: Health check passes after rollback"
echo "Rollback time: ${ROLLBACK_TIME}ms"

Database Migration Testing in Blue-Green

Database migrations are the trickiest part of blue-green. The safest pattern:

Expand phase (before cutover): add new columns, add new tables. Both blue and green must work with the expanded schema.

Contract phase (after cutover is stable): remove old columns. Only green needs to work.

Test the expand phase before cutover:

// test/migration-compatibility.test.js
describe('Database schema compatibility', () => {
  it('new schema columns are nullable (backwards compatible)', async () => {
    // Connect to test DB with expanded schema
    const result = await db.query(`
      SELECT column_name, is_nullable, column_default
      FROM information_schema.columns
      WHERE table_name = 'users'
      AND column_name = 'new_column'
    `);
    
    expect(result.rows[0].is_nullable).toBe('YES');  // Nullable = blue still works
  });

  it('blue application queries still work with new schema', async () => {
    // Run blue's query patterns against the expanded schema
    const result = await db.query(
      'SELECT id, email, name FROM users WHERE id = $1',
      ['test-user-id']
    );
    expect(result.rows).toHaveLength(1);
  });

  it('green application can read and write new columns', async () => {
    await db.query(
      'UPDATE users SET new_column = $1 WHERE id = $2',
      ['new-value', 'test-user-id']
    );
    const result = await db.query(
      'SELECT new_column FROM users WHERE id = $1',
      ['test-user-id']
    );
    expect(result.rows[0].new_column).toBe('new-value');
  });
});

CI/CD Pipeline Integration

# .github/workflows/blue-green-deploy.yml
name: Blue-Green Deployment

on:
  push:
    branches: [main]

jobs:
  deploy-green:
    runs-on: ubuntu-latest
    outputs:
      green_url: ${{ steps.deploy.outputs.green_url }}
    
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to green environment
        id: deploy
        run: |
          # Deploy green (your deployment mechanism)
          ./scripts/deploy-green.sh
          echo "green_url=$GREEN_URL" >> $GITHUB_OUTPUT

  validate-green:
    needs: deploy-green
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4

      - name: Validate green deployment
        run: bash scripts/validate-green.sh ${{ needs.deploy-green.outputs.green_url }}

      - name: Run smoke tests
        env:
          GREEN_URL: ${{ needs.deploy-green.outputs.green_url }}
          BLUE_URL: ${{ vars.BLUE_URL }}
          SMOKE_TEST_EMAIL: ${{ secrets.SMOKE_TEST_EMAIL }}
          SMOKE_TEST_PASSWORD: ${{ secrets.SMOKE_TEST_PASSWORD }}
        run: npx jest smoke-tests/smoke.test.js --forceExit

  cutover:
    needs: validate-green
    runs-on: ubuntu-latest
    environment: production  # Requires manual approval in GitHub
    
    steps:
      - name: Switch traffic to green
        run: |
          aws elbv2 modify-listener \
            --listener-arn "${{ vars.LISTENER_ARN }}" \
            --default-actions "Type=forward,TargetGroupArn=${{ vars.GREEN_TG_ARN }}"

      - name: Post-cutover validation
        run: |
          # Wait for steady state
          sleep 10
          
          # Verify public URL is now on green
          ACTIVE_VERSION=$(curl -s "${{ vars.PUBLIC_URL }}/version" | jq -r '.version')
          EXPECTED_VERSION=$(git rev-parse --short HEAD)
          
          if [ "$ACTIVE_VERSION" != "$EXPECTED_VERSION" ]; then
            echo "Cutover failed — rolling back"
            aws elbv2 modify-listener \
              --listener-arn "${{ vars.LISTENER_ARN }}" \
              --default-actions "Type=forward,TargetGroupArn=${{ vars.BLUE_TG_ARN }}"
            exit 1
          fi
          
          echo "Cutover successful. Version $ACTIVE_VERSION is live."

Read more

Start now free