Smoke Test CI/CD Integration: Block Deployments on Critical Failures

01 Oct 2026 — 5 min read

A smoke test suite that doesn't block deployments is just a report. Reports get ignored. Blocked deployments don't.

This guide covers how to wire smoke tests into CI/CD pipelines so failures stop deployments automatically — with patterns for GitHub Actions, GitLab CI, and Jenkins.

The Deployment Gate Model

Smoke tests belong at two points in a deployment pipeline:

Pre-production gate: Before promoting a build from staging to production
Post-deployment verification: After deploying to production, before announcing the release

Many teams implement only the pre-production gate. Both are necessary. A build can pass staging smoke tests and still fail in production due to environment differences, secret misconfigurations, or CDN caching issues.

Build → Unit Tests → Deploy Staging → [SMOKE GATE] → Deploy Production → [SMOKE VERIFY]

If the staging smoke gate fails: rollback the staging deployment, alert the team, block production. If the production smoke verify fails: trigger an immediate rollback, page on-call, open a P0 incident.

GitHub Actions: Complete Smoke Gate Setup

name: Deploy with Smoke Gates

on:
  push:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: npm run build
      - name: Unit tests
        run: npm test

  deploy-staging:
    needs: build-and-test
    runs-on: ubuntu-latest
    environment: staging
    outputs:
      deployment-url: ${{ steps.deploy.outputs.url }}
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging
        id: deploy
        run: |
          URL=$(./deploy.sh staging)
          echo "url=$URL" >> $GITHUB_OUTPUT

  smoke-gate-staging:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Install Playwright
        run: npx playwright install --with-deps chromium
      - name: Run smoke tests
        run: npx playwright test --project=smoke
        env:
          BASE_URL: ${{ needs.deploy-staging.outputs.deployment-url }}
          SMOKE_USER_EMAIL: ${{ secrets.SMOKE_USER_EMAIL }}
          SMOKE_USER_PASSWORD: ${{ secrets.SMOKE_USER_PASSWORD }}
      - name: Upload failure screenshots
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-test-failures
          path: test-results/

  deploy-production:
    needs: smoke-gate-staging
    runs-on: ubuntu-latest
    environment: production
    if: success()
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to production
        run: ./deploy.sh production

  smoke-verify-production:
    needs: deploy-production
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Install Playwright
        run: npx playwright install --with-deps chromium
      - name: Run production smoke tests
        run: npx playwright test --project=smoke
        env:
          BASE_URL: https://app.example.com
          SMOKE_USER_EMAIL: ${{ secrets.PROD_SMOKE_USER_EMAIL }}
          SMOKE_USER_PASSWORD: ${{ secrets.PROD_SMOKE_USER_PASSWORD }}

  rollback-on-failure:
    needs: smoke-verify-production
    runs-on: ubuntu-latest
    if: failure()
    steps:
      - name: Rollback production
        run: ./rollback.sh production
      - name: Alert on-call
        run: ./alert.sh "CRITICAL: Production smoke tests failed. Rollback initiated."
        env:
          PAGERDUTY_TOKEN: ${{ secrets.PAGERDUTY_TOKEN }}

The if: success() on deploy-production ensures production never gets a build that failed staging smoke tests. The rollback job handles the production failure case.

GitLab CI: Pipeline with Smoke Gates

stages:
  - build
  - test
  - deploy-staging
  - smoke-staging
  - deploy-production
  - smoke-production

variables:
  SMOKE_TIMEOUT: "300"

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/

unit-test:
  stage: test
  script:
    - npm test

deploy-staging:
  stage: deploy-staging
  script:
    - STAGING_URL=$(./deploy.sh staging)
    - echo "STAGING_URL=$STAGING_URL" >> deploy.env
  artifacts:
    reports:
      dotenv: deploy.env
  environment:
    name: staging

smoke-staging:
  stage: smoke-staging
  image: mcr.microsoft.com/playwright:v1.44.0-jammy
  script:
    - npm ci
    - npx playwright test --project=smoke
  variables:
    BASE_URL: $STAGING_URL
  artifacts:
    when: always
    paths:
      - test-results/
    reports:
      junit: test-results/junit.xml
  needs:
    - job: deploy-staging
      artifacts: true

deploy-production:
  stage: deploy-production
  script:
    - ./deploy.sh production
  environment:
    name: production
  when: on_success
  needs:
    - smoke-staging

smoke-production:
  stage: smoke-production
  image: mcr.microsoft.com/playwright:v1.44.0-jammy
  script:
    - npm ci
    - npx playwright test --project=smoke
  variables:
    BASE_URL: https://app.example.com
  allow_failure: false
  needs:
    - deploy-production

allow_failure: false on smoke-production means the pipeline fails if production smoke tests fail. Combine with GitLab environments to trigger automatic rollbacks.

Jenkins: Pipeline with Smoke Verification

pipeline {
    agent any
    
    environment {
        SMOKE_USER_EMAIL = credentials('smoke-user-email')
        SMOKE_USER_PASSWORD = credentials('smoke-user-password')
    }
    
    stages {
        stage('Build') {
            steps {
                sh 'npm run build'
            }
        }
        
        stage('Unit Tests') {
            steps {
                sh 'npm test'
            }
        }
        
        stage('Deploy Staging') {
            steps {
                script {
                    env.STAGING_URL = sh(
                        script: './deploy.sh staging',
                        returnStdout: true
                    ).trim()
                }
            }
        }
        
        stage('Smoke Tests - Staging') {
            steps {
                sh '''
                    npx playwright install --with-deps chromium
                    BASE_URL=${STAGING_URL} npx playwright test --project=smoke
                '''
            }
            post {
                always {
                    junit 'test-results/junit.xml'
                    archiveArtifacts artifacts: 'test-results/**', allowEmptyArchive: true
                }
                failure {
                    mail to: 'team@example.com',
                         subject: "Smoke Tests Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                         body: "Staging smoke tests failed. Build blocked from production. ${env.BUILD_URL}"
                }
            }
        }
        
        stage('Deploy Production') {
            when {
                expression { currentBuild.result == null || currentBuild.result == 'SUCCESS' }
            }
            steps {
                sh './deploy.sh production'
            }
        }
        
        stage('Smoke Tests - Production') {
            steps {
                sh 'BASE_URL=https://app.example.com npx playwright test --project=smoke'
            }
            post {
                failure {
                    sh './rollback.sh production'
                    mail to: 'oncall@example.com',
                         subject: "CRITICAL: Production Smoke Failure",
                         body: "Production smoke tests failed. Rollback initiated. ${env.BUILD_URL}"
                }
            }
        }
    }
}

Playwright Configuration for CI/CD

Configure Playwright to behave correctly in CI environments:

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  
  projects: [
    {
      name: 'smoke',
      testMatch: '**/*.smoke.spec.ts',
      use: {
        ...devices['Desktop Chrome'],
        baseURL: process.env.BASE_URL,
        // Shorter timeouts for smoke — fail fast
        actionTimeout: 10000,
        navigationTimeout: 20000,
      },
    },
    {
      name: 'regression',
      testMatch: '**/*.spec.ts',
      testIgnore: '**/*.smoke.spec.ts',
      use: {
        ...devices['Desktop Chrome'],
        baseURL: process.env.BASE_URL,
      },
    },
  ],
  
  // CI-specific settings
  ...(process.env.CI ? {
    workers: 4,
    retries: 1,          // Allow 1 retry in CI for flaky network
    reporter: [
      ['junit', { outputFile: 'test-results/junit.xml' }],
      ['html', { open: 'never' }],
    ],
  } : {
    workers: 1,
    retries: 0,
    reporter: [['html', { open: 'on-failure' }]],
  }),
  
  timeout: 30000,        // 30s global test timeout
  fullyParallel: true,   // Tests run in parallel
  forbidOnly: !!process.env.CI, // Fail if .only in CI
});

Handling Flaky Smoke Tests in CI

Flaky smoke tests are an existential threat to your pipeline. When a smoke test fails intermittently, teams learn to ignore failures — defeating the entire purpose.

Retry strategy: Allow one retry for smoke tests in CI to handle transient network issues:

# GitHub Actions
- name: Run smoke tests
  run: npx playwright test --project=smoke --retries=1

Alert on retries: Track when tests pass on the second attempt — this signals a flaky test that needs investigation:

// In your test reporter, track retry count
test.afterEach(async ({}, testInfo) => {
  if (testInfo.retry > 0 && testInfo.status === 'passed') {
    console.warn(`FLAKY: ${testInfo.title} passed on retry ${testInfo.retry}`);
    // Send to your metrics system
  }
});

Zero tolerance policy: A smoke test that fails more than 1% of the time (without a real bug) gets removed from the smoke suite immediately. Investigate and fix it, then re-add. Never leave flaky tests in the smoke suite.

Post-Deployment Smoke Monitoring

After the initial post-deploy smoke pass, consider running smoke tests on a schedule against production:

# GitHub Actions scheduled smoke monitoring
name: Production Smoke Monitor

on:
  schedule:
    - cron: '*/15 * * * *'  # Every 15 minutes

jobs:
  smoke-monitor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run smoke tests
        run: npx playwright test --project=smoke
        env:
          BASE_URL: https://app.example.com
      - name: Alert on failure
        if: failure()
        run: ./alert.sh "Production smoke monitor detected failures"

This is distinct from the deployment smoke gate — it catches regressions that happen outside deployments (infrastructure issues, third-party API failures, data corruption).

Metrics to Track

Instrument your smoke pipeline to track:

Pass rate by test: Identify consistently flaky tests
Pipeline duration: Alert if smoke tests start taking longer than 5 minutes
Deployment block rate: How often smoke tests prevent bad deploys (this is success, not failure)
MTTR: Mean time to recover after smoke failures

A healthy smoke gate blocks deployments 2-5% of the time. Less than 1% suggests your smoke tests aren't catching real issues. More than 10% suggests the suite is too strict or you have a systemic quality problem.

Summary

Smoke test CI/CD integration comes down to: hard deployment gates (not optional), fast execution with parallelization, one retry for transient failures, zero tolerance for persistent flakiness, and post-deployment verification separate from the pre-deployment gate. The pipeline configuration is boilerplate once you have the smoke suite; the hard part is keeping the suite small, fast, and trustworthy.

Smoke Test CI/CD Integration: Block Deployments on Critical Failures

The Deployment Gate Model

GitHub Actions: Complete Smoke Gate Setup

GitLab CI: Pipeline with Smoke Gates

Jenkins: Pipeline with Smoke Verification

Playwright Configuration for CI/CD

Handling Flaky Smoke Tests in CI

Post-Deployment Smoke Monitoring

Metrics to Track

Summary

Read more

Smoke Testing Automation Strategy: Build Tests That Run in Under 5 Minutes

Smoke Testing After Deployment: Verify Every Release Instantly

Sanity vs Smoke Testing: When to Use Each (And When to Use Both)

Smoke Testing After Deploy: The Essential Checklist