Testing

How to Add Smoke Tests to Your CI/CD Pipeline and Gate Deployments

HelpMeTest

HelpMeTest

— 5 min read
How to Add Smoke Tests to Your CI/CD Pipeline and Gate Deployments

A deployment pipeline without smoke tests is like a fire sprinkler system with no water pressure test. Everything looks fine until the moment you need it.

Smoke tests in CI/CD serve one purpose: stop bad builds before they cause damage. If the app doesn't boot, a critical endpoint returns 500, or the login flow is broken, you want to know in the pipeline — not in production, not from a customer, not from a 3am alert.

This post covers how to wire smoke tests into your pipeline, what to gate on, and what a working implementation looks like.

The Core Concept: The Deployment Gate

A deployment gate is a mandatory check that must pass before the pipeline proceeds. Smoke tests are the most common gate mechanism.

The flow looks like this:

Build → Unit Tests → Deploy to Staging → [SMOKE TESTS] → Deploy to Production
                                                ↓ FAIL
                                         Block deployment
                                         Notify team
                                         Rollback staging

If smoke tests fail, the pipeline stops. Nothing reaches production. The team gets a notification with the specific failure — not a vague "deployment failed" message.

This is the entire value proposition. You're not running smoke tests to find bugs. You're running them to prevent bad code from reaching production automatically.

What to Include in Smoke Tests

The hardest part of smoke testing isn't the automation — it's deciding what to test. Include too little and you miss critical failures. Include too much and your "smoke" test becomes a slow regression suite.

Include these:

  • Application startup (the server responds at all)
  • Authentication (login works with valid credentials, fails gracefully with invalid ones)
  • Navigation to each major section of the app
  • One representative action per core feature (add item to cart, create a record, send a message)
  • Critical third-party integrations (payment provider responds, email service is reachable)

Exclude these:

  • Edge cases and error scenarios (save for regression tests)
  • Performance checks (separate suite)
  • Full user journeys (too slow, too brittle)
  • Anything that requires complex test data setup

Target runtime: Under 5 minutes. If your smoke suite takes longer, cut it. Run the full suite less frequently.

GitHub Actions Example

Here's a complete GitHub Actions workflow that runs smoke tests after staging deployment and blocks production if they fail:

name: Deploy and Smoke Test

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build application
        run: npm run build

  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: ./scripts/deploy.sh staging
      - name: Wait for staging to be healthy
        run: |
          for i in {1..30}; do
            curl -sf https://staging.yourapp.com/health && break
            echo "Waiting for staging... ($i/30)"
            sleep 10
          done

  smoke-tests:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install dependencies
        run: npm ci
      - name: Run smoke tests
        run: npx playwright test tests/smoke/
        env:
          BASE_URL: https://staging.yourapp.com
          TEST_USER_EMAIL: ${{ secrets.SMOKE_TEST_EMAIL }}
          TEST_USER_PASSWORD: ${{ secrets.SMOKE_TEST_PASSWORD }}
      - name: Upload test results on failure
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: smoke-test-results
          path: playwright-report/

  deploy-production:
    needs: smoke-tests
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: ./scripts/deploy.sh production

The needs: smoke-tests line is the gate. Production deployment only runs if smoke tests pass.

Smoke Test Structure (Playwright)

Here's what a smoke test file looks like in Playwright:

// tests/smoke/critical-paths.spec.js
import { test, expect } from '@playwright/test';

test.describe('Smoke Tests', () => {
  test('homepage loads', async ({ page }) => {
    await page.goto('/');
    await expect(page).toHaveTitle(/YourApp/);
    await expect(page.locator('nav')).toBeVisible();
  });

  test('login flow works', async ({ page }) => {
    await page.goto('/login');
    await page.fill('[data-testid="email"]', process.env.TEST_USER_EMAIL);
    await page.fill('[data-testid="password"]', process.env.TEST_USER_PASSWORD);
    await page.click('[data-testid="login-button"]');
    await expect(page).toHaveURL(/dashboard/);
    await expect(page.locator('[data-testid="user-menu"]')).toBeVisible();
  });

  test('core feature is accessible', async ({ page }) => {
    // Log in first
    await page.goto('/login');
    await page.fill('[data-testid="email"]', process.env.TEST_USER_EMAIL);
    await page.fill('[data-testid="password"]', process.env.TEST_USER_PASSWORD);
    await page.click('[data-testid="login-button"]');

    // Navigate to core feature
    await page.click('[data-testid="main-nav-feature"]');
    await expect(page).toHaveURL(/\/feature/);
    await expect(page.locator('[data-testid="feature-container"]')).toBeVisible();
  });

  test('API health endpoint responds', async ({ request }) => {
    const response = await request.get('/api/health');
    expect(response.status()).toBe(200);
    const body = await response.json();
    expect(body.status).toBe('ok');
  });
});

Keep each test focused. No complex setup, no shared state between tests, no elaborate fixtures.

Handling Smoke Test Failures

When a smoke test fails, the pipeline should:

  1. Stop immediately. Don't run the remaining smoke tests if a fundamental check fails. Use bail: 1 in your Playwright config to exit after the first failure.
  2. Preserve evidence. Upload screenshots, videos, and logs as build artifacts. The person investigating needs to see what failed, not just that it failed.
  3. Notify the right people. Post to Slack or send an email with the test name, the error message, and a link to the artifact. Not just "build failed."
  4. Rollback if needed. For staging environments, a failed smoke test should trigger a rollback to the previous known-good version so the environment stays usable for other work.
- name: Notify Slack on smoke test failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "Smoke tests failed on staging :red_circle:",
        "blocks": [{
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "*Smoke tests failed* for commit `${{ github.sha }}`\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View results>"
          }
        }]
      }
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Dedicated Smoke Test User Accounts

Never run smoke tests with real user accounts or production data. Create dedicated test accounts:

  • Use a consistent email pattern: smoke-test@yourapp.com
  • Keep these accounts in a separate environment or mark them as test accounts in your database
  • Rotate credentials regularly and store them as CI/CD secrets, never in code
  • If your app has roles, create one test account per role your smoke tests need to cover

Running Smoke Tests Against Production

Smoke tests don't only belong in staging pipelines. Running a lightweight smoke test against production after deployment is called a post-deployment verification and it catches deployment-specific issues that staging doesn't replicate (infrastructure differences, CDN caching issues, environment variable misconfigurations).

Keep production smoke tests read-only. Don't create or modify data. Only check that things load and respond correctly.

// tests/smoke/post-deploy-readonly.spec.js
test('production homepage responds', async ({ page }) => {
  await page.goto('https://yourapp.com');
  await expect(page).toHaveTitle(/YourApp/);
});

test('production API is alive', async ({ request }) => {
  const response = await request.get('https://api.yourapp.com/health');
  expect(response.status()).toBe(200);
});

Using HelpMeTest for Smoke Tests

HelpMeTest lets you define smoke tests in natural language and run them on a schedule or triggered by webhooks. After a deployment, your CI can call the HelpMeTest API to kick off the smoke suite:

curl -X POST https://api.helpmetest.com/v1/runs \
  -H "Authorization: Bearer $HELPMETEST_API_KEY" \
  -d <span class="hljs-string">'{"suite": "smoke", "environment": "staging"}'

HelpMeTest handles the browser automation, retries on flaky network conditions, and returns a pass/fail result your pipeline can act on. Tests written in plain English are easier to maintain as the app evolves — no JavaScript expertise required.

Summary

Smoke tests in CI/CD work when they are:

  • Fast — under 5 minutes total
  • Automatic — triggered on every deployment, no manual step
  • Blocking — production deploys don't happen if smoke tests fail
  • Loud — failures produce clear notifications with evidence
  • Maintained — updated when the app changes, not treated as fire-and-forget

Add them once, run them forever. The first time they block a bad deployment from reaching production, they'll have paid for themselves.

Read more