Smoke Testing

Smoke Testing Automation Strategy: Build Tests That Run in Under 5 Minutes

HelpMeTest

HelpMeTest

— 5 min read
Smoke Testing Automation Strategy: Build Tests That Run in Under 5 Minutes

Smoke tests exist for one reason: to answer "is this build worth testing further?" in the shortest time possible. If it takes 20 minutes to know whether the login page loads, your smoke suite is broken by design.

This guide covers how to build an automated smoke testing strategy that runs fast, fails loudly, and integrates cleanly into CI/CD.

What Smoke Testing Actually Is

Smoke testing — originally called "build verification testing" — is a shallow pass over the most critical paths in your system. The name comes from hardware testing: power up a circuit board and see if smoke comes out. If it does, stop. Don't investigate further.

In software, a smoke test answers:

  • Does the application start?
  • Can users log in?
  • Do core navigation paths respond?
  • Are external dependencies (database, APIs) reachable?

Smoke tests are not comprehensive. They deliberately skip edge cases, error paths, and performance checks. Breadth matters more than depth.

The Smoke Test Selection Rule

A test belongs in your smoke suite if failing it means nothing else is worth running. Apply this filter ruthlessly:

Include:

  • Authentication (login/logout)
  • Home page / dashboard load
  • Primary user workflow (place order, create document, send message — whatever your app's core value is)
  • Critical API health endpoints
  • Database connectivity check

Exclude:

  • Input validation edge cases
  • Error message copy
  • UI styling or layout details
  • Report generation
  • Admin-only features
  • Performance benchmarks

If you have more than 20-30 smoke tests, you've included too much. The target execution time is under 5 minutes. Work backwards from that constraint.

Structuring a Smoke Test Suite

Layer 1: Infrastructure Checks (30 seconds)

Before testing application logic, verify infrastructure is up:

# Health endpoint check
curl -f https://app.example.com/health || exit 1

# Database connectivity
curl -f https://app.example.com/health/db || exit 1

# External API dependency
curl -f https://app.example.com/health/integrations || exit 1

Health endpoints should return 200 with a JSON payload indicating connection status. If they don't exist, add them. They're foundational.

Layer 2: Authentication (1 minute)

Every smoke suite needs a working login test. Use a dedicated smoke test account — never production user credentials, never shared test accounts that other tests modify.

// Playwright example
test('smoke: user can log in', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[data-testid="email"]', process.env.SMOKE_USER_EMAIL);
  await page.fill('[data-testid="password"]', process.env.SMOKE_USER_PASSWORD);
  await page.click('[data-testid="login-button"]');
  await expect(page).toHaveURL('/dashboard');
  await expect(page.locator('[data-testid="user-menu"]')).toBeVisible();
});

Store the auth state after login and reuse it across remaining smoke tests rather than re-authenticating each time. This alone can cut smoke suite time by 40%.

Layer 3: Core User Journeys (2-3 minutes)

Test the 2-3 flows that represent your application's primary value. For a SaaS product this might be: create a project, invite a team member, complete the core action.

Keep each journey test short — 5-8 steps maximum. You're verifying paths work, not that every detail is correct.

Layer 4: API Contract Checks (30 seconds)

If your frontend calls a backend API, verify the contracts directly:

test('smoke: API returns expected shape', async ({ request }) => {
  const response = await request.get('/api/v1/user/me', {
    headers: { Authorization: `Bearer ${process.env.SMOKE_API_TOKEN}` }
  });
  expect(response.status()).toBe(200);
  const body = await response.json();
  expect(body).toHaveProperty('id');
  expect(body).toHaveProperty('email');
});

API smoke tests run faster than UI tests and catch backend regressions before the UI layer is even exercised.

Running Smoke Tests in CI/CD

Gate on Every Deployment

Smoke tests should block deployments, not just report. Configure your CI pipeline to:

  1. Deploy to staging
  2. Run smoke tests against staging
  3. Block production promotion if smoke tests fail
  4. Run smoke tests again after production deployment
# GitHub Actions example
jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: ./deploy.sh staging

  smoke-test-staging:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - name: Run smoke tests
        run: npx playwright test --project=smoke
        env:
          BASE_URL: https://staging.example.com
          SMOKE_USER_EMAIL: ${{ secrets.SMOKE_USER_EMAIL }}
          SMOKE_USER_PASSWORD: ${{ secrets.SMOKE_USER_PASSWORD }}

  deploy-production:
    needs: smoke-test-staging
    if: success()
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: ./deploy.sh production

  smoke-test-production:
    needs: deploy-production
    runs-on: ubuntu-latest
    steps:
      - name: Run production smoke tests
        run: npx playwright test --project=smoke
        env:
          BASE_URL: https://app.example.com

Parallelization Strategy

Smoke tests should be independent — no test should depend on state left by another. This enables parallelization:

// playwright.config.ts
export default defineConfig({
  projects: [
    {
      name: 'smoke',
      testMatch: '**/*.smoke.spec.ts',
      use: { baseURL: process.env.BASE_URL },
    }
  ],
  workers: 4, // Run 4 smoke tests in parallel
  timeout: 30000, // Fail fast — 30s max per test
});

With 4 workers and 20 tests averaging 15 seconds each, total execution time is roughly 75 seconds.

Test Data Management for Smoke Tests

Smoke tests need stable, predictable data. Three approaches:

Dedicated smoke test accounts: Create specific user accounts and seed data that smoke tests always use. Never let other tests touch this data. Simple but requires maintenance.

Reset before run: Before smoke tests execute, reset test data to a known state via API or seed script. More reliable but adds setup time.

Read-only tests: Design smoke tests to only read data, never write. Works for read-heavy applications but is restrictive.

The dedicated account approach works for most teams. Create a smoke@company.com user with the minimum data your core flows require, and protect it.

What to Do When Smoke Tests Fail

A smoke test failure means: stop everything.

  1. Block the deployment immediately
  2. Page the on-call engineer (not just post in Slack)
  3. Do not run the full test suite — it's waste until smoke is green
  4. Revert the deployment if smoke fails in production

The severity model matters. A failing smoke test in production is a P0 incident, not a QA ticket.

Smoke Tests vs Monitoring

Smoke tests run at deployment time. Synthetic monitoring runs continuously. Both are needed:

  • Smoke tests: "Is this build safe to ship?"
  • Synthetic monitoring: "Is the system healthy right now?"

Consider running a subset of your smoke tests as synthetic monitors post-deployment — the same tests, but scheduled every 5 minutes against production. This gives you instant regression detection without writing duplicate test code.

Common Smoke Testing Mistakes

Including too many tests. If smoke takes 30 minutes, teams start skipping it. Under 5 minutes is the hard rule.

Flaky tests in the smoke suite. A flaky smoke test is worse than no smoke test — it trains teams to ignore failures. Fix or remove unstable tests immediately.

Testing the wrong layer. Smoke tests should test at the system level (real browser, real API), not unit level. Mocked smoke tests give false confidence.

No dedicated test data. Smoke tests that depend on production data or shared staging data break randomly. Isolated data is mandatory.

Ignoring smoke test failures. If your team has ever said "oh, smoke is just flaky sometimes," your smoke suite has failed its purpose.

Integrating HelpMeTest for Smoke Monitoring

HelpMeTest runs end-to-end browser tests on a schedule — making it a natural fit for continuous smoke monitoring. Define your core smoke scenarios once and run them every 5 minutes against production.

When a deployment breaks a critical flow, you get an alert within minutes rather than waiting for a user to report it. The free plan supports 24/7 monitoring, which covers most smoke monitoring needs.

Summary

A working smoke test strategy requires: a small focused test set (under 30 tests), fast execution (under 5 minutes), hard deployment gates, independent test data, and zero tolerance for flakiness. Everything else — comprehensive functional testing, performance testing, exploratory testing — comes after the smoke suite is green.

Build the suite small and trust it completely. That's the only version that actually works.

Read more

Start now free