How to Design an Effective Smoke Test Suite: What to Include and What to Cut

Most smoke test suites fail in one of two ways. Either they're so minimal they miss real problems, or they've grown into a 45-minute regression suite that nobody calls "smoke" with a straight face anymore.

Designing a good smoke test suite requires making deliberate decisions about what goes in and what stays out — and having a framework for those decisions so the suite doesn't drift over time.

The Design Principle: Breadth Over Depth

A smoke test suite should be broad and shallow. It answers: "Is every major area of this application functional?"

It does not answer: "Does every feature work correctly in every scenario?"

That's the job of your regression suite. Smoke tests exist to catch catastrophic failures — the kind where a deployment broke the entire login system, or the database connection is down, or a critical API started returning 500s.

If you keep this principle in mind, most "should we include this?" decisions become straightforward.

The Coverage Model: Feature Areas, Not User Journeys

The most reliable structure for a smoke suite is based on feature areas, not user journeys.

User journeys are tempting because they feel realistic. But long user journeys are brittle — a failure early in the journey blocks you from seeing what's working later, and they're slow to run because each test does so much setup.

Feature area coverage is more robust:

• Authentication — can users log in and out?
• Navigation — does the app's main navigation work?
• Core feature A — can the primary feature be accessed and does it render?
• Core feature B — same check for the next most critical feature
• Data display — does the app show data (even stubbed data)?
• Critical actions — one representative write action (create, submit, send)
• API health — do the backend endpoints respond?

For a typical SaaS application, this is 8–15 tests. Each test is focused on one area and runs independently.

A Practical Checklist for What to Include

Use this checklist when deciding whether a test belongs in your smoke suite:

Include it if:

• Failure would make the application unusable for most users
• The failure would not be obvious from a quick manual check (e.g., background service failures)
• It covers infrastructure that is often misconfigured during deployments (env vars, database connections)
• It verifies a third-party integration that the app cannot function without

Exclude it if:

• The test covers an edge case or error scenario
• The test requires more than 30 seconds to complete on its own
• The test depends on complex data setup or fixtures
• The test covers functionality that is rarely used
• The test is already covered by another smoke test in a redundant way

Mapping Your Application

Before writing any tests, map your application's feature areas. This takes 30 minutes and will save hours of rework.

Create a simple grid:

"Critical" means: if this breaks, users cannot do the core thing they came to your app to do.

Test Isolation: Each Test Stands Alone

Every smoke test should be fully independent. It should not depend on another test having run first, and it should not leave state that other tests depend on.

Why this matters for smoke tests specifically: when a smoke test fails, you want to know exactly what broke. If tests share state, a failure in test 2 might be caused by a side effect from test 1, and you're now debugging test interactions instead of the actual problem.

Practical rules:

• Each test that requires authentication should log in itself (or use a saved auth state that it loads independently)
• Tests that create data should clean up after themselves, or use isolated data namespaces (e.g., smoke-test-user-{timestamp}@test.com)
• Avoid relying on data created by previous test runs

Handling Authentication in Smoke Tests

Authentication is one of the trickiest parts of smoke test design. You need to log in to test most of the application, but you don't want every test to go through the full login flow — that's slow and brittle.

The standard approach is to save authentication state once and reuse it:

// tests/smoke/auth.setup.js
import { chromium } from '@playwright/test';
import path from 'path';

const authFile = path.join(__dirname, '../.auth/smoke-user.json');

async function globalSetup() {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto('/login');
  await page.fill('[data-testid="email"]', process.env.SMOKE_USER_EMAIL);
  await page.fill('[data-testid="password"]', process.env.SMOKE_USER_PASSWORD);
  await page.click('[data-testid="login-button"]');
  await page.waitForURL(/dashboard/);

  // Save auth state
  await page.context().storageState({ path: authFile });
  await browser.close();
}

export default globalSetup;

// playwright.config.js
export default {
  globalSetup: './tests/smoke/auth.setup.js',
  projects: [
    {
      name: 'smoke',
      use: {
        storageState: './tests/.auth/smoke-user.json',
      },
      testMatch: 'tests/smoke/**/*.spec.js',
    },
  ],
};

Now every smoke test starts already authenticated. The login flow itself is tested once in the setup phase.

Dealing With Dynamic Content

Smoke tests break when they assert on dynamic content that changes between runs. A common mistake:

// BAD: asserts on specific count that changes
test('dashboard shows items', async ({ page }) => {
  await page.goto('/dashboard');
  await expect(page.locator('.item-card')).toHaveCount(5);
});

This will fail whenever the data changes. Instead, assert on presence:

// GOOD: asserts that at least one item exists
test('dashboard shows items', async ({ page }) => {
  await page.goto('/dashboard');
  await expect(page.locator('.item-card').first()).toBeVisible();
});

Or better, seed specific test data that your smoke tests own:

// BETTER: use dedicated smoke test data
test('dashboard shows items', async ({ page }) => {
  await page.goto('/dashboard');
  // This account always has exactly the smoke test data
  await expect(page.locator('[data-testid="item-smoke-test-1"]')).toBeVisible();
});

Keeping the Suite From Growing Uncontrolled

Smoke test suites grow because saying "no" to adding tests is hard. Someone fixes a critical bug, and reasonably suggests "we should add a smoke test for this." The next person does the same. Within a year, you have 80 smoke tests and a 40-minute suite.

Two mechanisms prevent this:

1. A time budget. The smoke suite has a hard limit (e.g., 5 minutes). Adding a new test requires either proving it fits within the budget or removing another test. This forces explicit tradeoffs.

2. A purpose test. Before adding a test, answer: "Would this failure make the app unusable for most users?" If the answer is "no, but it's really important," it belongs in the regression suite, not the smoke suite.

Review and Prune Regularly

Schedule a smoke suite review every quarter. Delete tests that:

• Cover features that no longer exist
• Duplicate coverage provided by another test
• Haven't caught a real bug in 12 months
• Consistently fail for flaky reasons (fix or delete — never ignore)

A lean smoke suite that runs reliably in 3 minutes is worth ten times more than a comprehensive suite that takes 30 minutes and has 5% flakiness.

The Final Suite Structure

A well-designed smoke suite for a typical SaaS product looks like this:

tests/smoke/
├── auth.setup.js          # global setup: save auth state
├── 01-health.spec.js      # API health, homepage responds
├── 02-auth.spec.js        # login, logout, session persistence
├── 03-navigation.spec.js  # main nav links, routing
├── 04-core-feature.spec.js # primary feature renders and responds
├── 05-data.spec.js        # data loads and displays
└── 06-integrations.spec.js # critical third-party services respond

Total: 10–15 tests, under 5 minutes, runs on every deployment. This is your early warning system. Everything else — deeper scenarios, edge cases, regression coverage — lives in a separate suite that runs on a different schedule.

Keep the smoke suite small, fast, and unambiguous. When it fails, everyone knows something real broke.