Delta Testing: Focus Test Execution on What Actually Changed

Running your entire test suite on every commit is safe but slow. A 45-minute test run discourages frequent commits, creates a bottleneck in CI, and provides little additional safety over a well-selected subset of tests. Delta testing — also called change-based testing or test impact analysis — runs only the tests that cover code affected by a given change.

The premise: if you changed files A and B, only tests that exercise code paths through A and B can possibly catch a regression in those files. Running tests for C, D, and E is waste.

What Delta Testing Is

Delta testing is a technique, not a specific tool. The core process:

1. Determine the change — which files, functions, or modules changed in this commit or PR
2. Map changes to tests — which tests exercise the changed code paths
3. Run only those tests — skip everything else
4. Fall back to full suite for changes to shared infrastructure (database migrations, core utilities, configuration)

The "map changes to tests" step requires knowing which tests cover which code. This comes from code coverage data: run your full suite with coverage instrumentation, record which tests hit which lines, store that mapping. On subsequent runs, look up which tests cover the changed lines.

Test Impact Analysis in Practice

Python with pytest-testmon

pytest-testmon maintains a database of test-to-coverage mappings and re-runs only affected tests:

pip install pytest-testmon

First run (builds the mapping):

pytest --testmon

Subsequent runs (after making changes):

pytest --testmon
# Only tests covering changed code run
# Output: "collecting ... 3/87 tests run"

pytest-testmon stores coverage data in .testmondata alongside your tests. Commit this file to version control — CI will use the stored mapping rather than re-running everything on the first CI build.

# pytest.ini
[pytest]
addopts = --testmon

One flag in your pytest configuration, and every test run automatically limits execution to affected tests.

JavaScript with Jest --onlyFailures and --changedSince

Jest has built-in support for running tests related to changed files:

# Run tests related to files changed since main branch
jest --changedSince main

# Run tests related to uncommitted changes
jest --onlyChanged

Jest uses static dependency analysis (import/require graph traversal) rather than coverage data. It's faster to set up but less precise — it can include tests that import a changed module but don't actually exercise the changed function.

For more precise impact analysis with Jest, the --collectCoverage data from a previous run can feed into custom scripts:

# Run full suite once to collect coverage
jest --coverage --json --outputFile=coverage-report.json

# Use coverage data to determine impacted tests
node scripts/impacted-tests.js --diff="$(git diff HEAD main --name-only)" \
     --coverage=coverage-report.json

Bazel Build System

Bazel (used at Google, adopted by large monorepos) has delta testing built into its dependency graph. Every build target explicitly declares its dependencies, so Bazel can determine with certainty which tests need to rerun after any change:

# BUILD file
py_library(
    name = "payments",
    srcs = ["payments.py"],
    deps = ["//billing:invoice"],
)

py_test(
    name = "payments_test",
    srcs = ["payments_test.py"],
    deps = [":payments"],
)

# Bazel only rebuilds and re-tests affected targets
bazel test //... --build_event_publish_all_actions

Bazel's approach is highly reliable because the dependency graph is explicit and verified. The tradeoff is the overhead of maintaining BUILD files and migrating to the Bazel ecosystem.

Mapping Code Changes to Test Impact

For teams not using Bazel, the coverage-based approach requires a few components:

Coverage collection — run the full test suite with instrumentation:

# Python
pytest --cov=src --cov-report=json

# JavaScript
jest --coverage --coverageReporters=json

# Go
go test ./... -coverprofile=coverage.out

Change detection — determine what changed:

# Changed files vs main branch
git diff --name-only origin/main HEAD

# Changed lines (more precise)
git diff origin/main HEAD --unified=0

Impact mapping — find tests that cover the changed lines:

import json

def find_impacted_tests(changed_files, coverage_data):
    """
    coverage_data: pytest-cov JSON output
    changed_files: list of file paths that changed
    """
    impacted = set()
    
    for test_name, test_coverage in coverage_data['tests'].items():
        for covered_file in test_coverage['files']:
            if any(changed in covered_file for changed in changed_files):
                impacted.add(test_name)
    
    return list(impacted)

with open('coverage.json') as f:
    coverage = json.load(f)

changed = ['src/payments.py', 'src/invoice.py']
tests_to_run = find_impacted_tests(changed, coverage)

print('\n'.join(tests_to_run))
# payments_test.py::test_charge_card
# payments_test.py::test_refund
# invoice_test.py::test_generate_invoice

Handling Infrastructure Changes

Delta testing breaks down when changes affect code that everything depends on. A change to your database connection pool, logging framework, or core authentication middleware could affect every test. Heuristics for handling this:

High-impact file detection — if a changed file is imported by more than N% of your test suite, fall back to running the full suite:

FULL_SUITE_THRESHOLD = 0.3  # 30% of tests affected -> run everything

impacted_count = len(tests_to_run)
total_count = len(all_tests)

if impacted_count / total_count > FULL_SUITE_THRESHOLD:
    tests_to_run = all_tests  # Run full suite

Explicit full-suite paths — some files always trigger a full suite run:

# .delta-testing.yml
always_run_full_suite_for:
  - "src/database/**"
  - "src/auth/**"
  - "config/**"
  - "requirements.txt"
  - "package.json"

Migration detection — database schema migrations should always trigger a full suite run, as they can affect any test that touches the database.

CI Pipeline Integration

Delta testing in CI typically involves two stages:

PR/commit stage — fast feedback using delta testing:

# .github/workflows/ci.yml
jobs:
  fast-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Full history for git diff
      - name: Run delta tests
        run: |
          CHANGED=$(git diff --name-only origin/main HEAD)
          python scripts/run-delta-tests.py --changed="$CHANGED"

Main branch / nightly stage — full suite to catch any gaps in the impact mapping:

  full-tests:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Run full test suite
        run: pytest

This gives developers fast feedback (2-5 minutes instead of 45) while maintaining confidence that the full suite still passes before anything ships to production.

What Delta Testing Doesn't Catch

Delta testing has a known gap: it can't catch emergent failures between unchanged components. If module A and module B both pass their individual tests, but the interaction between them breaks when both are deployed together, delta testing won't catch it unless a test specifically covers that interaction and both modules are in the change set.

This is why delta testing is a speed optimization, not a replacement for full suite runs. Run delta tests on every commit for fast feedback; run full suite before merges to main or before deployments.

For production failures that slip through test gaps, HelpMeTest provides continuous API monitoring that catches behavioral regressions as soon as they appear in production — a safety net that operates independently of your test coverage.

When Delta Testing Is Worth It

Delta testing pays off when your full test suite takes more than 5-10 minutes and you want faster CI feedback. Below that threshold, the tooling overhead isn't worth it. Above 15-20 minutes, developer experience degrades enough that delta testing becomes essential for productivity.

The implementation investment is moderate: one full suite run with coverage instrumentation, a mapping script, and a CI step that uses the map. For Python with pytest-testmon or JavaScript with Jest's --changedSince, the investment is even lower — near-zero configuration for meaningful speedup.