Testing

Mutation Testing in CI Pipelines: How to Run It Without Slowing Down Your Build

HelpMeTest

HelpMeTest

— 5 min read
Mutation Testing in CI Pipelines: How to Run It Without Slowing Down Your Build

Mutation testing is valuable. It's also slow. A medium-sized Java project can take 45 minutes to run PIT against every class. A TypeScript monorepo with Stryker can run for hours.

This is why most teams that try mutation testing in CI give up after a week. But the slowness isn't a law of nature — it's a configuration problem. Here's how to run mutation testing in CI in a way that doesn't make your team hate you.

Why Mutation Testing Is Slow

Understanding the source of slowness helps you target the right optimizations.

Mutation testing runs your test suite once per mutant. A project with 500 mutants runs your tests 500 times. If your test suite takes 2 minutes, that's 1,000 minutes of test execution — over 16 hours.

Three factors control runtime:

  1. Number of mutants generated — scales with lines of code and enabled mutators
  2. Test suite runtime per mutant — the per-execution cost
  3. Test-to-mutant mapping efficiency — whether the tool runs only relevant tests per mutant

All three are controllable.

Strategy 1: Only Mutate Changed Code

The biggest win comes from not re-running mutations on code that hasn't changed. Most mutation tools support incremental mode — storing previous results and skipping unchanged files.

Stryker (JavaScript/TypeScript):

{
  "incremental": true,
  "incrementalFile": ".stryker-tmp/incremental.json"
}

Cache the incremental file in CI:

# GitHub Actions
- name: Cache Stryker incremental results
  uses: actions/cache@v3
  with:
    path: .stryker-tmp/incremental.json
    key: stryker-incremental-${{ github.ref }}-${{ hashFiles('src/**') }}
    restore-keys: |
      stryker-incremental-${{ github.ref }}-
      stryker-incremental-main-

- name: Run mutation tests
  run: npx stryker run

On a typical PR that touches 3-5 files, incremental Stryker runs in 5-10 minutes instead of 60.

PIT (Java):

<configuration>
  <withHistory>true</withHistory>
</configuration>

Cache .pitest-history.bin:

- name: Cache PIT history
  uses: actions/cache@v3
  with:
    path: target/.pitest-history.bin
    key: pit-history-${{ github.ref }}-${{ hashFiles('src/**/*.java') }}
    restore-keys: |
      pit-history-${{ github.ref }}-
      pit-history-main-

PIT's incremental mode checks which source files changed at the bytecode level and only re-mutates those. For a PR that changes two service classes, you might go from a 40-minute full run to a 3-minute incremental run.

Strategy 2: Per-Test Coverage Mapping

Both Stryker and PIT support mapping each test to the specific code it covers. When running mutations, they only execute tests that actually cover the mutated code.

Stryker:

{
  "coverageAnalysis": "perTest"
}

Without this, Stryker runs your entire test suite for every mutant. With it, a mutant in PaymentService only runs payment-related tests — not your user management tests, not your notification tests.

PIT: This is enabled by default. PIT always uses line coverage to select relevant tests per mutant.

The speedup depends on your project structure. If you have good test organization (tests in the same package as the code they test), perTest can reduce per-mutant execution time by 80-90%.

Strategy 3: Scope Mutation to High-Value Code

Don't mutate everything. Focus on code where bugs would hurt:

Stryker:

{
  "mutate": [
    "src/billing/**/*.ts",
    "src/auth/**/*.ts",
    "src/domain/**/*.ts",
    "!src/**/*.spec.ts",
    "!src/**/*.dto.ts",
    "!src/**/*.config.ts"
  ]
}

PIT:

<targetClasses>
  <param>com.example.billing.*</param>
  <param>com.example.auth.*</param>
  <param>com.example.domain.*</param>
</targetClasses>
<excludedClasses>
  <param>com.example.*.dto.*</param>
  <param>com.example.*.config.*</param>
  <param>com.example.generated.*</param>
</excludedClasses>

Excluding generated code, DTOs, configuration classes, and utility functions can cut mutant count by 40-60% with minimal loss in meaningful signal.

Strategy 4: Parallel Execution

Both tools support running multiple test processes in parallel.

Stryker:

{
  "concurrency": 4
}

Set to the number of available CPU cores minus one. On a 4-core CI runner, concurrency: 3. On an 8-core runner, concurrency: 7.

If your CI runner has limited memory (2-4GB), lower concurrency to avoid OOM kills — each process runs a full JVM or Node process with test dependencies loaded.

PIT:

<threads>4</threads>

Same guidance: cores minus one.

GitHub Actions runners: Standard ubuntu-latest runners have 2 cores. For mutation testing, use ubuntu-latest with concurrency: 2 or upgrade to a larger runner (4-8 cores) for the mutation job.

Strategy 5: Scheduled vs PR Runs

Not every commit needs mutation testing. A common pattern:

On every PR: Run mutation tests only on changed files, with incremental mode. Fail the PR if mutation score on changed files drops below threshold.

Nightly (scheduled): Run a full mutation sweep across the entire codebase. Report results to a dashboard. Don't block deployments on this, but track trends.

Weekly: Run with ALL mutators (not just DEFAULTS) to catch mutations the default set misses.

GitHub Actions schedule example:

name: Mutation Testing (Full)

on:
  schedule:
    - cron: '0 2 * * 1'  # Every Monday at 2am
  push:
    branches: [main]
    paths:
      - 'src/billing/**'  # Full run when billing changes
      - 'src/auth/**'     # Full run when auth changes

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      - run: npm ci
      - name: Run full mutation tests
        run: npx stryker run --reporters html,dashboard
        env:
          STRYKER_DASHBOARD_API_KEY: ${{ secrets.STRYKER_DASHBOARD_API_KEY }}
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: mutation-report
          path: reports/mutation/

PR integration with incremental mode:

name: Mutation Testing (Incremental)

on:
  pull_request:
    branches: [main]

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Full history for change detection
      - run: npm ci
      - name: Restore Stryker cache
        uses: actions/cache@v3
        with:
          path: .stryker-tmp/incremental.json
          key: stryker-${{ github.base_ref }}-${{ hashFiles('src/**') }}
          restore-keys: stryker-${{ github.base_ref }}-
      - name: Run incremental mutation tests
        run: npx stryker run
      - name: Save Stryker cache
        uses: actions/cache/save@v3
        with:
          path: .stryker-tmp/incremental.json
          key: stryker-${{ github.head_ref }}-${{ hashFiles('src/**') }}

Setting Meaningful Quality Gates

Thresholds without teeth are suggestions. Configure your tool to fail the build when mutation score drops:

Stryker:

{
  "thresholds": {
    "high": 80,
    "low": 65,
    "break": 55
  }
}

The break value exits with code 1 if mutation score drops below it, failing the CI job. This is the gate. Set it lower than your current score, then raise it by 5 points every quarter.

PIT:

<mutationThreshold>70</mutationThreshold>

PIT returns exit code 1 if the threshold is not met.

For incremental PR runs, consider a stricter threshold on new code specifically. A PR that introduces a new module with 45% mutation score on that module should fail, even if the overall project score is 80%.

Practical Timeline

For a team starting from scratch:

Week 1: Install the tool, run full baseline, don't fail CI yet. Just collect data.

Week 2: Enable incremental mode, set break at 10 points below current score. Enable on PRs.

Month 2: Raise the break threshold by 5 points. Scope mutation to critical paths. Run full sweep weekly.

Month 3: Integrate with dashboard for trend tracking. Add per-module thresholds for high-risk code.

The goal is sustainable adoption, not a one-time run. Mutation testing in CI works when the incremental cost per PR is low (2-10 minutes) and the feedback is actionable (a specific list of surviving mutants in your changed files).

Done right, it becomes the most useful quality gate in your pipeline — one that actually measures whether your tests would catch real bugs, not just whether they execute the right lines.

Read more