CI/CD Test Cost Optimization: Cutting Build Times Without Cutting Coverage

CI/CD testing costs compound fast. A 20-minute test suite running on every push, across a team of 20 engineers, adds up to thousands of compute-hours per month. As coverage grows, build times grow. As build times grow, engineers wait, context-switch, and eventually start skipping tests.

Cost optimization isn't just about money — it's about keeping the test feedback loop fast enough that it actually changes developer behavior.

The Four Levers

Test cost in CI reduces to four variables:

1. How many tests run (test selection)
2. How fast individual tests run (execution efficiency)
3. How much work is repeated (caching)
4. How efficiently work is distributed (parallelization)

Most teams have only worked on #4. The other three have higher ROI for most codebases.

Intelligent Test Selection

The highest-leverage optimization: don't run tests that can't possibly fail given the change.

Change-based test selection

For a commit that only touches payments/stripe.py, there's no reason to run tests for the notification system, the reporting module, or the user authentication flow.

Change-based selection works by maintaining a dependency graph: which tests depend on which source files.

Python example with pytest-testmon:

pip install pytest-testmon

# First run: builds dependency graph
pytest --testmon

# Subsequent runs: only runs tests affected by changes
pytest --testmon

pytest-testmon instruments test execution to build a map from source lines to test cases. When source changes, it identifies the affected tests and runs only those.

JavaScript with Nx:

npx nx affected:test --base=main

Nx builds a dependency graph from your monorepo structure and runs tests only for affected packages.

Bazel:

bazel test //... --build_tests_only

Bazel's strict dependency model makes test selection exact: only rebuild and retest targets whose transitive dependencies changed.

Coverage-based selection

For teams without a dependency graph tool, coverage-based selection approximates the same result:

1. Run the full suite with coverage enabled
2. Map each test to the source lines it covers
3. On each commit, identify changed source lines
4. Run tests whose coverage overlaps with the change

This is less precise than static dependency analysis but requires no build system changes. Tools like Launchable, Predictive Test Selection (PTS) from Gradle, and Microsoft Test Impact Analysis implement this at scale.

Risk-based selection

Not all tests need to run on every commit. A tiered approach:

This is the most common approach and requires the least tooling. Define the tiers explicitly; document which tests belong where. Without explicit rules, "unit test" and "integration test" labels drift.

Remote Caching

If your tests haven't changed, rerunning them is waste. Remote caching stores test results and returns them instead of executing.

How remote caching works

Input: test binary + test inputs + dependencies (hash of all)
Output: test result (pass/fail, stdout, artifacts)

Cache: if hash(inputs) == known_hash → return cached result
       else → execute test, cache result

Any test that ran green on the same code, same dependencies, and same inputs can be cache-hit — no execution needed.

Bazel remote cache

Bazel has native remote caching support:

# .bazelrc
build --remote_cache=grpcs://remotebuildexecution.googleapis.com
build --google_default_credentials
test --remote_cache=grpcs://remotebuildexecution.googleapis.com

With remote cache configured, bazel test //... checks the cache before executing any test. A clean checkout on a known commit gets ~100% cache hit rate — tests "run" in seconds.

Nx remote cache (Nx Cloud)

npx nx-cloud configure --nx-cloud-token=$NX_CLOUD_TOKEN

Nx Cloud caches all Nx task outputs including test results. Pull requests that touch unchanged packages get instant green status.

GitHub Actions caching

For test frameworks without native remote caching, manually cache based on content hashes:

- name: Cache test results
  id: cache-tests
  uses: actions/cache@v3
  with:
    path: .test-results/
    key: tests-${{ hashFiles('src/**', 'tests/**', 'package-lock.json') }}

- name: Run tests
  if: steps.cache-tests.outputs.cache-hit != 'true'
  run: npm test

- name: Use cached results
  if: steps.cache-tests.outputs.cache-hit == 'true'
  run: echo "Tests passed (cached)"

This is coarse — any change invalidates the entire cache — but effective for slow integration test suites.

Flaky Test Quarantine

Flaky tests inflate CI costs in two ways:

1. Retries: a 5% flake rate on 100 tests = ~5 retries per run = 5% more compute
2. Investigation time: every flake requires developer attention

A test that fails 5% of the time and gets retried automatically consumes ~1.05x the compute. At scale (10,000 tests, 100 commits/day), flaky tests add up.

Quarantine strategy

Identify flaky tests by tracking pass/fail rates over time:

# Track results: test_name → [pass, fail, pass, pass, fail, ...]
# Flakiness score: failures / total_runs
# Quarantine threshold: >3% flakiness over last 100 runs

Quarantined tests:

• Don't block CI (treated as skipped)
• Are reported separately in test results
• Get a 30-day SLA to fix or delete

Tools that automate this:

• BuildPulse: tracks flakiness across CI runs
• Trunk Flaky Tests: quarantine + fix workflow
• Allure TestOps: flakiness analytics

The fix-or-delete rule is important. Quarantined tests that sit for 90 days usually need deletion — the feature they tested has diverged enough that the test is no longer valid.

Retry policies

If you must retry flaky tests:

# GitHub Actions
- name: Run tests
  run: pytest tests/
  continue-on-error: true
  id: first-attempt

- name: Retry on failure
  if: steps.first-attempt.outcome == 'failure'
  run: pytest tests/ --last-failed

Only retry once. If a test fails twice, it's not flaky — it's failing. Retrying more than once trains developers to ignore repeated failures.

Test Execution Profiling

Before optimizing, measure. Most test suites have a fat tail: a small number of tests that take a disproportionate fraction of total runtime.

Identifying slow tests

# pytest: slowest 20 tests
pytest --durations=20

# Jest: slow test report
jest --verbose --json > results.json
cat results.json | jq '.testResults[].testFilePath + " " + (.testResults[].testResults | map(.duration) | add | tostring)'

Typical findings:

• Integration tests without proper mocking that make real HTTP calls
• Tests that start a full application server for a few assertions
• Tests with unnecessary sleep() calls for timing

The 80/20 rule for test time

In most test suites, 20% of tests take 80% of the time. Identify and fix the worst offenders:

Unnecessary network calls: Mock external services. A test that calls a real API takes 200–500ms; a mocked call takes < 1ms.

Full app startup: If your integration tests start a full server per test class, refactor to start it once per session. A 5-second startup × 100 test classes = 8 minutes of unnecessary overhead.

Heavy database fixtures: Tests that insert 10,000 rows when 100 would suffice for the assertion. Profile fixture creation time.

Missing test database cleanup: Tests that run sequentially because they all read from shared state. Make them independent, then parallelize.

Parallelization Beyond Simple Sharding

Sharding splits tests into N equal chunks and runs them on N machines. Simple but inefficient if test times are uneven — the slowest shard is the bottleneck.

Time-based sharding

Instead of equal counts, shard by equal estimated time:

# Given test timing history, create shards of equal duration
def shard_by_time(tests, n_shards, timing_data):
    # Sort tests by duration descending
    sorted_tests = sorted(tests, key=lambda t: timing_data.get(t, 0), reverse=True)
    
    # Greedy bin-packing: assign each test to the currently-lightest shard
    shards = [[] for _ in range(n_shards)]
    shard_times = [0] * n_shards
    
    for test in sorted_tests:
        lightest = min(range(n_shards), key=lambda i: shard_times[i])
        shards[lightest].append(test)
        shard_times[lightest] += timing_data.get(test, 0)
    
    return shards

Time-based sharding reduces the gap between fastest and slowest shard, improving total pipeline time.

Tools with built-in time-based sharding: Buildkite Test Splitting, CircleCI test splitting, RSpec::TestQueue.

Test container reuse

For integration tests that start containers (database, cache, message queue), container startup time is often the bottleneck.

Docker Compose pre-warm:

# Start containers before tests, keep them warm across test runs in the same CI job
services:
  postgres:
    image: postgres:16
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 2s
      timeout: 5s
      retries: 10

# Start once at job beginning
docker compose up -d
docker compose wait postgres

# Run all tests (containers stay running)
pytest integration_tests/

# Cleanup at end
docker compose down

Per-test container startup is ~2–5 seconds. Reusing containers reduces this to ~10ms. For 100 integration tests, the difference is 200–500 seconds vs. 1 second of overhead.

CI Cost Monitoring

Optimizations need measurement. Track:

• Cost per PR: compute minutes × cost per minute
• Test suite duration trend: is it growing? By how much per month?
• Cache hit rate: for remote caching, what fraction of tests are cache-hit?
• Flakiness rate: what fraction of test runs require a retry?
• P95 test time: the long tail matters more than mean for developer experience

Set budget alerts. When CI cost for a repository exceeds a threshold, investigate before it compounds further.

Implementation Priority

Start with profiling and flaky test quarantine — both are low effort, high impact, and produce immediate results.

Summary

CI/CD test costs compound without active management. The levers: test selection (don't run unnecessary tests), caching (don't re-run unchanged tests), flaky test quarantine (don't waste retries), and smart parallelization.

The result of applying these systematically: 50–80% reduction in CI compute costs is achievable in most codebases without reducing coverage or sacrificing reliability.

The business case for the investment: faster CI means faster developer feedback, more PRs merged per day, and lower infrastructure spend.