CI/CD

Optimizing CI Pipelines: Test Parallelization Strategies for Large Codebases

HelpMeTest

HelpMeTest

— 9 min read
Optimizing CI Pipelines: Test Parallelization Strategies for Large Codebases

A large codebase accumulates tests the same way it accumulates technical debt — gradually, then suddenly. A test suite that took 3 minutes a year ago now takes 25 minutes, and nobody can pinpoint when it crossed the threshold from "fast enough" to "everyone waits for CI." Parallelization can bring that 25-minute pipeline back under 5 minutes, but only if you apply the right strategies in the right order.

This guide covers the full parallelization stack: measuring where time actually goes, splitting tests by type into parallel jobs, dynamic sharding based on historical durations, running only tests affected by a change, and making caching work for you instead of against you. Each section includes concrete configuration for GitHub Actions, CircleCI, and GitLab CI.

Step 1: Measure Where Time Is Spent

Before changing anything, understand what's actually slow. Most teams assume their E2E tests are the bottleneck and are correct — but often unit tests have grown to 5,000 cases and now take longer than the E2E suite.

Profiling with Jest

# Get per-file timing
jest --json --outputFile=jest-results.json 2>/dev/null

# Extract and sort by duration
node -e "
const r = require('./jest-results.json');
const files = r.testResults.map(t => ({
  file: t.testFilePath.replace(process.cwd(), ''),
  duration: t.perfStats.runtime,
  tests: t.numPassingTests + t.numFailingTests,
})).sort((a, b) => b.duration - a.duration);

console.log('Top 20 slowest test files:');
files.slice(0, 20).forEach(f =>
  console.log(\`\${f.duration}ms\t\${f.tests} tests\t\${f.file}\`)
);
console.log(\`\nTotal: \${files.reduce((s, f) => s + f.duration, 0)}ms\`);
console.log(\`Files: \${files.length}\`);
"

Profiling with Vitest

# Vitest outputs timing to stdout with --reporter=verbose
vitest run --reporter=verbose 2>&1 | grep -E "✓|×|↓" | \
  awk '{print $NF, $0}' | sort -rn | head -20

Measuring CI Job Duration Breakdown

In GitHub Actions, use job summaries to understand where time goes:

- name: Record timings
  run: |
    echo "## Timing Report" >> $GITHUB_STEP_SUMMARY
    echo "| Step | Duration |" >> $GITHUB_STEP_SUMMARY
    echo "|---|---|" >> $GITHUB_STEP_SUMMARY
    echo "| npm ci | ${{ steps.npm-ci.outputs.duration }}s |" >> $GITHUB_STEP_SUMMARY
    echo "| Unit tests | ${{ steps.unit.outputs.duration }}s |" >> $GITHUB_STEP_SUMMARY
    echo "| Integration tests | ${{ steps.integration.outputs.duration }}s |" >> $GITHUB_STEP_SUMMARY

A simpler approach: look at the GitHub Actions job timeline view. The timeline shows each step as a bar — the longest bars are your targets.

Step 2: Split Tests by Type into Parallel Jobs

The highest-leverage parallelization for most codebases is splitting test types into separate jobs that run simultaneously. Unit tests, integration tests, and E2E tests have different dependencies, different runtimes, and different failure characteristics. There's no reason to run them sequentially.

GitHub Actions: Parallel Jobs by Type

# .github/workflows/ci.yml
name: CI

on: [push, pull_request]

jobs:
  unit-tests:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: jest --testPathPattern='\.unit\.test\.' --maxWorkers=4

  integration-tests:
    name: Integration Tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - run: jest --testPathPattern='\.integration\.test\.' --maxWorkers=2
        env:
          DATABASE_URL: postgresql://test:test@localhost:5432/testdb

  e2e-tests:
    name: E2E Tests
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run build
      - run: npx playwright test --shard=${{ matrix.shard }}/3

  lint-and-types:
    name: Lint & Type Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - run: npm run lint & npm run type-check & wait  # Run both in parallel within the job

All four jobs — unit tests, integration tests, E2E tests (sharded 3 ways), and lint/types — run simultaneously. The total pipeline duration is determined by the slowest job, not the sum of all jobs.

CircleCI: Parallel Jobs by Type

# .circleci/config.yml
version: 2.1

orbs:
  node: circleci/node@5.2

executors:
  default:
    docker:
      - image: cimg/node:20.12
    resource_class: medium  # 2 vCPUs, 4GB RAM

jobs:
  unit-tests:
    executor: default
    steps:
      - checkout
      - node/install-packages
      - run:
          name: Unit tests
          command: jest --testPathPattern='\.unit\.' --maxWorkers=2

  integration-tests:
    executor: default
    docker:
      - image: cimg/node:20.12
      - image: cimg/postgres:16.2
        environment:
          POSTGRES_DB: testdb
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
    steps:
      - checkout
      - node/install-packages
      - run:
          name: Wait for Postgres
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Integration tests
          command: jest --testPathPattern='\.integration\.' --maxWorkers=1

  e2e-tests:
    executor: default
    parallelism: 4  # CircleCI native parallelism
    steps:
      - checkout
      - node/install-packages
      - run: npx playwright install chromium
      - run:
          name: E2E tests
          command: |
            # Use CircleCI's built-in test splitting
            TESTS=$(circleci tests glob "tests/e2e/**/*.spec.ts" | \
              circleci tests split --split-by=timings)
            npx playwright test $TESTS

workflows:
  ci:
    jobs:
      - unit-tests
      - integration-tests
      - e2e-tests

CircleCI's parallelism key and circleci tests split --split-by=timings automatically balance tests across parallel containers using stored timing data from previous runs.

GitLab CI: Parallel Jobs by Type

# .gitlab-ci.yml
stages:
  - test

variables:
  NODE_VERSION: "20"

.node-base:
  image: node:20-slim
  cache:
    key: $CI_COMMIT_REF_SLUG
    paths:
      - node_modules/
    policy: pull-push
  before_script:
    - npm ci

unit-tests:
  extends: .node-base
  stage: test
  script:
    - jest --testPathPattern='\.unit\.' --maxWorkers=4
  coverage: '/Statements\s*:\s*(\d+\.?\d*)%/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

integration-tests:
  extends: .node-base
  stage: test
  services:
    - name: postgres:16
      alias: postgres
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test
    DATABASE_URL: postgresql://test:test@postgres:5432/testdb
  script:
    - jest --testPathPattern='\.integration\.' --maxWorkers=2

e2e-tests:
  extends: .node-base
  stage: test
  parallel: 4  # GitLab native parallel jobs
  script:
    - npx playwright install chromium
    - npm run build
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    when: always
    paths:
      - playwright-report/
    expire_in: 1 week

Step 3: Dynamic Sharding Based on Historical Durations

Static sharding by file count leaves slow shards as bottlenecks. Dynamic sharding uses timing data from previous runs to create balanced groups.

Storing and Using Timing Data

# GitHub Actions — store timing data as artifact
jobs:
  test:
    steps:
      - name: Download previous timing data
        uses: actions/cache@v4
        with:
          path: .test-timings.json
          key: test-timings-${{ github.ref }}
          restore-keys: test-timings-

      - name: Run tests with timing output
        run: |
          jest --json --outputFile=jest-results.json
          # Extract timings for next run
          node scripts/extract-timings.js jest-results.json > .test-timings.json

      - name: Cache timing data
        uses: actions/cache@v4
        with:
          path: .test-timings.json
          key: test-timings-${{ github.ref }}
// scripts/shard-by-timing.js
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');

const SHARD_COUNT = parseInt(process.env.SHARD_COUNT || '5');
const SHARD_INDEX = parseInt(process.env.SHARD_INDEX || '1') - 1; // 0-based

// Load timing data or fall back to equal split
let timings = {};
try {
  timings = JSON.parse(fs.readFileSync('.test-timings.json', 'utf-8'));
} catch {
  console.error('No timing data found, using equal distribution');
}

// Get all test files
const allFiles = execSync('jest --listTests', { encoding: 'utf-8' })
  .trim()
  .split('\n');

// Assign durations (default to median for unknown files)
const durations = allFiles.map(f => ({
  file: f,
  duration: timings[f] || median(Object.values(timings)),
}));

// Sort descending by duration (helps with bin-packing)
durations.sort((a, b) => b.duration - a.duration);

// Greedy bin-packing into shards
const shards = Array.from({ length: SHARD_COUNT }, () => ({ files: [], total: 0 }));
for (const item of durations) {
  const minShard = shards.reduce((min, s, i) =>
    s.total < shards[min].total ? i : min, 0);
  shards[minShard].files.push(item.file);
  shards[minShard].total += item.duration;
}

// Output files for the requested shard
const myFiles = shards[SHARD_INDEX].files;
console.log(myFiles.join('\n'));

function median(values) {
  if (!values.length) return 30000; // default 30s
  const sorted = [...values].sort((a, b) => a - b);
  return sorted[Math.floor(sorted.length / 2)];
}
# Use in GitHub Actions
- name: Run tests for my shard
  run: |
    FILES=$(SHARD_COUNT=5 SHARD_INDEX=${{ matrix.shard }} \
      node scripts/shard-by-timing.js | tr '\n' ' ')
    jest $FILES

Step 4: Test Impact Analysis — Only Run Affected Tests

For large codebases, the fastest tests are the ones you don't run. Test impact analysis maps source files to the tests that cover them, so a change to src/auth/login.ts only runs tests/auth/*.test.ts and any other tests that import login.ts.

Jest with --findRelatedTests

# Find and run only tests related to changed files
CHANGED=$(git diff --name-only origin/main | grep -E '\.(ts|tsx|js|jsx)$')
jest --findRelatedTests $CHANGED

In GitHub Actions:

- name: Get changed files
  id: changes
  run: |
    CHANGED=$(git diff --name-only origin/${{ github.base_ref }} | \
      grep -E '\.(ts|tsx|js|jsx)$' | tr '\n' ' ')
    echo "files=$CHANGED" >> $GITHUB_OUTPUT

- name: Run affected tests
  run: |
    if [ -n "${{ steps.changes.outputs.files }}" ]; then
      jest --findRelatedTests ${{ steps.changes.outputs.files }}
    else
      echo "No JS/TS files changed, skipping tests"
    fi

Always Run Full Suite on Main

Test impact analysis is for PR builds only. On main, run the full suite:

- name: Run tests
  run: |
    if [ "${{ github.ref }}" == "refs/heads/main" ]; then
      jest  # Full suite
    else
      CHANGED=$(git diff --name-only origin/main | grep -E '\.(ts|tsx)$' | tr '\n' ' ')
      if [ -n "$CHANGED" ]; then
        jest --findRelatedTests $CHANGED
      else
        jest --passWithNoTests
      fi
    fi

Vitest with --related

# Run only tests covering changed files
vitest run --changed origin/main

# Or specify files explicitly
vitest run --related src/auth/login.ts src/checkout/cart.ts

Step 5: Caching Strategies

Cache misses can add 2-5 minutes to each CI job. Effective caching is as important as parallelization.

Node Modules Caching

# GitHub Actions — optimal cache key
- uses: actions/setup-node@v4
  with:
    node-version: '20'
    cache: 'npm'
    # Automatically caches ~/.npm keyed on package-lock.json hash

# For workspaces/monorepos
- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      node_modules
      packages/*/node_modules
    key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      npm-${{ runner.os }}-

Build Output Caching

If your tests require a build step (e.g., TypeScript compilation, Next.js build):

- name: Cache Next.js build
  uses: actions/cache@v4
  with:
    path: |
      .next/cache
    key: nextjs-${{ runner.os }}-${{ hashFiles('**/*.ts', '**/*.tsx', '**/package-lock.json') }}
    restore-keys: nextjs-${{ runner.os }}-

- name: Build
  run: npm run build  # Uses cached .next/cache if available

Test Result Caching (Skip Unchanged Tests)

For unit tests, you can skip tests whose source files haven't changed:

- name: Cache Jest results
  uses: actions/cache@v4
  with:
    path: .jest-cache
    key: jest-${{ runner.os }}-${{ hashFiles('src/**', 'tests/**') }}
    restore-keys: jest-${{ runner.os }}-

- name: Run tests
  run: jest --cache --cacheDirectory=.jest-cache
  # Jest skips re-running tests when inputs haven't changed

Bringing It All Together: A Real-World Pipeline

Here's a complete pipeline for a monorepo with multiple packages:

name: CI

on:
  push:
    branches: [main]
  pull_request:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
      shared: ${{ steps.filter.outputs.shared }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'packages/api/**'
            web:
              - 'packages/web/**'
            shared:
              - 'packages/shared/**'

  api-tests:
    needs: changes
    if: needs.changes.outputs.api == 'true' || needs.changes.outputs.shared == 'true'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3]
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
        options: --health-cmd pg_isready --health-interval 5s
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - run: |
          cd packages/api
          jest --shard=${{ matrix.shard }}/3 --maxWorkers=2
        env:
          DATABASE_URL: postgresql://test:test@localhost:5432/testdb

  web-unit-tests:
    needs: changes
    if: needs.changes.outputs.web == 'true' || needs.changes.outputs.shared == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - run: cd packages/web && jest --maxWorkers=4

  web-e2e-tests:
    needs: [web-unit-tests]
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - name: Cache Playwright
        uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ hashFiles('package-lock.json') }}
      - run: npx playwright install chromium
      - run: npm run build
      - run: npx playwright test --shard=${{ matrix.shard }}/4
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pw-blob-${{ matrix.shard }}
          path: blob-report

  merge-e2e-reports:
    needs: web-e2e-tests
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          cache: 'npm'
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          pattern: pw-blob-*
          merge-multiple: true
          path: all-blobs
      - run: npx playwright merge-reports --reporter=html ./all-blobs
      - uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report

Measuring the Impact

Track pipeline duration over time to validate that optimizations are working. Store timing data in a structured format and alert when it regresses:

// scripts/check-ci-budget.js
const MAX_PIPELINE_MINUTES = 10;
const actualMinutes = parseFloat(process.env.PIPELINE_DURATION_MINUTES);

if (actualMinutes > MAX_PIPELINE_MINUTES) {
  console.error(
    `CI pipeline took ${actualMinutes}m, exceeding budget of ${MAX_PIPELINE_MINUTES}m`
  );
  process.exit(1);
}

Tools like HelpMeTest can aggregate test results across pipeline runs, showing you which tests are consistently slow, which are flaky, and where your optimization budget is best spent.

Summary

Optimizing a slow CI pipeline follows a clear sequence:

  1. Measure first — profile which files and test types are actually slow before changing anything
  2. Split by type — run unit, integration, and E2E tests as parallel jobs immediately; this is the highest-leverage change
  3. Shard within types — split large test types into multiple parallel jobs using duration-balanced sharding
  4. Test impact analysis — on PRs, only run tests affected by the changed files
  5. Cache aggressively — node_modules, build outputs, and Jest cache all have high ROI

Each strategy compounds. A codebase with 25-minute CI can typically reach 4-5 minutes by applying all five strategies. The work pays back within days for any team running more than a few CI builds per day.

Read more

Start now free