CI/CD Testing

GitLab CI Matrix and Parallel Testing: Sharding Test Suites for Speed

HelpMeTest

HelpMeTest

— 5 min read
GitLab CI Matrix and Parallel Testing: Sharding Test Suites for Speed

A test suite that takes 45 minutes to run sequentially can run in 5 minutes when split across 10 parallel jobs. GitLab CI's parallel keyword and matrix jobs provide the infrastructure for this — but making parallel testing work well requires thinking carefully about how you split your tests and aggregate results.

This guide covers GitLab CI's parallelism features in depth, with practical examples for different testing frameworks.

The parallel Keyword

The simplest form of parallelism in GitLab CI is the parallel keyword. When you set parallel: N on a job, GitLab runs N simultaneous copies of that job, each with environment variables indicating its position in the group:

test:
  stage: test
  image: ruby:3.2
  parallel: 5
  script:
    - bundle exec rspec
  variables:
    CI_NODE_TOTAL: $CI_NODE_TOTAL
    CI_NODE_INDEX: $CI_NODE_INDEX

The automatically injected variables:

  • CI_NODE_TOTAL: Total number of parallel jobs (5 in this example)
  • CI_NODE_INDEX: This job's index (1 through N)

These variables are available to your test script. Most CI-aware test sharding tools can use them directly.

RSpec with knapsack_pro:

test:
  stage: test
  image: ruby:3.2
  parallel: 8
  script:
    - bundle exec rake knapsack_pro:rspec
  variables:
    KNAPSACK_PRO_CI_NODE_TOTAL: $CI_NODE_TOTAL
    KNAPSACK_PRO_CI_NODE_INDEX: $CI_NODE_INDEX

Pytest with pytest-split:

pytest:
  stage: test
  image: python:3.11
  parallel: 6
  script:
    - pip install pytest pytest-split
    - pytest --splits $CI_NODE_TOTAL --group $CI_NODE_INDEX
  artifacts:
    reports:
      junit: report.xml

Jest with shard option (Jest 27+):

jest:
  stage: test
  image: node:20
  parallel: 4
  script:
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    reports:
      junit: junit.xml

Vitest sharding:

vitest:
  stage: test
  image: node:20
  parallel: 4
  script:
    - npx vitest --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL --reporter=junit --outputFile=junit-$CI_NODE_INDEX.xml
  artifacts:
    reports:
      junit: junit-$CI_NODE_INDEX.xml

Matrix Jobs

While parallel runs the same job N times with an index, matrix runs jobs with different combinations of variables. This is useful for testing across multiple versions, configurations, or environments:

Testing across multiple language versions:

test:
  stage: test
  parallel:
    matrix:
      - RUBY_VERSION: ["3.1", "3.2", "3.3"]
  image: ruby:$RUBY_VERSION
  script:
    - ruby --version
    - bundle exec rspec

This creates three jobs: one for each Ruby version.

Testing across multiple databases:

integration-test:
  stage: test
  parallel:
    matrix:
      - DB: ["postgres:14", "postgres:15", "mysql:8.0"]
  services:
    - $DB
  script:
    - bundle exec rspec spec/integration/

Multiple dimensions — matrix expands to all combinations:

test:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: ["18", "20", "22"]
        OS: ["ubuntu", "alpine"]
  image: node:${NODE_VERSION}-${OS}
  script:
    - node --version
    - npm test

This creates 6 jobs: 3 Node versions × 2 OS variants.

Matrix with test sharding:

You can combine matrix for environment combinations with test sharding for speed:

e2e-test:
  stage: test
  parallel:
    matrix:
      - BROWSER: ["chrome", "firefox", "safari"]
        SHARD: ["1/3", "2/3", "3/3"]
  script:
    - npx playwright test --project=$BROWSER --shard=$SHARD
  artifacts:
    when: always
    paths:
      - playwright-report/

This creates 9 jobs: each browser × 3 shards.

Intelligent Test Sharding

Random or sequential test splitting often produces unbalanced shards — one shard finishes in 3 minutes while another takes 12, making the overall time determined by the slowest shard. Intelligent sharding uses historical timing data to split tests into equal-duration groups.

Knapsack Pro (commercial, supports many frameworks):

rspec:
  parallel: 10
  script:
    - bundle exec rake knapsack_pro:rspec

Knapsack Pro tracks per-test timing, allocates tests to minimize the slowest shard's duration, and adapts to test time changes over time.

pytest-split with stored timing:

pytest:
  parallel: 6
  script:
    # First run generates .test_durations
    - pytest --splits $CI_NODE_TOTAL --group $CI_NODE_INDEX --store-durations
  artifacts:
    paths:
      - .test_durations
    expire_in: 1 week
  cache:
    key: pytest-durations-$CI_COMMIT_REF_SLUG
    paths:
      - .test_durations

On subsequent runs, pytest-split uses the stored durations to balance splits.

Manual sharding by test file:

For frameworks without built-in sharding support, distribute test files manually:

#!/bin/bash
# scripts/get-test-files.sh
ALL_FILES=($(find spec -name "*_spec.rb" | sort))
TOTAL=${#ALL_FILES[@]}
FILES_PER_NODE=$(( (TOTAL + CI_NODE_TOTAL - 1) / CI_NODE_TOTAL ))
START=$(( (CI_NODE_INDEX - 1) * FILES_PER_NODE ))
END=$(( START + FILES_PER_NODE ))

echo "${ALL_FILES[@]:$START:$FILES_PER_NODE}"
test:
  parallel: 5
  script:
    - TEST_FILES=$(bash scripts/get-test-files.sh)
    - bundle exec rspec $TEST_FILES

Aggregating Test Results

When tests run in parallel, test results are split across N jobs. GitLab's JUnit report integration aggregates these automatically when each job uploads a JUnit XML artifact:

test:
  parallel: 6
  script:
    - pytest --splits $CI_NODE_TOTAL --group $CI_NODE_INDEX \
        --junitxml=report-$CI_NODE_INDEX.xml
  artifacts:
    when: always
    reports:
      junit: report-$CI_NODE_INDEX.xml
    expire_in: 1 week

GitLab merges all JUnit reports from parallel jobs and shows the combined results in the merge request test summary. Failed tests appear regardless of which shard they ran in.

Coverage aggregation is more complex. Each shard produces a partial coverage report. Combining them requires a merge step:

test:
  parallel: 4
  script:
    - pytest --splits $CI_NODE_TOTAL --group $CI_NODE_INDEX \
        --cov=app --cov-report=xml:coverage-$CI_NODE_INDEX.xml
  artifacts:
    paths:
      - coverage-*.xml

coverage:
  stage: report
  needs:
    - test
  script:
    - pip install coverage
    - coverage combine
    - coverage report
    - coverage xml -o coverage.xml
  coverage: '/TOTAL.+?(\d+%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

For Istanbul/NYC coverage in JavaScript:

merge-coverage:
  stage: report
  needs:
    - job: test
      artifacts: true
  script:
    - npx nyc merge .nyc_output merged-coverage.json
    - npx nyc report --reporter=lcov --reporter=text
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

Parallel Database Testing

Tests that touch a database need either isolation (each test runs in a transaction that gets rolled back) or separate databases per shard.

Using separate databases per shard (Rails example):

test:
  parallel: 5
  services:
    - postgres:14
  variables:
    DATABASE_URL: postgresql://postgres@postgres/testdb_$CI_NODE_INDEX
  before_script:
    - RAILS_ENV=test bundle exec rails db:create db:schema:load
  script:
    - bundle exec rspec

Each shard creates its own database, preventing cross-shard interference. The cost is N separate database setup operations.

Using transaction rollback (single database, works for most test suites):

test:
  parallel: 5
  services:
    - postgres:14
  variables:
    DATABASE_URL: postgresql://postgres@postgres/testdb
  before_script:
    - RAILS_ENV=test bundle exec rails db:create db:schema:load
  script:
    - bundle exec rspec  # Uses DatabaseCleaner with :transaction strategy

With transaction-based cleaning, all shards can share the same database — each test runs in a transaction that never commits. This requires your test framework to support this pattern (DatabaseCleaner for Rails, TestContainers for Java, etc.).

Controlling Parallelism with needs and DAGs

Parallel jobs can create pipeline bottlenecks if not organized correctly. GitLab CI's DAG (Directed Acyclic Graph) via needs lets you start jobs as soon as their dependencies complete:

stages:
  - build
  - test
  - report

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/

unit-test:
  stage: test
  needs: [build]
  parallel: 4
  script:
    - npm run test:unit -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    reports:
      junit: junit.xml

e2e-test:
  stage: test
  needs: [build]
  parallel: 3
  script:
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    paths:
      - playwright-report/

report:
  stage: report
  needs:
    - job: unit-test
      artifacts: true
    - job: e2e-test
      artifacts: true
  script:
    - merge_reports.sh

unit-test and e2e-test both start immediately after build finishes, running their parallel shards simultaneously. The report job collects artifacts from all shards.

Caching for Parallel Jobs

Parallel jobs need to be careful with shared caches. If multiple jobs try to write the same cache key simultaneously, the result is non-deterministic.

For read-heavy caches (npm packages, gem dependencies), sharing works fine:

test:
  parallel: 5
  cache:
    key: node-modules-$CI_COMMIT_REF_SLUG
    paths:
      - node_modules/
    policy: pull  # Never write, only read
  before_script:
    - npm ci  # Install from cache if available

For write-heavy caches (test timing data, generated fixtures), use per-shard cache keys:

test:
  parallel: 5
  cache:
    key: test-timings-$CI_NODE_INDEX-$CI_COMMIT_REF_SLUG
    paths:
      - .test_durations

Measuring Parallelization Effectiveness

Before adding parallelism, measure your test suite's current distribution. If your tests are already fast (under 5 minutes), parallelization overhead may not be worth it. The overhead includes:

  • Job startup time (typically 30-60 seconds for Docker-based jobs)
  • Dependency installation time (mitigated by caching)
  • Test reporting/artifact upload time

For a test suite that takes 20 minutes sequentially:

  • 4 parallel jobs: ~5 minutes test time + 1 minute overhead = ~6 minutes total
  • 10 parallel jobs: ~2 minutes test time + 1 minute overhead = ~3 minutes total
  • Diminishing returns beyond the point where test time equals overhead

Check your pipeline analytics in GitLab (CI/CD > Analytics > Pipeline Charts) to measure the actual impact of parallelization changes over time.

Read more

Start now free