CI/CD

CircleCI Testing Guide: Parallel Tests and Coverage Reports

HelpMeTest

HelpMeTest

— 5 min read
CircleCI Testing Guide: Parallel Tests and Coverage Reports

CircleCI is built around developer experience — fast pipelines, intelligent test splitting, and native Docker support. The config.yml format is explicit but powerful. This guide covers how to configure CircleCI for fast, reliable automated testing across unit, integration, and end-to-end tests.

Basic config.yml Structure

CircleCI pipelines are defined in .circleci/config.yml:

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - restore_cache:
          keys:
            - npm-{{ checksum "package-lock.json" }}
      - run:
          name: Install dependencies
          command: npm ci
      - save_cache:
          key: npm-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - run:
          name: Run tests
          command: npm test

workflows:
  build-and-test:
    jobs:
      - test

Every job defines its own Docker image. CircleCI starts a fresh container for each job run.

Caching Dependencies

CircleCI caches are keyed by file checksums. Use the lock file as the key:

- restore_cache:
    keys:
      - npm-v1-{{ checksum "package-lock.json" }}
      - npm-v1-  # fallback: latest cache if exact key misses

- run:
    name: Install
    command: npm ci

- save_cache:
    key: npm-v1-{{ checksum "package-lock.json" }}
    paths:
      - ~/.npm
      - node_modules

The fallback key (npm-v1-) restores the most recent cache matching that prefix when the exact checksum doesn't exist (new lock file, first run). This avoids a full download on every dependency change.

Multiple Docker Containers (Services)

Run services alongside your job with secondary Docker containers:

jobs:
  integration-test:
    docker:
      - image: cimg/node:20.0
      - image: cimg/postgres:16.0
        environment:
          POSTGRES_USER: testuser
          POSTGRES_PASSWORD: testpass
          POSTGRES_DB: testdb
      - image: cimg/redis:7.0
    environment:
      DATABASE_URL: postgresql://testuser:testpass@localhost/testdb
      REDIS_URL: redis://localhost:6379
    steps:
      - checkout
      - restore_cache:
          keys:
            - npm-{{ checksum "package-lock.json" }}
      - run: npm ci
      - run:
          name: Wait for Postgres
          command: |
            dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Run integration tests
          command: npm run test:integration

The first container in the docker list is the primary container where your steps run. Additional containers are accessible on localhost at their default ports. dockerize (included in CircleCI convenience images) waits for services to become available.

Parallelism: The CircleCI Advantage

CircleCI's native test splitting is one of its strongest features. It analyzes historical timing data to split tests evenly across parallel containers:

jobs:
  test:
    docker:
      - image: cimg/node:20.0
    parallelism: 4
    steps:
      - checkout
      - restore_cache:
          keys:
            - npm-{{ checksum "package-lock.json" }}
      - run: npm ci
      - run:
          name: Run tests
          command: |
            TESTFILES=$(circleci tests glob "tests/**/*.test.js" | \
              circleci tests split --split-by=timings)
            npx jest $TESTFILES
      - store_test_results:
          path: test-results

circleci tests split --split-by=timings distributes test files based on how long each file took in previous runs. If timing data doesn't exist yet, it falls back to even file-count splitting.

For pytest:

- run:
    name: Run Python tests
    command: |
      TESTFILES=$(circleci tests glob "tests/**/*.py" | \
        circleci tests split --split-by=timings)
      pytest $TESTFILES --junitxml=test-results/pytest.xml

Storing Test Results

CircleCI displays test results from JUnit XML files in the pipeline UI:

- store_test_results:
    path: test-results

- store_artifacts:
    path: test-results
    destination: test-results

store_test_results parses JUnit XML for CircleCI's built-in test summary. store_artifacts makes the same files downloadable from the artifacts tab.

Configure your test runner to output JUnit XML:

# Jest
- run:
    command: npx jest --reporters=jest-junit
    environment:
      JEST_JUNIT_OUTPUT_DIR: test-results
      JEST_JUNIT_OUTPUT_NAME: results.xml
# Go
- run:
    command: gotestsum --junitfile test-results/go-test.xml ./...

Coverage Reports

- run:
    name: Run tests with coverage
    command: |
      npx jest --coverage --coverageReporters=lcov --coverageReporters=text-summary

- store_artifacts:
    path: coverage
    destination: coverage

- run:
    name: Upload to Codecov
    command: |
      curl -Os https://uploader.codecov.io/latest/linux/codecov
      chmod +x codecov
      ./codecov -t $CODECOV_TOKEN

To enforce coverage thresholds in the pipeline:

- run:
    name: Check coverage threshold
    command: |
      LINES=$(npx jest --coverage --coverageReporters=json-summary 2>/dev/null | \
        jq '.total.lines.pct' coverage/coverage-summary.json)
      if (( $(echo "$LINES < 80" | bc -l) )); then
        echo "Line coverage ${LINES}% is below 80% threshold"
        exit 1
      fi

Workflows: Sequencing and Parallelism

CircleCI workflows define how jobs relate to each other:

workflows:
  test-and-deploy:
    jobs:
      - install
      - unit-test:
          requires:
            - install
      - integration-test:
          requires:
            - install
      - e2e-test:
          requires:
            - unit-test
            - integration-test
      - deploy:
          requires:
            - e2e-test
          filters:
            branches:
              only: main

unit-test and integration-test run in parallel (both require only install). e2e-test waits for both. deploy only runs on the main branch after all tests pass.

Orbs: Reusable Configuration

CircleCI orbs are reusable configuration packages. Instead of writing boilerplate, use community orbs:

version: 2.1

orbs:
  node: circleci/node@5
  playwright: playwright-community/playwright@1

jobs:
  test:
    executor: node/default
    steps:
      - checkout
      - node/install-packages
      - run:
          name: Run unit tests
          command: npm test

  e2e:
    executor: playwright/default
    steps:
      - checkout
      - node/install-packages
      - playwright/install-browsers
      - run:
          name: Run E2E tests
          command: npx playwright test

workflows:
  main:
    jobs:
      - test
      - e2e:
          requires:
            - test

The node orb handles Node.js installation, caching, and package installation. The Playwright orb handles browser installation. Using orbs reduces your config by 60-70% for common setups.

Environment Variables and Contexts

Store secrets in CircleCI project settings or in Organization Contexts for shared secrets:

jobs:
  integration-test:
    steps:
      - run:
          name: Run tests
          command: npm run test:integration
          # Environment variables from project settings are auto-injected

workflows:
  test:
    jobs:
      - integration-test:
          context:
            - shared-secrets  # Organization context with DATABASE_URL, API_KEY

Contexts are shared across projects in your organization. Restrict context access to specific branches or groups using context restrictions.

Flaky Test Detection

CircleCI tracks flaky tests automatically — tests that sometimes pass and sometimes fail in the same build. View them under Insights > Flaky Tests.

For immediate visibility, configure auto-retry on the test step (not recommended for application tests — only for infrastructure flakiness):

- run:
    name: Run tests
    command: npm test
    no_output_timeout: 10m

# For known-flaky infrastructure (not application tests)
- run:
    name: Run E2E with retry
    command: |
      for i in 1 2 3; do
        npx playwright test && break || echo "Attempt $i failed"
      done

Treat flaky tests as bugs. Use CircleCI Insights data to identify which tests fail intermittently, then fix the root cause.

SSH Debugging

When a build fails and you can't reproduce it locally, CircleCI allows SSH access to the build container:

  1. Click "Rerun with SSH" from a failed pipeline run
  2. SSH into the container using the provided key and address
  3. Inspect the environment, run commands manually, check file system state

This is invaluable for debugging environment-specific failures — dependency versions, file permissions, environment variables.

Testing the Deployed Application

CircleCI validates that your code builds and tests pass in isolation. For verifying that the entire deployed application works correctly — real user flows, cross-browser behavior, end-to-end scenarios — you need something different.

HelpMeTest runs functional tests against your live application. Define test scenarios in plain English, trigger runs from your CircleCI deploy job via API, and get results back without maintaining a separate Playwright or Selenium infrastructure.

Summary

A production CircleCI test configuration:

  • Caches dependencies with checksum-based keys and fallback prefixes
  • Uses multiple Docker containers for databases and services
  • Enables parallelism with circleci tests split --split-by=timings for fast E2E suites
  • Stores JUnit XML results with store_test_results for CircleCI's test summary
  • Uses Workflows to parallelize independent jobs and sequence dependent ones
  • Uses Orbs to reduce boilerplate for common tools
  • Uses Organization Contexts for shared secrets across projects

CircleCI's test splitting and timing-based parallelism is genuinely faster than naive sharding — it distributes test load evenly based on actual historical runtimes rather than file count. For large test suites, this alone justifies the platform.

Read more