Testing

Semaphore CI Testing: Fast Parallel Test Pipelines

HelpMeTest

16 May 2026 — 5 min read

Semaphore CI is a hosted continuous integration and delivery platform focused on speed. Its main selling points are fast build machines and first-class support for test parallelism — both within a single job and across multiple jobs.

If your test suite takes too long and you want parallel execution without complex self-hosted infrastructure, Semaphore is worth evaluating.

How Semaphore Works

Semaphore pipelines are defined in .semaphore/semaphore.yml. Unlike some CI tools where parallelism is an afterthought, Semaphore's pipeline model has parallelism built into the structure:

Blocks contain groups of related jobs
Jobs within a block run in parallel
Blocks run sequentially (each block waits for the previous to complete)

This maps naturally to test workflows: run unit tests and lint in parallel, then run integration tests only after both pass.

Basic Pipeline Structure

# .semaphore/semaphore.yml
version: v1.0
name: Test Pipeline

agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu2204

blocks:
  - name: Unit Tests
    task:
      jobs:
        - name: Run Tests
          commands:
            - checkout
            - npm ci
            - npm test

The checkout command clones your repository. It's a built-in Semaphore command.

Environment Setup

Configure the environment at the task level (shared across jobs) or per-job:

blocks:
  - name: Tests
    task:
      env_vars:
        - name: NODE_ENV
          value: test
        - name: LOG_LEVEL
          value: error
      
      prologue:
        commands:
          - checkout
          - nvm use 20
          - npm ci
      
      jobs:
        - name: Unit Tests
          commands:
            - npm run test:unit

        - name: Integration Tests
          commands:
            - npm run test:integration

The prologue runs before every job in the task. Since unit tests and integration tests both need npm ci, putting it in the prologue avoids repeating it.

Parallel Jobs in a Block

Jobs within a block run in parallel automatically:

blocks:
  - name: Test Suite
    task:
      prologue:
        commands:
          - checkout
          - npm ci
      
      jobs:
        - name: Unit Tests
          commands:
            - npm run test:unit

        - name: Linting
          commands:
            - npm run lint

        - name: Type Check
          commands:
            - npx tsc --noEmit

All three jobs run simultaneously. The block completes when all jobs finish.

Test Parallelism with Test Splitting

Semaphore has built-in test splitting for distributing tests across parallel jobs.

Using Semaphore's test boosting

blocks:
  - name: Jest Tests
    task:
      env_vars:
        - name: RAILS_ENV
          value: test
      
      prologue:
        commands:
          - checkout
          - npm ci
      
      jobs:
        - name: "Jest Tests"
          parallelism: 4
          commands:
            - npx jest --shard=$((SEMAPHORE_JOB_INDEX + 1))/$SEMAPHORE_JOB_COUNT --ci

parallelism: 4 creates 4 job instances. Semaphore provides:

$SEMAPHORE_JOB_INDEX — 0-based index (0, 1, 2, 3)
$SEMAPHORE_JOB_COUNT — total (4)

Playwright parallel sharding

blocks:
  - name: E2E Tests
    task:
      prologue:
        commands:
          - checkout
          - npm ci
          - npx playwright install --with-deps chromium
      
      jobs:
        - name: "Playwright Shards"
          parallelism: 4
          commands:
            - npx playwright test --shard=$((SEMAPHORE_JOB_INDEX + 1))/$SEMAPHORE_JOB_COUNT
      
      epilogue:
        always:
          commands:
            - '[[ -d playwright-report ]] && artifact push workflow playwright-report'

The epilogue runs after every job regardless of pass/fail. artifact push workflow stores the Playwright report as a Semaphore artifact.

pytest parallelism

blocks:
  - name: Python Tests
    task:
      prologue:
        commands:
          - checkout
          - pip install -r requirements.txt
      
      jobs:
        - name: "pytest Shards"
          parallelism: 3
          commands:
            - |
              # Split test files into shards
              TEST_FILES=$(python -c "
              import glob
              files = sorted(glob.glob('tests/**/*.py', recursive=True))
              shard = int('$SEMAPHORE_JOB_INDEX')
              total = int('$SEMAPHORE_JOB_COUNT')
              chunk = [files[i] for i in range(shard, len(files), total)]
              print(' '.join(chunk))
              ")
              pytest $TEST_FILES -v --junitxml=test-results/junit-$SEMAPHORE_JOB_INDEX.xml
      
      epilogue:
        always:
          commands:
            - test-results/publish test-results/*.xml

Caching

Semaphore has a built-in cache for fast dependency restoration:

blocks:
  - name: Tests
    task:
      prologue:
        commands:
          - checkout
          - cache restore node-modules-$(checksum package-lock.json)
          - npm ci
          - cache store node-modules-$(checksum package-lock.json) node_modules
      
      jobs:
        - name: Tests
          commands:
            - npm test

The cache restore and cache store commands are Semaphore built-ins. The key uses checksum of package-lock.json to invalidate when dependencies change.

Python caching

prologue:
  commands:
    - checkout
    - cache restore pip-packages-$(checksum requirements.txt)
    - pip install -r requirements.txt
    - cache store pip-packages-$(checksum requirements.txt) ~/.cache/pip

Go module caching

prologue:
  commands:
    - checkout
    - cache restore go-modules-$(checksum go.sum)
    - go mod download
    - cache store go-modules-$(checksum go.sum) ~/go/pkg/mod

Database Services

Run databases as services alongside your tests:

blocks:
  - name: Integration Tests
    task:
      services:
        - postgres

      env_vars:
        - name: DATABASE_URL
          value: postgresql://semaphore:semaphore@localhost/semaphore_test
      
      prologue:
        commands:
          - checkout
          - npm ci
      
      jobs:
        - name: Integration Tests
          commands:
            - npm run db:migrate
            - npm run test:integration

Semaphore provides several pre-configured services:

postgres — PostgreSQL on port 5432
mysql — MySQL on port 3306
redis — Redis on port 6379
mongodb — MongoDB on port 27017
rabbitmq — RabbitMQ on port 5672
elasticsearch — Elasticsearch on port 9200
memcached — Memcached on port 11211
cassandra — Cassandra on port 9042

Service credentials use standard defaults (documented in Semaphore docs).

Browser Testing

For browser tests, Semaphore provides machines with browsers pre-installed:

blocks:
  - name: Browser Tests
    task:
      agent:
        machine:
          type: e1-standard-4  # larger machine for browser tests
          os_image: ubuntu2204
      
      prologue:
        commands:
          - checkout
          - npm ci
          - npx playwright install chromium firefox webkit
      
      jobs:
        - name: "Browser Tests"
          parallelism: 3
          commands:
            - npx playwright test --shard=$((SEMAPHORE_JOB_INDEX + 1))/$SEMAPHORE_JOB_COUNT
      
      epilogue:
        always:
          commands:
            - '[[ -d playwright-report ]] && artifact push workflow playwright-report'

Multi-Stage Pipelines

Define sequential stages with the after_pipeline and pipeline promotion:

# .semaphore/semaphore.yml
version: v1.0
name: CI Pipeline

agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu2204

blocks:
  - name: Unit Tests
    task:
      jobs:
        - name: Tests
          commands:
            - checkout
            - npm ci
            - npm run test:unit

  - name: Integration Tests
    task:
      jobs:
        - name: Integration
          commands:
            - checkout
            - npm ci
            - npm run test:integration

promotions:
  - name: Deploy to Staging
    pipeline_file: deploy-staging.yml
    auto_promote:
      when: "branch = 'main' AND result = 'passed'"

The deploy-staging.yml pipeline runs automatically when tests pass on main.

Secrets and Environment Variables

Store secrets in Semaphore's secret management:

# Add a secret via CLI
sem create secret production-env \
  -e DATABASE_URL=postgresql://prod-host/app \
  -e API_KEY=your-api-key

Use in pipelines:

blocks:
  - name: Integration Tests
    task:
      secrets:
        - name: production-env
      
      jobs:
        - name: Tests
          commands:
            - checkout
            - npm ci
            - npm run test:integration  # DATABASE_URL and API_KEY are available

Secrets are decrypted into environment variables for each job that requests them.

Test Reports

Semaphore supports JUnit XML test reports with the test-results command:

jobs:
  - name: Tests
    commands:
      - checkout
      - npm ci
      - npx jest --ci --reporters=default --reporters=jest-junit
      - '[[ $? -ne 0 ]] && true'  # continue even if tests fail

epilogue:
  always:
    commands:
      - '[[ -f junit.xml ]] && test-results publish junit.xml'

Published test results appear in the Semaphore UI with pass/fail counts, duration, and test names.

Branch and Tag Filtering

Skip blocks for specific conditions:

blocks:
  - name: Unit Tests
    skip:
      when: "branch =~ 'dependabot/.*'"
    task:
      jobs:
        - name: Test
          commands:
            - npm test

Run blocks only on specific conditions:

blocks:
  - name: Deploy
    run:
      when: "branch = 'main'"
    task:
      jobs:
        - name: Deploy
          commands:
            - npm run deploy

Choosing Machine Type

Select machine types based on your workload:

agent:
  machine:
    type: e1-standard-4  # 4 vCPU, 8GB RAM
    os_image: ubuntu2204

Available types (sizes vary):

e1-standard-2 — 2 vCPU, 4GB RAM (default)
e1-standard-4 — 4 vCPU, 8GB RAM
e1-standard-8 — 8 vCPU, 16GB RAM
Mac machines available for iOS/macOS testing

For parallel jobs, a larger machine lets multiple jobs run without resource contention.

Workflow Artifacts

Store and share files between pipeline stages:

# Store artifact in one block
epilogue:
  always:
    commands:
      - artifact push workflow build/

# Access in a later pipeline
prologue:
  commands:
    - artifact pull workflow build/

Artifacts persist across the pipeline run and are downloadable from the Semaphore UI.

Debugging Failing Tests

When tests fail, use Semaphore's SSH session feature to debug:

# Open SSH session to a failed job
sem debug job <job-id>

This gives you an interactive shell in the same environment as the failed job — useful for reproducing failures that only occur in CI.

Comparing Semaphore to Alternatives

Feature	Semaphore	GitHub Actions	CircleCI
Native test parallelism	✓ Built-in	Via matrix	✓ Built-in
Hosted machines	✓ Fast SSD-backed	Standard	Standard
Self-hosted agents	✓	✓	✓
Test analytics	Basic via XML	Via 3rd party	✓
Pipeline complexity	Moderate	Low	Moderate
macOS support	✓	✓	✓

Semaphore's speed advantage comes from SSD-backed machines and efficient job startup. Teams that have outgrown slow builds on other platforms often see 30-50% improvement by switching.

Conclusion

Semaphore's pipeline model — parallel jobs within sequential blocks — maps well to how test suites are actually structured. Unit tests and linting run in parallel. Integration tests run after. E2E tests run after that. Expressing this in Semaphore's YAML is natural.

The built-in parallelism variables (SEMAPHORE_JOB_INDEX, SEMAPHORE_JOB_COUNT) make test sharding straightforward. Combined with the cache system and pre-configured database services, you can set up a fast, parallel test pipeline without much configuration overhead.

Start with a single block for your primary test suite, add parallelism once the suite gets slow, and configure caching to keep install times from dominating build duration.