Artillery in CI: GitHub Actions, GitLab, Thresholds, and Fail-Fast Strategies

HelpMeTest

26 Jul 2026 — 7 min read

Running load tests manually before releases is better than not running them. Running them automatically on every pull request is better still. Blocking a merge when performance regresses is the goal.

This post covers integrating Artillery into GitHub Actions and GitLab CI, setting thresholds that actually gate deployments, and strategies for keeping load tests fast enough to be useful in CI without burning your CI budget.

The Core Problem: Load Tests Are Slow

A proper load test runs for minutes. CI pipelines need to finish in minutes. These pull in opposite directions.

The solution is layered testing:

Smoke load test (~30 seconds) — runs on every PR. Small load, basic sanity check.
Full load test (~5-15 minutes) — runs on merge to main or on-demand. Catches real performance regressions.
Stress test (~30+ minutes) — runs on a schedule (nightly) or before major releases.

Don't try to run your full load test on every PR. You'll get slow pipelines, flaky results (because PR environments are noisy), and developers who disable the step.

GitHub Actions Setup

Basic setup for a smoke load test on every PR:

# .github/workflows/load-test.yaml
name: Load Test

on:
  pull_request:
    branches: [main]
  workflow_dispatch:
    inputs:
      test_type:
        description: "Test type (smoke/full/stress)"
        required: true
        default: "smoke"

jobs:
  load-test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install dependencies
        run: npm ci

      - name: Run smoke load test
        if: github.event_name == 'pull_request'
        env:
          TARGET_URL: ${{ secrets.STAGING_URL }}
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
        run: npx artillery run --output results.json tests/load/smoke.yaml

      - name: Run full load test
        if: github.event_name == 'workflow_dispatch' && github.event.inputs.test_type == 'full'
        env:
          TARGET_URL: ${{ secrets.STAGING_URL }}
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
        run: npx artillery run --output results.json tests/load/full.yaml

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: artillery-results
          path: results.json

      - name: Generate HTML report
        if: always()
        run: npx artillery report --output results.html results.json

      - name: Upload HTML report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: artillery-report
          path: results.html

The workflow_dispatch trigger with an input lets you manually kick off the full or stress test without a code change. Useful for pre-release verification.

Thresholds: Making CI Actually Gate on Performance

The most important part. Without thresholds, your load test runs but never blocks a bad deploy.

# tests/load/smoke.yaml
config:
  target: "{{ $processEnvironment.TARGET_URL }}"
  phases:
    - duration: 30
      arrivalRate: 10
  defaults:
    headers:
      Authorization: "Bearer {{ $processEnvironment.API_TOKEN }}"
  ensure:
    thresholds:
      - http.response_time.p95: 500      # p95 under 500ms
      - http.response_time.p99: 1000     # p99 under 1000ms
      - http.request_rate: 5             # must sustain at least 5 RPS
    conditions:
      - expression: "return stats['http.codes.500'] == 0"
        strict: true
      - expression: "return stats['vusers.failed'] / stats['vusers.created'] < 0.01"
        strict: true

scenarios:
  - name: "API health"
    flow:
      - get:
          url: "/api/health"
      - get:
          url: "/api/products"
      - get:
          url: "/api/products/1"

The ensure block defines what success means:

thresholds — metric must be below (for latency) or above (for rate) the value. Artillery exits with code 1 if any threshold is violated.

conditions — JavaScript expressions over the final stats object. strict: true means failure here is a hard failure, not just a warning.

Available metrics in conditions:

stats['http.response_time.min']
stats['http.response_time.max']
stats['http.response_time.mean']
stats['http.response_time.p95']
stats['http.response_time.p99']
stats['http.codes.200'], stats['http.codes.500'], etc.
stats['vusers.created']
stats['vusers.completed']
stats['vusers.failed']
stats['http.request_rate']

Artillery exits with a non-zero code when thresholds are violated, which causes the CI step to fail and blocks the PR merge.

Setting Meaningful Thresholds

The hard part isn't the YAML — it's picking numbers that matter.

Start from your current baseline, not from theory. Run your test against a known-good environment. Look at the p95 and p99. Set your thresholds to 1.5–2× those numbers. If your API normally returns in 80ms p95, set the threshold at 150ms. This catches real regressions while allowing for CI environment noise.

Don't use mean latency as a threshold. Mean hides the tail. A p95 of 2000ms with a mean of 100ms means 5% of your users are waiting 2 seconds. The mean looks fine; the user experience isn't.

Zero tolerance for 5xx errors. If your API returns server errors under load, that's a regression. No threshold — just assert zero 5xx responses.

Scenario completion rate. If 5% of virtual users fail to complete their flow (session expires mid-flow, upstream dependency fails), your test is catching something real. Set a max failure rate and stick to it:

conditions:
  - expression: "return (stats['vusers.failed'] / stats['vusers.created']) < 0.05"
    strict: true

GitLab CI Setup

# .gitlab-ci.yml

stages:
  - test
  - load-test
  - deploy

smoke-load-test:
  stage: load-test
  image: node:20-alpine
  only:
    - merge_requests
  script:
    - npm ci
    - npx artillery run --output results.json tests/load/smoke.yaml
  after_script:
    - npx artillery report --output results.html results.json || true
  artifacts:
    when: always
    paths:
      - results.json
      - results.html
    expire_in: 7 days
  variables:
    TARGET_URL: $STAGING_URL
    API_TOKEN: $STAGING_API_TOKEN

full-load-test:
  stage: load-test
  image: node:20-alpine
  only:
    - main
  when: manual
  allow_failure: false
  script:
    - npm ci
    - npx artillery run --output results.json tests/load/full.yaml
  artifacts:
    when: always
    paths:
      - results.json
      - results.html
    expire_in: 30 days
  variables:
    TARGET_URL: $STAGING_URL
    API_TOKEN: $STAGING_API_TOKEN

nightly-stress-test:
  stage: load-test
  image: node:20-alpine
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
  script:
    - npm ci
    - npx artillery run --output results.json tests/load/stress.yaml
  variables:
    TARGET_URL: $PRODUCTION_URL
    API_TOKEN: $PRODUCTION_API_TOKEN

The when: manual on full-load-test means it appears as an optional step in the pipeline UI. A human has to click "Run" — it doesn't trigger automatically. Use this for expensive tests that you want available but not running on every merge.

GitLab's artifact system stores the HTML report and JSON results. Team members can browse to the pipeline and download the report without needing access to the server that ran the test.

Structuring Test Files for CI

Maintain separate config files for each test tier:

tests/
  load/
    smoke.yaml        # 30s, 10 RPS — runs on every PR
    full.yaml         # 5min, 50 RPS — runs on merge to main
    stress.yaml       # 30min, ramp to 200 RPS — runs nightly
    scenarios/
      auth.yaml
      products.yaml
      checkout.yaml
    data/
      users.csv
      products.csv

The smoke, full, and stress configs share the same scenario files but differ in phases and thresholds:

# smoke.yaml
config:
  target: "{{ $processEnvironment.TARGET_URL }}"
  phases:
    - duration: 30
      arrivalRate: 10
  ensure:
    thresholds:
      - http.response_time.p95: 500

scenarios:
  - $ref: "./scenarios/auth.yaml"
  - $ref: "./scenarios/products.yaml"

# full.yaml
config:
  target: "{{ $processEnvironment.TARGET_URL }}"
  phases:
    - duration: 60
      arrivalRate: 10
      rampTo: 50
    - duration: 240
      arrivalRate: 50
    - duration: 60
      arrivalRate: 50
      rampTo: 10
  ensure:
    thresholds:
      - http.response_time.p95: 400   # tighter threshold for full test

scenarios:
  - $ref: "./scenarios/auth.yaml"
  - $ref: "./scenarios/products.yaml"
  - $ref: "./scenarios/checkout.yaml"

The full test uses tighter thresholds because it's running more load for longer — if p95 is worse under sustained load than it is in a 30-second smoke test, that's a real scalability issue.

Handling Staging Environment Noise

CI environments are shared. Your staging instance might be shared with other test runs, deploys happening mid-test, or background jobs. This causes false failures.

Strategies:

Add retry logic in the CI step, not in Artillery. Run the test up to 3 times before declaring failure:

# GitHub Actions
- name: Run smoke load test
  run: |
    for i in 1 2 3; do
      npx artillery run --output results.json tests/load/smoke.yaml && break
      echo "Attempt $i failed, retrying..."
      sleep 10
    done

This is crude but effective for transient failures. Don't use it to hide real regressions — if your test fails 3 times in a row, it's not noise.

Use a dedicated load test environment. The ideal setup: a staging environment that exists only for load testing, spun up with the PR's Docker images, load-tested, then torn down. This eliminates shared-environment noise completely.

Widen thresholds for CI vs. pre-release testing. Your nightly stress test on a dedicated environment can use tight thresholds. Your PR smoke test on shared staging should have wider thresholds to account for environment variability.

Publishing Results to PR Comments

Getting the results into the PR review flow makes them actionable:

# GitHub Actions
- name: Comment results on PR
  if: github.event_name == 'pull_request' && always()
  uses: actions/github-script@v7
  with:
    script: |
      const fs = require('fs');
      const results = JSON.parse(fs.readFileSync('results.json', 'utf8'));
      const agg = results.aggregate;
      
      const p95 = agg.latency?.p95 || 'N/A';
      const p99 = agg.latency?.p99 || 'N/A';
      const rps = Math.round(agg.rps?.mean || 0);
      const codes = JSON.stringify(agg.codes || {});
      const failed = agg.counters?.['vusers.failed'] || 0;
      const completed = agg.counters?.['vusers.completed'] || 0;
      
      const body = `## Load Test Results
      
      | Metric | Value |
      |--------|-------|
      | p95 Latency | ${p95}ms |
      | p99 Latency | ${p99}ms |
      | Requests/sec | ${rps} |
      | VUs Completed | ${completed} |
      | VUs Failed | ${failed} |
      | Status Codes | ${codes} |`;
      
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body
      });

Now every PR gets an automatic comment showing the load test results. Reviewers can see performance impact without downloading the artifact.

Secrets Management

Never put credentials in your YAML files. Use CI secrets:

GitHub Actions:

env:
  TARGET_URL: ${{ secrets.STAGING_URL }}
  API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
  DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }}

GitLab CI:

variables:
  TARGET_URL: $STAGING_URL        # from GitLab CI/CD Variables
  API_TOKEN: $STAGING_API_TOKEN

In your Artillery YAML, reference them via $processEnvironment:

config:
  target: "{{ $processEnvironment.TARGET_URL }}"
  defaults:
    headers:
      Authorization: "Bearer {{ $processEnvironment.API_TOKEN }}"

Exit Codes and CI Failure Modes

Artillery exits with:

0 — test completed successfully, all thresholds met
1 — test completed but thresholds were violated
2 — test failed to run (config error, network error, etc.)

Most CI systems treat any non-zero exit as a step failure. That's what you want — if Artillery exits 1 because p95 is too high, the CI step fails, and the PR can't merge.

If you want to collect artifacts even on failure (always() in GitHub Actions, when: always in GitLab), make sure the report generation step runs regardless:

- name: Generate HTML report
  if: always()   # Run even if the load test step failed
  run: npx artillery report --output results.html results.json

A Realistic CI Load Test Budget

For a typical API with a 5-minute CI pipeline budget:

Smoke test: 30 seconds of load + 30 seconds CI overhead = 1 minute
Full test: 5 minutes of load + 1 minute overhead = 6 minutes (run on merge to main, not on PR)
Nightly: 20-30 minutes, no time constraint

This gives you fast feedback on PRs (1 minute), comprehensive verification on merge (6 minutes), and regular deep testing nightly.

The worst outcome is a load test step that takes 15 minutes on every PR. Developers will disable it within a week. Keep PR load tests under 2 minutes and they'll stay enabled.