Artillery Distributed Load Testing: Worker Fleet and AWS Lambda

The Single-Machine Ceiling

Running Artillery on a laptop or a CI runner is enough for low-to-medium load tests. A single Node.js process on modern hardware can generate 500–1000 virtual users comfortably. Beyond that, the machine itself becomes the bottleneck — CPU-bound scripting logic, TCP port exhaustion, and memory pressure start skewing results before your server does.

Distributed load testing solves this by splitting the load generation across multiple machines (workers) and aggregating their results into one report. Artillery supports two distribution models: a self-managed worker fleet and a serverless model running on AWS Lambda. This post covers both, along with YAML configuration, result aggregation, and spotting regressions in the output.

Artillery Overview and YAML Config

Artillery scenarios live in YAML files. Here is a minimal configuration:

config:
  target: "https://api.example.com"
  phases:
    - duration: 60       # seconds
      arrivalRate: 10    # new virtual users per second
    - duration: 120
      arrivalRate: 50
    - duration: 60
      arrivalRate: 10    # cool-down

scenarios:
  - name: "Browse and add to cart"
    flow:
      - get:
          url: "/products"
          expect:
            - statusCode: 200
      - post:
          url: "/cart"
          json:
            productId: "prod_001"
            quantity: 1
          expect:
            - statusCode: 201
      - get:
          url: "/cart"
          expect:
            - statusCode: 200

Run it locally:

artillery run load-test.yaml

Artillery prints a summary after the run:

Summary report @ 14:32:01(+0000)
  Scenarios launched:  3600
  Scenarios completed: 3598
  Requests completed:  10794
  Mean response/sec:   59.9
  Response time (msec):
    min:   18
    max:   1203
    median: 87
    95th:  312
    99th:  589
  Scenario counts:
    Browse and add to cart: 3600 (100%)
  Codes:
    200: 7196
    201: 3598

Variables and Payloads

Use CSV payloads to vary request data:

config:
  target: "https://api.example.com"
  payload:
    path: "./users.csv"
    fields:
      - "email"
      - "password"

scenarios:
  - name: "Login flow"
    flow:
      - post:
          url: "/auth/login"
          json:
            email: "{{ email }}"
            password: "{{ password }}"
          capture:
            - json: "$.token"
              as: "authToken"
      - get:
          url: "/account"
          headers:
            Authorization: "Bearer {{ authToken }}"

The capture directive extracts values from responses and stores them as variables for subsequent requests in the same scenario — the load-testing equivalent of pm.collectionVariables.set.

Distributed Testing with a Worker Fleet

How the Worker Model Works

In distributed mode, Artillery designates one machine as the coordinator and one or more as workers. The coordinator:

1. Splits the load scenario into equal fractions
2. Sends each fraction to a worker
3. Each worker runs its fraction independently and streams result chunks back
4. The coordinator aggregates the chunks into a single report

This model scales linearly: 10 workers generating 500 VU each gives you 5,000 concurrent virtual users with no single-machine bottleneck.

Self-Managed Worker Fleet

Artillery Pro (commercial) and the open-source artillery-engine-playwright can run on a self-managed fleet. For a DIY approach using the open-source CLI, you can coordinate workers manually using environment variables and a shared results file, but the recommended path for serious distributed testing is Artillery Cloud or the Lambda backend.

For a self-managed setup with two EC2 workers:

# On coordinator — split the scenario
artillery run --count 1 --output worker1-results.json load-test.yaml &
artillery run --count 1 --output worker2-results.json load-test.yaml &
wait

# Merge results
artillery report worker1-results.json worker2-results.json \
  --output combined-report.html

This is rough but works for small fleets. For automated coordination, use Artillery Cloud.

Artillery Cloud (Managed Workers)

Artillery Cloud provides managed workers with a single command:

# Authenticate
artillery cloud-auth --key $ARTILLERY_CLOUD_KEY

# Run distributed test on 10 workers
artillery run --cloud --count 10 load-test.yaml

The --count flag controls the number of workers. The tool handles provisioning, coordination, and result aggregation automatically. You get a report URL at the end of the run.

Running on AWS Lambda (Serverless Load Testing)

The AWS Lambda backend removes the need to manage any machines at all. Artillery invokes Lambda functions as load workers — each Lambda invocation handles a slice of the test, and results are collected via CloudWatch or Artillery Cloud.

Why Lambda?

• No servers to provision or terminate — Lambda scales to thousands of concurrent invocations
• Geographically distributed — run workers in multiple AWS regions for realistic global load
• Cost-efficient for burst tests — pay per invocation, not per idle hour
• Integrates with existing AWS infrastructure — VPC access, IAM roles, environment variables

Setup

Install the Lambda plugin:

npm install -g @artilleryio/artillery-engine-lambda

Configure AWS credentials:

export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_DEFAULT_REGION=us-east-1

Create the Artillery Lambda function (one-time setup):

artillery lambda setup

This deploys a Lambda function to your AWS account that Artillery uses as the worker runtime.

Lambda Test Config

config:
  target: "https://api.example.com"
  plugins:
    lambda:
      region: us-east-1
      function: artillery-worker    # function name from setup step
  phases:
    - duration: 300
      arrivalRate: 100
  payload:
    path: "./products.csv"
    fields:
      - "productId"

scenarios:
  - name: "Product page load"
    flow:
      - get:
          url: "/products/{{ productId }}"
          expect:
            - statusCode: 200
            - contentType: json

Run on Lambda:

artillery run --lambda --count 20 load-test.yaml

This invokes 20 Lambda functions simultaneously, each running a slice of the 100 arrival rate (5 virtual users/second per worker). The coordinator aggregates results as Lambda invocations complete.

Multi-Region Lambda

To generate load from multiple AWS regions simultaneously:

config:
  plugins:
    lambda:
      regions:
        - us-east-1
        - eu-west-1
        - ap-southeast-1

Each region runs its proportional share of workers. This simulates genuinely global load and can expose latency differences between regions before real users do.

Aggregating Results Across Workers

When a distributed run completes, Artillery merges worker results into a unified statistical summary. The aggregation is not a simple average — it combines the raw data points from all workers and recomputes percentiles on the full dataset.

This matters because averaging percentiles is statistically wrong. If worker 1 has p99 = 500ms and worker 2 has p99 = 800ms, the true p99 of the combined dataset is not 650ms — it depends on the full latency distribution of both workers combined.

Saving Results for Later Analysis

artillery run --output results.json load-test.yaml

The results.json file contains the full raw metrics. Generate an HTML report from it:

artillery report results.json
# Opens results.json.html in browser

For distributed runs via Artillery Cloud, the report is hosted and linked automatically after the run completes.

Analyzing Reports and Spotting Regressions

What to Look For

Artillery's report surfaces several signals:

p95 and p99 latency — these percentiles tell you what your slowest users experience. A passing median with a bad p99 means some portion of requests are unacceptably slow.

Error rate — any non-2xx responses during a load test. Even a 0.1% error rate at 10,000 req/s is 10 errors per second.

Throughput over time — does actual req/s match the configured arrival rate? If actual throughput is lower than configured, your server is dropping connections or timing out.

VU count vs completed — if scenarios launched ≠ scenarios completed, something is failing mid-flow.

Regression Detection with Baselines

Save a results file from a known-good run (e.g., post-last-release):

artillery run --output baseline.json load-test.yaml

After the next release, run again:

artillery run --output current.json load-test.yaml

Compare with a simple script:

const baseline = require('./baseline.json');
const current = require('./current.json');

const baseP95 = baseline.aggregate.latency.p95;
const currentP95 = current.aggregate.latency.p95;
const regression = ((currentP95 - baseP95) / baseP95) * 100;

if (regression > 20) {
  console.error(`REGRESSION: p95 latency increased by ${regression.toFixed(1)}%`);
  console.error(`Baseline: ${baseP95}ms → Current: ${currentP95}ms`);
  process.exit(1);
}

console.log(`p95 latency delta: ${regression.toFixed(1)}% (within threshold)`);

Run this script as a step in CI after the load test. If latency regressed by more than 20%, the build fails.

CI Integration

# GitHub Actions example
- name: Run load test
  run: |
    artillery run \
      --output /tmp/load-results.json \
      load-test.yaml

- name: Check for regressions
  run: node scripts/check-regression.js
  env:
    BASELINE_FILE: baselines/last-release.json
    CURRENT_FILE: /tmp/load-results.json

Store your baseline in the repo (baselines/) and update it as part of the release process. This creates a concrete, versioned performance contract that CI enforces automatically.

Practical Tips

Start small, scale up. Run with arrivalRate: 5 first to confirm the scenario works correctly before adding workers. A flawed scenario at 5 VU is easy to debug; at 500 VU it floods your logs.

Warm up first. Add a short ramp phase before the sustained load phase. Cold-start effects (JVM warm-up, connection pool initialization, CDN edge node caching) will skew your first-minute numbers.

Isolate the environment. Run load tests against a dedicated staging environment, not production. Load tests should not affect real users.

Watch your load generator too. Monitor CPU, memory, and open file descriptors on the Artillery coordinator or Lambda invocations. If the load generator is saturated, your results undercount true load.

Pin Artillery versions. Load test results are only comparable if the tool version is the same. Pin with npm install artillery@2.x.x and commit the lockfile.

Wrapping Up

Artillery's distributed mode — whether through a self-managed fleet, Artillery Cloud, or AWS Lambda — removes the ceiling imposed by single-machine load generation. The YAML configuration model keeps tests readable and version-controlled, result aggregation handles the statistics correctly, and regression detection in CI creates a durable performance contract.

The natural next step is making pass/fail decisions more precise: not just "did the test complete" but "did latency stay within SLO bounds at every phase of the load curve". That is where k6's threshold system shines.