Load Testing

Getting Started with Artillery: Your First HTTP Load Test

HelpMeTest

HelpMeTest

— 5 min read
Getting Started with Artillery: Your First HTTP Load Test

Artillery is a load testing tool built for developers. You write YAML, you run a command, you get results. No GUI to learn, no Java heap to tune, no XML to wrestle. If you can write a curl command, you can write an Artillery test.

This post covers installation, the YAML config structure, running your first test, and reading the output. By the end you'll have a working load test against a real endpoint.

Installation

Artillery runs on Node.js. You need Node 18 or later.

npm install -g artillery@latest
artillery version

That's it. No Docker image required, no separate agent process, no license key.

If you're in a team environment, install it as a dev dependency instead:

npm install --save-dev artillery
npx artillery version

The npx approach is cleaner for CI — the version is pinned in package.json and everyone runs the same binary.

Your First Config File

Artillery tests are YAML files. Here's the minimal structure:

config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 10

scenarios:
  - name: "Homepage load"
    flow:
      - get:
          url: "/health"

Save this as load-test.yaml and run it:

artillery run load-test.yaml

This sends 10 new virtual users per second for 60 seconds — 600 total requests — to GET /health.

Understanding the Config Structure

The Artillery YAML has two top-level keys: config and scenarios.

config

config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 10
    - duration: 120
      arrivalRate: 50
    - duration: 60
      arrivalRate: 10
  defaults:
    headers:
      Authorization: "Bearer {{ $processEnvironment.API_TOKEN }}"
      Content-Type: "application/json"

target — the base URL. All scenario URLs are relative to this.

phases — the load shape. Each phase runs sequentially. In the example above: ramp to 10 RPS for 1 minute, hold at 50 RPS for 2 minutes, cool down to 10 RPS for 1 minute.

defaults — headers and other options applied to every request. Note the $processEnvironment syntax for pulling in environment variables. Never hardcode tokens in test files.

Phase options you'll use regularly:

phases:
  # Constant load
  - duration: 60
    arrivalRate: 20

  # Ramp up linearly
  - duration: 120
    arrivalRate: 5
    rampTo: 50

  # Fixed number of users (not per-second rate)
  - duration: 60
    fixedArrival: 100

arrivalRate creates N new virtual users per second. fixedArrival creates exactly N users total for the phase. Use fixedArrival when you want to model a fixed-size user pool rather than a stream of new arrivals.

scenarios

scenarios:
  - name: "API health check"
    weight: 1
    flow:
      - get:
          url: "/health"
          expect:
            - statusCode: 200

If you have multiple scenarios, weight controls the proportion of traffic each gets. A scenario with weight: 2 gets twice as much traffic as one with weight: 1.

HTTP Requests

Artillery supports GET, POST, PUT, PATCH, DELETE, and HEAD.

scenarios:
  - name: "Create and fetch user"
    flow:
      - post:
          url: "/users"
          json:
            name: "Test User"
            email: "test@example.com"
          capture:
            - json: "$.id"
              as: "userId"

      - get:
          url: "/users/{{ userId }}"
          expect:
            - statusCode: 200

Key request options:

  • json — sends a JSON body with Content-Type: application/json
  • form — sends form-encoded body
  • body — sends raw string body
  • headers — per-request headers (merged with defaults)
  • capture — extracts values from the response to use in later requests
  • expect — assertions about the response

The capture block above uses a JSONPath expression ($.id) to extract the user ID from the response body and store it in a variable called userId. That variable is then used in the next request URL.

Running Tests and Reading Output

artillery run load-test.yaml

While the test runs, Artillery prints a summary every 10 seconds:

All virtual users finished
Summary report @ 14:23:01(+0000) 2024-01-15

Scenarios launched:  600
Scenarios completed: 598
Requests completed:  1196
Mean response/sec:   19.87
Response time (msec):
  min: 12
  max: 847
  median: 45
  p95: 234
  p99: 612

Scenario counts:
  Create and fetch user: 598 (99.7%)

Codes:
  200: 1195
  201: 598
  500: 3

Read this output carefully:

  • Scenarios launched vs completed — if these differ significantly, virtual users are timing out or erroring before completing their flow
  • p95 and p99 — your slow requests. p95 of 234ms means 95% of requests finished in under 234ms
  • Codes — HTTP status code distribution. 3 500s in this run means something went wrong 3 times

To save the output as JSON for later processing:

artillery run --output results.json load-test.yaml
artillery report results.json

The report command generates an HTML file with charts. Open it in a browser.

Setting Environment Variables

Don't hardcode credentials or URLs in your test files. Use environment variables:

config:
  target: "{{ $processEnvironment.TARGET_URL }}"
  defaults:
    headers:
      Authorization: "Bearer {{ $processEnvironment.API_TOKEN }}"

Run with:

TARGET_URL=https://staging.example.com API_TOKEN=abc123 artillery run load-test.yaml

Or use a .env file with a tool like dotenv-cli:

npx dotenv -e .env -- artillery run load-test.yaml

Targeting Different Environments

A common pattern is a base config with environment-specific overrides:

# base.yaml
config:
  phases:
    - duration: 60
      arrivalRate: 10
  defaults:
    headers:
      Content-Type: "application/json"

scenarios:
  - name: "API test"
    flow:
      - get:
          url: "/api/status"
# staging.yaml
config:
  target: "https://staging.example.com"
artillery run --config staging.yaml base.yaml

The --config flag merges the two files. This keeps environment-specific values out of your main test file.

Timeouts and Retries

By default Artillery waits 10 seconds for a response before marking a request as failed. Adjust per-request:

scenarios:
  - name: "Slow endpoint"
    flow:
      - get:
          url: "/slow-report"
          timeout: 30

Artillery doesn't retry failed requests by default — failed is failed. This is intentional. A retry during a load test masks the failure and inflates your success numbers. If your API is failing under load, you want to see that, not hide it.

Quick Smoke Test Before Load

Before running a load test, run a single virtual user to verify your scenario actually works:

artillery run --count 1 --overrides '{"config": {"phases": [{"duration": 1, "arrivalRate": 1}]}}' load-test.yaml

Or use artillery quick for a throwaway test:

artillery quick --count 10 --num 5 https://api.example.com/health

This sends 10 requests from 5 virtual users. Useful for a quick sanity check without writing a config file.

Common Mistakes

Testing production directly. Start with staging. A misconfigured test with arrivalRate: 1000 will knock over a production API.

Not checking scenario completion rate. If your scenario has 5 steps and the API fails on step 3, Artillery records the scenario as incomplete. A high completion rate with bad status codes tells you the flow works but the API is erroring. A low completion rate means your flow itself is breaking.

Ignoring the p99. The median looks great but your slowest 1% of users have a bad experience. For user-facing APIs, p99 matters.

Running tests from your laptop. Network latency from your local machine to the server skews results. Run from the same cloud region as your users, or use Artillery Cloud.

Not warming up the API. Many APIs are slower on cold start (cold database connections, JIT compilation, cache misses). Start with a small ramp phase before your main load phase to get the system into steady state.

What's Next

This gets you running. The real power comes from multi-step scenarios — logging in, creating resources, querying them, cleaning up — which reflect how real users actually interact with your API. That's covered in the next post on Artillery scenarios.

For now: write a config, point it at staging, run it. The first time you see your p99 spike to 3 seconds at 50 RPS, you'll know why load testing matters.

Read more

Start now free