Tsung Distributed Load Testing: Multi-Node Setup Guide

Tsung Distributed Load Testing: Multi-Node Setup Guide

A single Tsung node can generate substantial load — typically 50,000–100,000 concurrent users on modern hardware. But if you need to simulate millions of users, or if you want to generate load from multiple geographic locations, you need Tsung's distributed mode.

Tsung's distributed architecture is built on Erlang clustering. The controller node coordinates scenario execution; injector nodes generate the actual load. The architecture scales horizontally: add more injector nodes to increase the maximum concurrent user count.

Architecture Overview

                    ┌─────────────┐
                    │  Controller │  (your laptop or CI server)
                    │  tsung.xml  │
                    └──────┬──────┘
                           │ SSH + Erlang RPC
              ┌────────────┼────────────┐
              ▼            ▼            ▼
       ┌──────────┐ ┌──────────┐ ┌──────────┐
       │Injector 1│ │Injector 2│ │Injector 3│
       │ 50k users│ │ 50k users│ │ 50k users│
       └──────────┘ └──────────┘ └──────────┘
                           │
                           ▼
                    ┌─────────────┐
                    │   Target    │
                    │   Server    │
                    └─────────────┘

The controller distributes the load phase across all configured injector nodes proportionally.

Prerequisites

On all nodes (controller and injectors):

  • Same version of Tsung installed
  • Same version of Erlang installed
  • Erlang cookie must match across all nodes

Network requirements:

  • Controller must be able to SSH to all injector nodes without password
  • Erlang distributed protocol uses TCP (port 4369 for epmd, plus dynamic ports)
  • All nodes must be able to reach the target server

Step 1: Set Up SSH Key Authentication

The controller uses SSH to start Tsung processes on injector nodes.

On the controller:

# Generate SSH key if you don't have one
ssh-keygen -t ed25519 -f ~/.ssh/tsung_key -N ""

# Copy to each injector
ssh-copy-id -i ~/.ssh/tsung_key.pub user@injector1.example.com
ssh-copy-id -i ~/.ssh/tsung_key.pub user@injector2.example.com
ssh-copy-id -i ~/.ssh/tsung_key.pub user@injector3.example.com

# Test passwordless SSH
ssh -i ~/.ssh/tsung_key user@injector1.example.com hostname

All Erlang nodes in the cluster must share the same cookie (a shared secret for node authentication).

Set the same cookie on all nodes:

# On controller and all injectors:
echo "my_secret_tsung_cookie" > ~/.erlang.cookie
chmod 400 ~/.erlang.cookie

Or use the Tsung config:

<tsung loglevel="notice" version="1.0" 
       erlang_cookie="my_secret_tsung_cookie">

Step 3: Configure Distributed Clients

In tsung.xml, list all injector nodes:

<clients>
  <!-- Controller node also acts as injector -->
  <client host="controller.example.com" 
          use_controller_vm="false"
          maxusers="50000"/>
  
  <!-- Additional injector nodes -->
  <client host="injector1.example.com" 
          cpu="4"
          maxusers="50000"/>
  
  <client host="injector2.example.com" 
          cpu="4"
          maxusers="50000"/>
  
  <client host="injector3.example.com" 
          cpu="4"
          maxusers="50000"/>
</clients>

cpu: Number of Erlang beam processes to start on this node. Typically matches the number of CPU cores.

maxusers: Maximum concurrent users this node will handle. Total capacity = sum across all nodes.

use_controller_vm: If true, reuses the controller's Erlang VM instead of starting a separate process. Only for single-machine setups.

Step 4: Distribute Load Proportionally

By default, Tsung distributes load evenly across all client nodes. You can weight nodes differently:

<clients>
  <!-- High-capacity node: 4 CPU cores, takes more load -->
  <client host="injector1.example.com" 
          cpu="8"
          maxusers="100000"/>
  
  <!-- Lower-capacity node -->
  <client host="injector2.example.com" 
          cpu="2"
          maxusers="25000"/>
</clients>

Tsung distributes user arrivals proportionally to cpu counts.

Step 5: Configure OS Limits on Injectors

Each injector node needs OS-level tuning for high concurrency.

On each injector node:

# Increase open file descriptors
ulimit -n 200000

# Add to /etc/security/limits.conf:
echo "* soft nofile 200000" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 200000" | sudo tee -a /etc/security/limits.conf

# Tune TCP settings
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
sudo sysctl -w net.ipv4.tcp_fin_timeout=15
sudo sysctl -w net.core.somaxconn=65536
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

# Make sysctl changes persistent
sudo tee -a /etc/sysctl.conf <<EOF
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=15
net.core.somaxconn=65536
net.ipv4.ip_local_port_range=1024 65535
EOF

Complete Distributed Configuration Example

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd">

<tsung loglevel="notice" version="1.0"
       erlang_cookie="cluster_secret_cookie">

  <!-- Four injector nodes, 50k users each = 200k total -->
  <clients>
    <client host="injector1.us-east.example.com" cpu="8" maxusers="50000"/>
    <client host="injector2.us-east.example.com" cpu="8" maxusers="50000"/>
    <client host="injector3.us-west.example.com" cpu="8" maxusers="50000"/>
    <client host="injector4.eu-west.example.com" cpu="8" maxusers="50000"/>
  </clients>

  <servers>
    <server host="api.example.com" port="443" type="ssl"/>
  </servers>

  <!-- 200k users over 10 minutes -->
  <load duration="10" unit="minute">
    <!-- Ramp up: 0 → 50k users over 3 minutes -->
    <arrivalphase phase="1" duration="3" unit="minute">
      <users interarrival="0.0036" unit="second"/>
    </arrivalphase>
    
    <!-- Sustained peak: 50k/s for 5 minutes -->
    <arrivalphase phase="2" duration="5" unit="minute">
      <users interarrival="0.00002" unit="second"/>
    </arrivalphase>
    
    <!-- Cool-down: 2 minutes -->
    <arrivalphase phase="3" duration="2" unit="minute">
      <users interarrival="0.01" unit="second"/>
    </arrivalphase>
  </load>

  <sessions>
    <session name="api_load_test" probability="100" type="ts_http">
      
      <setdynvars sourcetype="file" fileid="users" delimiter=",">
        <var name="user_id"/>
        <var name="auth_token"/>
      </setdynvars>

      <!-- Authenticated API request sequence -->
      <request>
        <http url="/api/feed" method="GET" version="1.1">
          <http_header name="Authorization" value="Bearer %%_auth_token%%"/>
        </http>
      </request>

      <thinktime value="2" random="true"/>

      <request>
        <http url="/api/items/%%_user_id%%" method="GET" version="1.1">
          <http_header name="Authorization" value="Bearer %%_auth_token%%"/>
        </http>
      </request>

    </session>
  </sessions>

  <!-- Data files referenced above -->
  <options>
    <option name="file_server" id="users" value="/etc/tsung/users.csv"/>
  </options>

</tsung>

Running a Distributed Test

# Start the test from the controller
tsung -f tsung-distributed.xml start

# Tsung will:
# 1. SSH to each injector node
# 2. Start Erlang beam processes
# 3. Form an Erlang cluster
# 4. Distribute user arrivals across nodes
# 5. Collect stats centrally on the controller

# Monitor in real-time via the web dashboard (default: http://localhost:8091)

Monitoring During the Test

Tsung includes a built-in web dashboard:

# Access while test is running
open http://localhost:8091

The dashboard shows live charts of:

  • Current concurrent users (per node and total)
  • Requests per second
  • Response time mean and 95th percentile
  • Error rate

For richer monitoring, configure Tsung to send metrics to a monitoring system:

<monitoring>
  <monitor host="injector1.example.com" type="erlang">
    <cpu interval="10"/>
    <memory interval="10"/>
  </monitor>
</monitoring>

Collecting Distributed Results

All injectors stream stats back to the controller in real time. After the test:

cd ~/.tsung/log/TIMESTAMP/
tsung_stats.pl --stats tsung.log
# Opens tsung.html with aggregate results across all nodes

The HTML report aggregates results from all injectors — you see one unified view of latency, throughput, and errors.

Troubleshooting

"Cannot connect to node" error:

Error: Cannot connect to tsung@injector1.example.com

Check: SSH passwordless access works, Erlang cookies match, firewalls allow Erlang ports.

Uneven load distribution: Verify cpu values are set correctly. Tsung distributes proportionally to CPU count. High-memory, low-CPU nodes may be underutilized.

Erlang port conflicts: Tsung uses epmd (port 4369) and dynamic Erlang distribution ports. Ensure these aren't blocked between controller and injectors.

Injector falls behind: The injector's OS isn't keeping up. Increase the file descriptor limit, tune TCP parameters, or add more injector nodes.

Cloud-Based Injectors

For ephemeral distributed load tests, provision injectors on cloud VMs:

# Example: AWS EC2 spot instances via Terraform
terraform apply -var="injector_count=10" -var="instance_type=c5.2xlarge"

# Run test
tsung -f tsung.xml start

# Destroy injectors when done
terraform destroy

This approach costs much less than maintaining dedicated injector hardware.

Summary

Tsung's distributed mode is straightforward to configure once SSH access and Erlang cookies are sorted. The horizontal scaling model — add injector nodes to increase capacity — is clean and predictable.

For teams that need to generate millions of concurrent users without paying for a cloud load testing service, self-managed Tsung clusters are a cost-effective approach. The tooling is mature, the Erlang foundation is efficient, and the distributed coordination is built-in rather than bolted on.

Read more

Start now free