SRE Capacity Planning and Load Testing: How to Test for Growth
Capacity planning is the practice of ensuring your infrastructure can handle future load before that load arrives. SRE teams use a combination of load testing, resource profiling, and traffic modeling to answer: "How much load can we handle, and when do we need to add more capacity?"
The Capacity Planning Framework
SRE capacity planning has four phases:
- Measure — what resources does each service consume at current load?
- Model — how will traffic grow over the next 6-12 months?
- Test — does the system meet SLOs at projected load?
- Plan — when do we need to add capacity, and how much?
Understanding Your Current Baseline
Before planning for growth, understand the present. For each service, collect:
Service: user-api
Current traffic: 500 RPS (peak)
Resource utilization at peak:
CPU: 45% of 4 cores = 1.8 cores
Memory: 2.1 GB of 4 GB (53%)
Network: 120 Mbps of 1 Gbps
DB connections: 45 of 100 pool
SLO at peak load:
p50 latency: 23ms
p95 latency: 87ms
p99 latency: 210ms
Error rate: 0.02%Collect this from your APM tool (Datadog, New Relic, Prometheus) for a typical week. Identify the pattern:
- Daily peak: when during the day is traffic highest?
- Weekly peak: which days see the most traffic?
- Seasonal peaks: do you have anticipated high-traffic events?
Traffic Modeling
Model future traffic based on business growth:
# traffic_model.py
def project_traffic(current_rps, growth_rate_monthly, months_ahead):
"""
Project traffic N months ahead given monthly growth rate.
growth_rate_monthly: 0.05 = 5% per month
"""
return current_rps * (1 + growth_rate_monthly) ** months_ahead
# Current: 500 RPS
# Business projects 8% month-over-month growth
# Question: what's our traffic in 6 months?
current = 500 # RPS
growth = 0.08 # 8% monthly
months = 6
projected = project_traffic(current, growth, months)
# Result: 500 * (1.08)^6 = 793 RPS
print(f"Projected 6-month peak: {projected:.0f} RPS")
print(f"Growth factor: {projected/current:.1f}x")Add event-driven peaks:
# Black Friday example
black_friday_multiplier = 8.0 # historically 8x normal
black_friday_traffic = projected * black_friday_multiplier
# 793 * 8 = 6,344 RPS peakLoad Testing to Find Capacity Limits
Saturation Testing
Find the point where SLOs break:
# artillery-saturation.yml
config:
target: "https://staging.api.example.com"
phases:
- name: "Baseline"
duration: 120
arrivalRate: 100 # current peak RPS
- name: "2x capacity"
duration: 120
arrivalRate: 100
rampTo: 200
- name: "Hold 2x"
duration: 180
arrivalRate: 200
- name: "4x capacity"
duration: 120
arrivalRate: 200
rampTo: 400
- name: "Hold 4x"
duration: 180
arrivalRate: 400
- name: "8x capacity"
duration: 120
arrivalRate: 400
rampTo: 800
- name: "Hold 8x"
duration: 180
arrivalRate: 800
- name: "Find limit"
duration: 120
arrivalRate: 800
rampTo: 2000During this test, record:
- At what RPS does p95 latency exceed SLO?
- At what RPS does error rate exceed SLO?
- What resource hits saturation first (CPU, memory, DB, network)?
Test results:
SLO (p95 < 200ms, errors < 0.1%)
100 RPS: p95=87ms, errors=0.02% ✓
200 RPS: p95=112ms, errors=0.03% ✓
400 RPS: p95=156ms, errors=0.05% ✓
600 RPS: p95=189ms, errors=0.08% ✓
800 RPS: p95=243ms, errors=0.15% ✗ ← SLO breaks here
Bottleneck: Database CPU at 100% at 700 RPS
Capacity limit: ~700 RPS (SLO boundary)
Current traffic: 500 RPS
Headroom: 40%Headroom Calculation
SRE targets running at no more than 70% capacity under normal conditions:
Current traffic: 500 RPS
Capacity limit: 700 RPS
Current utilization: 71% ← dangerously close to limit
Headroom: 700 - 500 = 200 RPS (29%)
Target headroom: 30% minimum
Action needed: scale up databaseResource Profiling
Understand what each resource costs per unit of traffic:
# resource_profiling.py
# Measured from saturation test
measurements = [
# (RPS, CPU cores, Memory GB, DB connections)
(100, 0.5, 0.8, 12),
(200, 0.9, 1.1, 22),
(400, 1.7, 1.6, 42),
(600, 2.5, 2.2, 65),
]
# Linear regression: resource per RPS
import numpy as np
rps_values = [m[0] for m in measurements]
cpu_values = [m[1] for m in measurements]
cpu_per_rps = np.polyfit(rps_values, cpu_values, 1)[0]
print(f"CPU cores per 100 RPS: {cpu_per_rps * 100:.2f}")
# Output: CPU cores per 100 RPS: 0.41
# Project resource needs at 800 RPS
target_rps = 800
required_cpu = cpu_per_rps * target_rps + 0.3 # +0.3 for overhead
print(f"Required CPU cores at {target_rps} RPS: {required_cpu:.1f}")
# Output: Required CPU cores at 800 RPS: 3.6Capacity Planning Spreadsheet Model
Service: user-api
Planning horizon: 6 months
Month | Traffic (RPS) | CPU Needed | Memory Needed | DB Connections | Action
------|---------------|------------|---------------|----------------|-------
Now | 500 | 2.1 | 2.1 GB | 45 | Monitor
+1m | 540 | 2.3 | 2.2 GB | 49 | Monitor
+2m | 583 | 2.4 | 2.4 GB | 53 | Monitor
+3m | 630 | 2.6 | 2.6 GB | 57 | Scale pods
+4m | 680 | 2.8 | 2.8 GB | 62 | Scale DB read replicas
+5m | 735 | 3.0 | 3.0 GB | 67 | Scale pods
+6m | 793 | 3.3 | 3.3 GB | 72 | ← at capacity limit
Action items:
- Month 3: Increase pod count from 4 → 6
- Month 4: Add read replica to handle DB load
- Before month 6: Complete caching layer implementationPre-Event Load Testing
For known traffic spikes (product launches, sales events):
# pre_launch_test.py
"""
Scenario: Product launch in 2 weeks.
Marketing projects 10x normal traffic for 2 hours.
Current normal: 500 RPS peak
Launch peak: 5,000 RPS
Test plan:
1. Test at 2x (1,000 RPS) - verify baseline headroom
2. Test at 5x (2,500 RPS) - verify half-capacity
3. Test at 10x (5,000 RPS) - verify full launch capacity
4. Test at 15x (7,500 RPS) - find breaking point
"""
test_phases = [
{"name": "2x", "rps": 1000, "duration": 300},
{"name": "5x", "rps": 2500, "duration": 300},
{"name": "10x", "rps": 5000, "duration": 300},
{"name": "15x", "rps": 7500, "duration": 180},
]Questions to answer before the event:
- What's the maximum RPS before SLOs break?
- Which component hits capacity first?
- Is auto-scaling configured and verified to work?
- What manual scaling can be done pre-event?
- What's the rollback plan if load is higher than expected?
Auto-Scaling Verification
Load test to verify auto-scaling works before you need it:
def test_autoscaling_triggers_and_scales_correctly():
"""
Verify: as load increases, the service auto-scales and
maintains SLO throughout the scaling event.
"""
initial_pods = count_pods('user-api')
assert initial_pods == 3 # minimum pods
# Generate load that should trigger scaling
load_test_thread = Thread(target=run_load_at, args=(800,)) # 800 RPS
load_test_thread.start()
# Wait for scale event (up to 5 minutes)
scaled = False
for _ in range(30): # check every 10 seconds
time.sleep(10)
current_pods = count_pods('user-api')
if current_pods > initial_pods:
scaled = True
print(f"Scaled from {initial_pods} to {current_pods} pods")
break
assert scaled, "Auto-scaling did not trigger within 5 minutes"
# SLO should be maintained during scale event
p95 = get_metric('user-api', 'http_request_duration_p95', last_seconds=300)
assert p95 < 200, f"SLO violated during scale event: p95={p95}ms"
# Stop load
load_test_thread.stop()
# Verify scale-down happens (eventually)
time.sleep(600) # wait 10 minutes
final_pods = count_pods('user-api')
assert final_pods <= initial_pods + 1 # should scale back downSoak Testing for Slow Degradation
Capacity issues often show up slowly: memory leaks, connection leaks, file descriptor leaks:
# soak-test.yml
config:
target: "https://api.example.com"
phases:
# Run at 80% of capacity limit for 12 hours
- name: "Soak"
duration: 43200 # 12 hours
arrivalRate: 560 # 80% of 700 RPS limit
plugins:
metrics-by-endpoint: {}
scenarios:
- name: "Mixed traffic"
flow:
- get:
url: "/api/users"
- post:
url: "/api/events"
json:
type: "page_view"Monitor during soak test:
- Memory trend (should be flat, not climbing)
- Open file descriptors
- Database connection count
- Thread count
- Garbage collection frequency (JVM services)
- Response time trend (p95 should stay flat)
If p95 climbs over 12 hours, you have a slow degradation issue that will eventually cause an incident.
Capacity Planning for Stateful Services
Databases require different capacity planning than stateless services:
Database capacity model:
Current state:
Storage: 2.4 TB used, 4 TB total
IOPS: 8,000 of 16,000 IOPS
Connections: 45 of 100 (max_connections)
Replication lag: < 100ms
Growth rate:
Data: +120 GB/month
IOPS: +400/month (following query growth)
Connections: +2/month per new service instance
Projections:
Storage exhaustion: 13 months at current rate
IOPS saturation: 20 months at current rate
Connection limit: when traffic reaches 2.2x current (at month 8)
Actions:
Month 6: Add read replica to offload read traffic
Month 10: Storage expansion (provision 8 TB)
Month 12: Connection pooling (PgBouncer)Building a Capacity Planning Culture
Capacity planning works as a quarterly exercise:
Q1 Capacity Review Agenda:
1. Current utilization vs. last quarter (30 min)
2. Traffic projections for next quarter (15 min)
3. Load test results: headroom per service (30 min)
4. Bottlenecks identified and ranked (15 min)
5. Capacity actions: what to scale, when, estimated cost (30 min)
6. Pre-event planning: upcoming launches, seasonal peaks (15 min)
Output: capacity action plan with owners and deadlinesThe review catches problems before they become incidents. A service at 85% capacity with 10% monthly growth is fine to monitor this month but needs action next month.
Summary
SRE capacity planning prevents the most avoidable class of incidents — running out of capacity:
- Baseline your current resource consumption per service at known traffic levels
- Model traffic growth using business projections and historical trends
- Saturation test to find where SLOs break and what resource hits limits first
- Calculate headroom — maintain 30%+ buffer above current peak
- Auto-scale verification — test that scaling actually works before you need it
- Soak test to catch slow degradation issues
- Quarterly reviews to keep projections current
The goal isn't perfect prediction — traffic patterns change. The goal is early warning: enough lead time that when you need more capacity, you've already ordered it.
