Testing Distributed Traces with Jaeger and Grafana Tempo
In-memory exporter tests verify that your instrumentation emits the right spans in isolation. But in a distributed system, spans across multiple services need to connect into coherent traces — same trace ID, correct parent/child relationships, no orphaned spans. That's what integration tests against a real tracing backend validate.
This post covers how to test distributed traces programmatically against Jaeger and Grafana Tempo, using Docker Compose for local and CI environments.
Why Test Trace Structure
Instrumentation correctness tests (in-memory exporters) verify that individual spans have the right attributes. Trace structure tests verify something different:
- Trace completeness: all expected services appear in the trace
- Parent/child relationships: the checkout span is the parent of the payment span, which is the parent of the DB query span
- Span ordering: operations appear in the correct temporal sequence
- Missing spans: a refund flow that should have 6 spans is only producing 4
- Unexpected spans: a span from a service that should be bypassed is appearing
These problems are invisible in unit tests. They only appear when you exercise the full request flow across services and inspect the resulting trace.
Docker Compose Setup
# docker-compose.test.yml
services:
# Jaeger all-in-one (for simpler setup)
jaeger:
image: jaegertracing/all-in-one:1.52
environment:
COLLECTOR_OTLP_ENABLED: "true"
ports:
- "16686:16686" # UI
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:16686/"]
interval: 5s
timeout: 5s
retries: 10
# Grafana Tempo (for Prometheus/Loki integration)
tempo:
image: grafana/tempo:2.3.1
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- ./tempo-test.yaml:/etc/tempo.yaml
ports:
- "3200:3200" # HTTP API
- "4319:4317" # OTLP gRPC (different port to avoid conflict with Jaeger)
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3200/ready"]
interval: 5s
retries: 10
# Your service under test
payments-api:
build: .
environment:
OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
OTEL_SERVICE_NAME: payments-api
OTEL_TRACES_EXPORTER: otlp
depends_on:
jaeger:
condition: service_healthy
ports:
- "8080:8080"# tempo-test.yaml
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
ingester:
trace_idle_period: 10s
storage:
trace:
backend: local
wal:
path: /tmp/tempo/wal
local:
path: /tmp/tempo/blocks
compactor:
compaction:
compaction_window: 1hQuerying Jaeger Programmatically
Jaeger exposes a REST API for querying traces. Use it from your integration tests:
# tests/trace_helpers.py
import time
import requests
from typing import Optional, List, Dict, Any
from dataclasses import dataclass
JAEGER_URL = "http://localhost:16686"
@dataclass
class Span:
span_id: str
operation_name: str
service_name: str
parent_span_id: Optional[str]
tags: Dict[str, Any]
duration_us: int
logs: List[Dict]
def get_traces_for_service(
service: str,
operation: Optional[str] = None,
limit: int = 20,
lookback: str = "1h"
) -> List[Dict]:
"""Fetch recent traces from Jaeger for a given service"""
params = {
"service": service,
"limit": limit,
"lookback": lookback,
}
if operation:
params["operation"] = operation
response = requests.get(f"{JAEGER_URL}/api/traces", params=params)
response.raise_for_status()
return response.json()["data"]
def get_trace_by_id(trace_id: str) -> Optional[Dict]:
"""Fetch a specific trace by ID"""
response = requests.get(f"{JAEGER_URL}/api/traces/{trace_id}")
if response.status_code == 404:
return None
response.raise_for_status()
return response.json()["data"][0]
def flatten_trace_spans(trace: Dict) -> List[Span]:
"""Convert Jaeger trace format to a list of Span objects"""
processes = trace["processes"]
spans = []
for raw_span in trace["spans"]:
process_id = raw_span["processID"]
service_name = processes[process_id]["serviceName"]
parent_span_id = None
for ref in raw_span.get("references", []):
if ref["refType"] == "CHILD_OF":
parent_span_id = ref["spanID"]
break
tags = {tag["key"]: tag["value"] for tag in raw_span.get("tags", [])}
spans.append(Span(
span_id=raw_span["spanID"],
operation_name=raw_span["operationName"],
service_name=service_name,
parent_span_id=parent_span_id,
tags=tags,
duration_us=raw_span["duration"],
logs=raw_span.get("logs", [])
))
return spans
def wait_for_trace(service: str, operation: str,
timeout_seconds: int = 30) -> Optional[Dict]:
"""Poll Jaeger until a trace appears, or timeout"""
start = time.time()
while time.time() - start < timeout_seconds:
traces = get_traces_for_service(service, operation, lookback="5m")
if traces:
return traces[0]
time.sleep(2)
return NoneAsserting Trace Structure
# tests/integration/test_trace_structure.py
import pytest
import requests
from tests.trace_helpers import (
wait_for_trace, flatten_trace_spans, get_trace_by_id
)
BASE_URL = "http://localhost:8080"
class TestCheckoutTraceStructure:
def test_checkout_produces_complete_trace(self):
"""A checkout request should produce spans from 3 services"""
# Trigger a checkout
response = requests.post(f"{BASE_URL}/api/checkout", json={
"userId": "test-user",
"cartId": "cart-integration-test-1"
})
assert response.status_code == 200
trace_id = response.headers.get("X-Trace-Id")
assert trace_id, "Response should include X-Trace-Id header"
# Wait for all spans to be flushed to Jaeger
import time; time.sleep(3)
trace = get_trace_by_id(trace_id)
assert trace is not None, f"Trace {trace_id} not found in Jaeger"
spans = flatten_trace_spans(trace)
service_names = {s.service_name for s in spans}
# Verify all expected services appear
expected_services = {"payments-api", "catalog-service", "notification-service"}
missing = expected_services - service_names
assert not missing, f"Missing spans from services: {missing}"
def test_checkout_span_hierarchy_is_correct(self):
"""Spans should have correct parent/child relationships"""
response = requests.post(f"{BASE_URL}/api/checkout", json={
"userId": "test-user",
"cartId": "cart-integration-test-2"
})
trace_id = response.headers["X-Trace-Id"]
import time; time.sleep(3)
trace = get_trace_by_id(trace_id)
spans = flatten_trace_spans(trace)
# Find the root span (no parent)
root_spans = [s for s in spans if s.parent_span_id is None]
assert len(root_spans) == 1, f"Expected 1 root span, got {len(root_spans)}"
root = root_spans[0]
assert root.operation_name == "POST /api/checkout"
# The payment span should be a child of the root
payment_spans = [s for s in spans if s.operation_name == "payments.process"]
assert len(payment_spans) == 1
assert payment_spans[0].parent_span_id == root.span_id, (
"Payment span should be a direct child of the HTTP handler span"
)
# The DB query span should be a child of the payment span
db_spans = [s for s in spans if "db." in s.operation_name.lower()]
assert len(db_spans) > 0, "Expected at least one DB span"
for db_span in db_spans:
# Each DB span should have a payment span or handler span as parent
valid_parent_ids = {payment_spans[0].span_id, root.span_id}
assert db_span.parent_span_id in valid_parent_ids, (
f"DB span parent {db_span.parent_span_id} is not a known span"
)
def test_failed_payment_marks_all_relevant_spans_as_error(self):
"""When payment fails, the payment span and root span should be ERROR"""
response = requests.post(f"{BASE_URL}/api/checkout", json={
"userId": "test-user-declined", # configured to decline in test env
"cartId": "cart-will-decline"
})
assert response.status_code == 402 # Payment Required
trace_id = response.headers["X-Trace-Id"]
import time; time.sleep(3)
trace = get_trace_by_id(trace_id)
spans = flatten_trace_spans(trace)
# Payment span should be marked as error
payment_spans = [s for s in spans if s.operation_name == "payments.process"]
assert len(payment_spans) == 1
payment_span = payment_spans[0]
assert payment_span.tags.get("otel.status_code") == "ERROR", (
"Payment span should be marked ERROR for declined payment"
)
assert "error" in payment_span.tags, "Payment span should have error=true tag"
def test_no_orphaned_spans(self):
"""Every non-root span should have a parent that exists in the trace"""
response = requests.post(f"{BASE_URL}/api/checkout", json={
"userId": "test-user",
"cartId": "cart-orphan-check"
})
trace_id = response.headers["X-Trace-Id"]
import time; time.sleep(3)
trace = get_trace_by_id(trace_id)
spans = flatten_trace_spans(trace)
span_ids = {s.span_id for s in spans}
for span in spans:
if span.parent_span_id is not None:
assert span.parent_span_id in span_ids, (
f"Span {span.operation_name} ({span.span_id}) has orphaned parent "
f"{span.parent_span_id} — parent not found in trace"
)Querying Grafana Tempo
Tempo's HTTP API is more limited than Jaeger's but supports TraceQL for structured queries:
# tests/tempo_helpers.py
import requests
import time
from typing import Optional, List
TEMPO_URL = "http://localhost:3200"
def search_traces(service_name: str, span_name: str = None) -> List[Dict]:
"""Search for traces using Tempo's search API"""
query = f'{{resource.service.name="{service_name}"}}'
if span_name:
query = f'{{resource.service.name="{service_name}" && name="{span_name}"}}'
response = requests.get(
f"{TEMPO_URL}/api/search",
params={"q": query, "limit": 20}
)
response.raise_for_status()
return response.json().get("traces", [])
def get_trace_tempo(trace_id: str) -> Optional[Dict]:
"""Fetch a specific trace from Tempo"""
response = requests.get(f"{TEMPO_URL}/api/traces/{trace_id}")
if response.status_code == 404:
return None
response.raise_for_status()
return response.json()
def query_traceql(query: str) -> List[Dict]:
"""Execute a TraceQL query"""
response = requests.get(
f"{TEMPO_URL}/api/search",
params={"q": query}
)
response.raise_for_status()
return response.json().get("traces", [])def test_traceql_finds_slow_payment_spans(self):
"""Use TraceQL to find traces where payment processing took > 500ms"""
# Generate some slow payments in the test environment
# (assumes a test endpoint that simulates latency)
for _ in range(3):
requests.post(f"{BASE_URL}/api/checkout", json={
"userId": "test-user",
"cartId": f"cart-slow-{time.time()}",
"simulateLatencyMs": 600
})
time.sleep(5) # wait for ingestion
# TraceQL query: find spans named "payments.process" slower than 500ms
traces = query_traceql(
'{resource.service.name="payments-api" && name="payments.process" && duration > 500ms}'
)
assert len(traces) >= 3, (
f"Expected at least 3 slow payment traces, found {len(traces)}"
)Detecting Missing Spans
One of the most useful trace tests is asserting that a specific operation always produces a specific span:
def test_refund_flow_emits_all_required_spans(self):
"""Refund flow should produce exactly these spans"""
REQUIRED_SPANS = {
("payments-api", "POST /api/refunds"),
("payments-api", "refund.validate"),
("payments-api", "refund.process"),
("payments-api", "db.update orders"),
("notification-service", "email.send"),
}
response = requests.post(f"{BASE_URL}/api/refunds", json={
"orderId": "order-integration-test-1",
"reason": "customer_request"
})
assert response.status_code == 200
trace_id = response.headers["X-Trace-Id"]
time.sleep(3)
trace = get_trace_by_id(trace_id)
spans = flatten_trace_spans(trace)
actual_spans = {(s.service_name, s.operation_name) for s in spans}
missing = REQUIRED_SPANS - actual_spans
assert not missing, (
f"Missing required spans in refund flow trace:\n"
+ "\n".join(f" - {service}: {op}" for service, op in sorted(missing))
)CI Workflow
# .github/workflows/trace-integration-tests.yml
name: Distributed Trace Integration Tests
on:
pull_request:
paths:
- 'src/**'
- 'tests/integration/**'
jobs:
trace-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Start test infrastructure
run: docker compose -f docker-compose.test.yml up -d
- name: Wait for services
run: |
timeout 60 bash -c 'until curl -sf http://localhost:16686/; do sleep 2; done'
timeout 60 bash -c 'until curl -sf http://localhost:3200/ready; do sleep 2; done'
timeout 60 bash -c 'until curl -sf http://localhost:8080/health; do sleep 2; done'
- name: Run trace integration tests
run: |
pip install pytest requests
pytest tests/integration/test_trace_structure.py -v --tb=short
- name: Collect traces on failure
if: failure()
run: |
curl -s "http://localhost:16686/api/services" | jq '.'
curl -s "http://localhost:16686/api/traces?service=payments-api&limit=5" \
> /tmp/failing-traces.json
cat /tmp/failing-traces.json
- name: Tear down
if: always()
run: docker compose -f docker-compose.test.yml downTesting distributed traces programmatically gives you the observability equivalent of integration tests: confidence that the pieces fit together, not just that each piece works in isolation. When your trace structure tests are green, you know your dashboards and alerts will have the data they need.