Testing Distributed Traces with Jaeger and Grafana Tempo

Testing Distributed Traces with Jaeger and Grafana Tempo

In-memory exporter tests verify that your instrumentation emits the right spans in isolation. But in a distributed system, spans across multiple services need to connect into coherent traces — same trace ID, correct parent/child relationships, no orphaned spans. That's what integration tests against a real tracing backend validate.

This post covers how to test distributed traces programmatically against Jaeger and Grafana Tempo, using Docker Compose for local and CI environments.

Why Test Trace Structure

Instrumentation correctness tests (in-memory exporters) verify that individual spans have the right attributes. Trace structure tests verify something different:

  • Trace completeness: all expected services appear in the trace
  • Parent/child relationships: the checkout span is the parent of the payment span, which is the parent of the DB query span
  • Span ordering: operations appear in the correct temporal sequence
  • Missing spans: a refund flow that should have 6 spans is only producing 4
  • Unexpected spans: a span from a service that should be bypassed is appearing

These problems are invisible in unit tests. They only appear when you exercise the full request flow across services and inspect the resulting trace.

Docker Compose Setup

# docker-compose.test.yml
services:
  # Jaeger all-in-one (for simpler setup)
  jaeger:
    image: jaegertracing/all-in-one:1.52
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:16686/"]
      interval: 5s
      timeout: 5s
      retries: 10

  # Grafana Tempo (for Prometheus/Loki integration)
  tempo:
    image: grafana/tempo:2.3.1
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo-test.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"   # HTTP API
      - "4319:4317"   # OTLP gRPC (different port to avoid conflict with Jaeger)
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3200/ready"]
      interval: 5s
      retries: 10

  # Your service under test
  payments-api:
    build: .
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
      OTEL_SERVICE_NAME: payments-api
      OTEL_TRACES_EXPORTER: otlp
    depends_on:
      jaeger:
        condition: service_healthy
    ports:
      - "8080:8080"
# tempo-test.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"

ingester:
  trace_idle_period: 10s

storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal
    local:
      path: /tmp/tempo/blocks

compactor:
  compaction:
    compaction_window: 1h

Querying Jaeger Programmatically

Jaeger exposes a REST API for querying traces. Use it from your integration tests:

# tests/trace_helpers.py
import time
import requests
from typing import Optional, List, Dict, Any
from dataclasses import dataclass

JAEGER_URL = "http://localhost:16686"

@dataclass
class Span:
    span_id: str
    operation_name: str
    service_name: str
    parent_span_id: Optional[str]
    tags: Dict[str, Any]
    duration_us: int
    logs: List[Dict]

def get_traces_for_service(
    service: str, 
    operation: Optional[str] = None,
    limit: int = 20,
    lookback: str = "1h"
) -> List[Dict]:
    """Fetch recent traces from Jaeger for a given service"""
    params = {
        "service": service,
        "limit": limit,
        "lookback": lookback,
    }
    if operation:
        params["operation"] = operation
    
    response = requests.get(f"{JAEGER_URL}/api/traces", params=params)
    response.raise_for_status()
    return response.json()["data"]

def get_trace_by_id(trace_id: str) -> Optional[Dict]:
    """Fetch a specific trace by ID"""
    response = requests.get(f"{JAEGER_URL}/api/traces/{trace_id}")
    if response.status_code == 404:
        return None
    response.raise_for_status()
    return response.json()["data"][0]

def flatten_trace_spans(trace: Dict) -> List[Span]:
    """Convert Jaeger trace format to a list of Span objects"""
    processes = trace["processes"]
    spans = []
    
    for raw_span in trace["spans"]:
        process_id = raw_span["processID"]
        service_name = processes[process_id]["serviceName"]
        
        parent_span_id = None
        for ref in raw_span.get("references", []):
            if ref["refType"] == "CHILD_OF":
                parent_span_id = ref["spanID"]
                break
        
        tags = {tag["key"]: tag["value"] for tag in raw_span.get("tags", [])}
        
        spans.append(Span(
            span_id=raw_span["spanID"],
            operation_name=raw_span["operationName"],
            service_name=service_name,
            parent_span_id=parent_span_id,
            tags=tags,
            duration_us=raw_span["duration"],
            logs=raw_span.get("logs", [])
        ))
    
    return spans

def wait_for_trace(service: str, operation: str, 
                   timeout_seconds: int = 30) -> Optional[Dict]:
    """Poll Jaeger until a trace appears, or timeout"""
    start = time.time()
    while time.time() - start < timeout_seconds:
        traces = get_traces_for_service(service, operation, lookback="5m")
        if traces:
            return traces[0]
        time.sleep(2)
    return None

Asserting Trace Structure

# tests/integration/test_trace_structure.py
import pytest
import requests
from tests.trace_helpers import (
    wait_for_trace, flatten_trace_spans, get_trace_by_id
)

BASE_URL = "http://localhost:8080"

class TestCheckoutTraceStructure:
    def test_checkout_produces_complete_trace(self):
        """A checkout request should produce spans from 3 services"""
        # Trigger a checkout
        response = requests.post(f"{BASE_URL}/api/checkout", json={
            "userId": "test-user",
            "cartId": "cart-integration-test-1"
        })
        assert response.status_code == 200
        
        trace_id = response.headers.get("X-Trace-Id")
        assert trace_id, "Response should include X-Trace-Id header"
        
        # Wait for all spans to be flushed to Jaeger
        import time; time.sleep(3)
        
        trace = get_trace_by_id(trace_id)
        assert trace is not None, f"Trace {trace_id} not found in Jaeger"
        
        spans = flatten_trace_spans(trace)
        service_names = {s.service_name for s in spans}
        
        # Verify all expected services appear
        expected_services = {"payments-api", "catalog-service", "notification-service"}
        missing = expected_services - service_names
        assert not missing, f"Missing spans from services: {missing}"

    def test_checkout_span_hierarchy_is_correct(self):
        """Spans should have correct parent/child relationships"""
        response = requests.post(f"{BASE_URL}/api/checkout", json={
            "userId": "test-user",
            "cartId": "cart-integration-test-2"
        })
        trace_id = response.headers["X-Trace-Id"]
        
        import time; time.sleep(3)
        trace = get_trace_by_id(trace_id)
        spans = flatten_trace_spans(trace)
        
        # Find the root span (no parent)
        root_spans = [s for s in spans if s.parent_span_id is None]
        assert len(root_spans) == 1, f"Expected 1 root span, got {len(root_spans)}"
        root = root_spans[0]
        assert root.operation_name == "POST /api/checkout"
        
        # The payment span should be a child of the root
        payment_spans = [s for s in spans if s.operation_name == "payments.process"]
        assert len(payment_spans) == 1
        assert payment_spans[0].parent_span_id == root.span_id, (
            "Payment span should be a direct child of the HTTP handler span"
        )
        
        # The DB query span should be a child of the payment span
        db_spans = [s for s in spans if "db." in s.operation_name.lower()]
        assert len(db_spans) > 0, "Expected at least one DB span"
        for db_span in db_spans:
            # Each DB span should have a payment span or handler span as parent
            valid_parent_ids = {payment_spans[0].span_id, root.span_id}
            assert db_span.parent_span_id in valid_parent_ids, (
                f"DB span parent {db_span.parent_span_id} is not a known span"
            )

    def test_failed_payment_marks_all_relevant_spans_as_error(self):
        """When payment fails, the payment span and root span should be ERROR"""
        response = requests.post(f"{BASE_URL}/api/checkout", json={
            "userId": "test-user-declined",  # configured to decline in test env
            "cartId": "cart-will-decline"
        })
        assert response.status_code == 402  # Payment Required
        
        trace_id = response.headers["X-Trace-Id"]
        import time; time.sleep(3)
        
        trace = get_trace_by_id(trace_id)
        spans = flatten_trace_spans(trace)
        
        # Payment span should be marked as error
        payment_spans = [s for s in spans if s.operation_name == "payments.process"]
        assert len(payment_spans) == 1
        
        payment_span = payment_spans[0]
        assert payment_span.tags.get("otel.status_code") == "ERROR", (
            "Payment span should be marked ERROR for declined payment"
        )
        assert "error" in payment_span.tags, "Payment span should have error=true tag"

    def test_no_orphaned_spans(self):
        """Every non-root span should have a parent that exists in the trace"""
        response = requests.post(f"{BASE_URL}/api/checkout", json={
            "userId": "test-user",
            "cartId": "cart-orphan-check"
        })
        trace_id = response.headers["X-Trace-Id"]
        
        import time; time.sleep(3)
        trace = get_trace_by_id(trace_id)
        spans = flatten_trace_spans(trace)
        
        span_ids = {s.span_id for s in spans}
        
        for span in spans:
            if span.parent_span_id is not None:
                assert span.parent_span_id in span_ids, (
                    f"Span {span.operation_name} ({span.span_id}) has orphaned parent "
                    f"{span.parent_span_id} — parent not found in trace"
                )

Querying Grafana Tempo

Tempo's HTTP API is more limited than Jaeger's but supports TraceQL for structured queries:

# tests/tempo_helpers.py
import requests
import time
from typing import Optional, List

TEMPO_URL = "http://localhost:3200"

def search_traces(service_name: str, span_name: str = None) -> List[Dict]:
    """Search for traces using Tempo's search API"""
    query = f'{{resource.service.name="{service_name}"}}'
    if span_name:
        query = f'{{resource.service.name="{service_name}" && name="{span_name}"}}'
    
    response = requests.get(
        f"{TEMPO_URL}/api/search",
        params={"q": query, "limit": 20}
    )
    response.raise_for_status()
    return response.json().get("traces", [])

def get_trace_tempo(trace_id: str) -> Optional[Dict]:
    """Fetch a specific trace from Tempo"""
    response = requests.get(f"{TEMPO_URL}/api/traces/{trace_id}")
    if response.status_code == 404:
        return None
    response.raise_for_status()
    return response.json()

def query_traceql(query: str) -> List[Dict]:
    """Execute a TraceQL query"""
    response = requests.get(
        f"{TEMPO_URL}/api/search",
        params={"q": query}
    )
    response.raise_for_status()
    return response.json().get("traces", [])
def test_traceql_finds_slow_payment_spans(self):
    """Use TraceQL to find traces where payment processing took > 500ms"""
    # Generate some slow payments in the test environment
    # (assumes a test endpoint that simulates latency)
    for _ in range(3):
        requests.post(f"{BASE_URL}/api/checkout", json={
            "userId": "test-user",
            "cartId": f"cart-slow-{time.time()}",
            "simulateLatencyMs": 600
        })
    
    time.sleep(5)  # wait for ingestion
    
    # TraceQL query: find spans named "payments.process" slower than 500ms
    traces = query_traceql(
        '{resource.service.name="payments-api" && name="payments.process" && duration > 500ms}'
    )
    
    assert len(traces) >= 3, (
        f"Expected at least 3 slow payment traces, found {len(traces)}"
    )

Detecting Missing Spans

One of the most useful trace tests is asserting that a specific operation always produces a specific span:

def test_refund_flow_emits_all_required_spans(self):
    """Refund flow should produce exactly these spans"""
    REQUIRED_SPANS = {
        ("payments-api", "POST /api/refunds"),
        ("payments-api", "refund.validate"),
        ("payments-api", "refund.process"),
        ("payments-api", "db.update orders"),
        ("notification-service", "email.send"),
    }
    
    response = requests.post(f"{BASE_URL}/api/refunds", json={
        "orderId": "order-integration-test-1",
        "reason": "customer_request"
    })
    assert response.status_code == 200
    
    trace_id = response.headers["X-Trace-Id"]
    time.sleep(3)
    
    trace = get_trace_by_id(trace_id)
    spans = flatten_trace_spans(trace)
    
    actual_spans = {(s.service_name, s.operation_name) for s in spans}
    missing = REQUIRED_SPANS - actual_spans
    
    assert not missing, (
        f"Missing required spans in refund flow trace:\n"
        + "\n".join(f"  - {service}: {op}" for service, op in sorted(missing))
    )

CI Workflow

# .github/workflows/trace-integration-tests.yml
name: Distributed Trace Integration Tests

on:
  pull_request:
    paths:
      - 'src/**'
      - 'tests/integration/**'

jobs:
  trace-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Start test infrastructure
        run: docker compose -f docker-compose.test.yml up -d
      
      - name: Wait for services
        run: |
          timeout 60 bash -c 'until curl -sf http://localhost:16686/; do sleep 2; done'
          timeout 60 bash -c 'until curl -sf http://localhost:3200/ready; do sleep 2; done'
          timeout 60 bash -c 'until curl -sf http://localhost:8080/health; do sleep 2; done'

      - name: Run trace integration tests
        run: |
          pip install pytest requests
          pytest tests/integration/test_trace_structure.py -v --tb=short

      - name: Collect traces on failure
        if: failure()
        run: |
          curl -s "http://localhost:16686/api/services" | jq '.'
          curl -s "http://localhost:16686/api/traces?service=payments-api&limit=5" \
            > /tmp/failing-traces.json
          cat /tmp/failing-traces.json
      
      - name: Tear down
        if: always()
        run: docker compose -f docker-compose.test.yml down

Testing distributed traces programmatically gives you the observability equivalent of integration tests: confidence that the pieces fit together, not just that each piece works in isolation. When your trace structure tests are green, you know your dashboards and alerts will have the data they need.

Read more

Start now free