Testing Distributed Tracing with OpenTelemetry

HelpMeTest

30 Jul 2026 — 8 min read

Distributed tracing is one of the most valuable observability tools in a microservices architecture. When a request touches ten services before returning a response, traces are what tell you where the time was spent, where the errors happened, and which service is the actual bottleneck.

But tracing instrumentation is code. Code can have bugs. And those bugs are particularly insidious because they fail silently — your service works fine, but the traces are malformed, missing context propagation, or not emitting at all. You discover this at 2am when you actually need the traces to debug an incident.

This post covers how to test your OpenTelemetry instrumentation — making sure it emits the spans you expect, propagates context correctly, and records the right attributes.

Why Tracing Instrumentation Needs Tests

Consider what can go wrong silently:

Broken context propagation: A service extracts the traceparent header but doesn't inject it into outgoing requests. All downstream spans become orphaned root spans. Your traces are split across multiple trace IDs with no way to correlate them.
Missing spans: A critical database call or external API call isn't instrumented. You see the parent span taking 2 seconds with no explanation why.
Wrong span names: Automated instrumentation names spans after HTTP method and path (GET /users/:id), but your cardinality blows up when user IDs end up in span names (GET /users/12345, GET /users/67890, ...).
Dropped attributes: Error details, user IDs, or business context aren't being attached to spans. Traces are present but not useful.
Wrong span status: Errors are being swallowed without marking the span as failed, so error analysis in Jaeger or Tempo is wrong.

None of these cause your service to misbehave. They only matter when you try to use traces to diagnose a problem.

OpenTelemetry Architecture Overview

OpenTelemetry provides:

API: The interfaces your code calls (tracer.Start(), span.SetAttribute(), etc.)
SDK: The implementation that actually records spans and exports them
Exporters: Plugins that send spans to backends (Jaeger, Zipkin, OTLP-compatible backends like Tempo, Honeycomb, etc.)
Collector: An optional intermediary that receives, processes, and exports telemetry

For testing, the key insight is that the SDK is pluggable. You can swap the exporter for an in-memory exporter that collects spans during tests, then assert on them.

Testing with the In-Memory Exporter (Java)

The OpenTelemetry Java SDK ships with an in-memory exporter designed for testing. Here's how to use it:

// pom.xml dependency
// <dependency>
//   <groupId>io.opentelemetry</groupId>
//   <artifactId>opentelemetry-sdk-testing</artifactId>
//   <scope>test</scope>
// </dependency>

import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.sdk.testing.junit5.OpenTelemetryExtension;
import io.opentelemetry.sdk.trace.data.SpanData;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;

import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

class OrderServiceTracingTest {

    @RegisterExtension
    static final OpenTelemetryExtension otelTesting = OpenTelemetryExtension.create();

    private final OrderService orderService = new OrderService(
        otelTesting.getOpenTelemetry()
    );

    @Test
    void createOrder_emitsSpanWithCorrectAttributes() {
        Order order = orderService.createOrder("user-123", List.of("product-456"));

        List<SpanData> spans = otelTesting.getSpans();
        assertThat(spans).hasSize(1);

        SpanData span = spans.get(0);
        assertThat(span.getName()).isEqualTo("order.create");
        assertThat(span.getKind()).isEqualTo(SpanKind.INTERNAL);
        assertThat(span.getStatus().getStatusCode()).isEqualTo(StatusCode.OK);

        // Verify business context is recorded
        assertThat(span.getAttributes().get(
            io.opentelemetry.api.common.AttributeKey.stringKey("user.id")
        )).isEqualTo("user-123");

        assertThat(span.getAttributes().get(
            io.opentelemetry.api.common.AttributeKey.stringKey("order.id")
        )).isEqualTo(order.getId());
    }

    @Test
    void createOrder_marksSpanAsErrorOnFailure() {
        // Simulate a payment failure
        assertThatThrownBy(() -> orderService.createOrder("invalid-user", List.of()))
            .isInstanceOf(OrderCreationException.class);

        List<SpanData> spans = otelTesting.getSpans();
        assertThat(spans).isNotEmpty();

        SpanData rootSpan = spans.stream()
            .filter(s -> s.getName().equals("order.create"))
            .findFirst()
            .orElseThrow();

        assertThat(rootSpan.getStatus().getStatusCode()).isEqualTo(StatusCode.ERROR);
        assertThat(rootSpan.getStatus().getDescription()).contains("invalid user");

        // Verify the exception was recorded
        assertThat(rootSpan.getEvents()).anyMatch(event ->
            event.getName().equals("exception") &&
            event.getAttributes().get(
                io.opentelemetry.api.common.AttributeKey.stringKey("exception.type")
            ).contains("OrderCreationException")
        );
    }

    @Test
    void createOrder_createsChildSpanForDatabaseCall() {
        orderService.createOrder("user-123", List.of("product-456"));

        List<SpanData> spans = otelTesting.getSpans();

        // Find the database span
        SpanData dbSpan = spans.stream()
            .filter(s -> s.getKind() == SpanKind.CLIENT)
            .filter(s -> s.getName().startsWith("INSERT"))
            .findFirst()
            .orElseThrow(() -> new AssertionError("No database span found"));

        // Verify it's a child of the order.create span
        SpanData parentSpan = spans.stream()
            .filter(s -> s.getName().equals("order.create"))
            .findFirst()
            .orElseThrow();

        assertThat(dbSpan.getParentSpanId()).isEqualTo(parentSpan.getSpanId());

        // Verify db.system attribute (semantic conventions)
        assertThat(dbSpan.getAttributes().get(
            io.opentelemetry.api.common.AttributeKey.stringKey("db.system")
        )).isEqualTo("postgresql");
    }
}

Testing Context Propagation (Go)

Context propagation is the most commonly broken part of distributed tracing. This Go test verifies that your HTTP client correctly injects trace context into outgoing requests:

package tracing_test

import (
    "context"
    "net/http"
    "net/http/httptest"
    "testing"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/sdk/trace/tracetest"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/otel/trace"
)

func setupTestTracer(t *testing.T) (*tracetest.SpanRecorder, trace.Tracer) {
    t.Helper()
    recorder := tracetest.NewSpanRecorder()
    provider := sdktrace.NewTracerProvider(
        sdktrace.WithSpanProcessor(recorder),
    )
    otel.SetTracerProvider(provider)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))
    t.Cleanup(func() {
        provider.Shutdown(context.Background())
    })
    return recorder, provider.Tracer("test")
}

func TestHTTPClientPropagatesTraceContext(t *testing.T) {
    recorder, tracer := setupTestTracer(t)

    // Set up a downstream server that captures what headers it receives
    var receivedTraceParent string
    downstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        receivedTraceParent = r.Header.Get("traceparent")
        w.WriteHeader(http.StatusOK)
    }))
    defer downstream.Close()

    // Create a root span
    ctx, rootSpan := tracer.Start(context.Background(), "test-root")
    defer rootSpan.End()

    // Make an instrumented HTTP call
    client := NewInstrumentedHTTPClient() // your instrumented client
    req, _ := http.NewRequestWithContext(ctx, "GET", downstream.URL+"/api/data", nil)
    resp, err := client.Do(req)
    if err != nil {
        t.Fatalf("request failed: %v", err)
    }
    defer resp.Body.Close()

    // Verify the traceparent header was injected
    if receivedTraceParent == "" {
        t.Error("downstream server did not receive traceparent header — context propagation is broken")
    }

    // Verify the trace ID matches the root span
    traceID := rootSpan.SpanContext().TraceID().String()
    if !containsTraceID(receivedTraceParent, traceID) {
        t.Errorf("traceparent header %q does not contain root trace ID %q", receivedTraceParent, traceID)
    }

    // Verify an outgoing span was created
    spans := recorder.Ended()
    outgoingSpans := filterByKind(spans, trace.SpanKindClient)
    if len(outgoingSpans) == 0 {
        t.Error("no CLIENT span was created for the outgoing HTTP request")
    }

    // Verify HTTP semantic conventions
    outgoing := outgoingSpans[0]
    assertAttribute(t, outgoing, string(semconv.HTTPMethodKey), "GET")
    assertAttribute(t, outgoing, string(semconv.HTTPStatusCodeKey), "200")
}

func TestHTTPServerExtractsTraceContext(t *testing.T) {
    recorder, _ := setupTestTracer(t)

    // Start your actual HTTP server (or use httptest)
    handler := NewInstrumentedHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // The handler should have a span in context
        span := trace.SpanFromContext(r.Context())
        if !span.SpanContext().IsValid() {
            t.Error("no valid span in request context")
        }
        w.WriteHeader(http.StatusOK)
    }))

    server := httptest.NewServer(handler)
    defer server.Close()

    // Send request with traceparent header
    req, _ := http.NewRequest("GET", server.URL+"/api/health", nil)
    req.Header.Set("traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        t.Fatalf("request failed: %v", err)
    }
    defer resp.Body.Close()

    spans := recorder.Ended()
    serverSpans := filterByKind(spans, trace.SpanKindServer)
    if len(serverSpans) == 0 {
        t.Fatal("no SERVER span was created")
    }

    serverSpan := serverSpans[0]
    // Verify the server span's parent is the upstream trace
    if serverSpan.Parent().TraceID().String() != "4bf92f3577b34da6a3ce929d0e0e4736" {
        t.Errorf("server span trace ID %q does not match upstream trace ID",
            serverSpan.SpanContext().TraceID())
    }
}

func filterByKind(spans []sdktrace.ReadOnlySpan, kind trace.SpanKind) []sdktrace.ReadOnlySpan {
    var result []sdktrace.ReadOnlySpan
    for _, s := range spans {
        if s.SpanKind() == kind {
            result = append(result, s)
        }
    }
    return result
}

Testing Span Cardinality (Preventing High-Cardinality Names)

High-cardinality span names are a common performance problem in tracing backends. This test verifies your instrumentation uses parameterized names rather than including variable data:

@Test
void httpHandler_usesLowCardinalitySpanNames() {
    // Simulate requests with different user IDs
    orderService.getOrder("order-001");
    orderService.getOrder("order-002");
    orderService.getOrder("order-999");

    List<SpanData> spans = otelTesting.getSpans();
    Set<String> spanNames = spans.stream()
        .map(SpanData::getName)
        .collect(Collectors.toSet());

    // All three requests should produce the same span name
    assertThat(spanNames).containsOnly("order.get");

    // The order ID should be in an attribute, not the name
    spans.forEach(span -> {
        String orderId = span.getAttributes().get(
            AttributeKey.stringKey("order.id")
        );
        assertThat(orderId).isNotNull();
        assertThat(span.getName()).doesNotContain(orderId);
    });
}

Integration Testing with a Real Collector

Unit tests with in-memory exporters cover instrumentation correctness. But you also want to verify that your OTel collector configuration is correct and that spans actually reach your backend. For this, use a test collector:

package integration_test

import (
    "context"
    "testing"
    "time"

    "go.opentelemetry.io/collector/component"
    "go.opentelemetry.io/collector/otelcol"
    // ... collector dependencies
)

func TestSpansReachCollector(t *testing.T) {
    // Start a minimal OTel collector with an in-memory exporter
    // This verifies your OTLP export configuration is correct
    collectorAddr := startTestCollector(t)

    // Configure your service to export to this collector
    tp := newTracerProviderWithOTLP(t, collectorAddr)
    defer tp.Shutdown(context.Background())

    tracer := tp.Tracer("integration-test")
    ctx, span := tracer.Start(context.Background(), "integration-test-span")
    span.SetAttributes(attribute.String("test.id", "integration-001"))
    span.End()
    ctx.Done()

    // Force flush
    tp.ForceFlush(context.Background())

    // Give the collector time to receive
    time.Sleep(500 * time.Millisecond)

    // Query the test collector's captured spans
    receivedSpans := getCollectorSpans(t, collectorAddr)
    if len(receivedSpans) == 0 {
        t.Fatal("no spans received by collector")
    }
}

Testing Baggage Propagation

OpenTelemetry Baggage allows you to propagate key-value pairs across service boundaries. This is useful for passing request context (tenant ID, feature flags, A/B test variant) through a trace. Test it explicitly:

@Test
void baggageIsForwardedToDownstreamServices() {
    Baggage baggage = Baggage.builder()
        .put("tenant.id", "acme-corp")
        .put("feature.flag", "new-checkout-flow")
        .build();

    Context ctxWithBaggage = baggage.storeInContext(Context.current());

    try (Scope scope = ctxWithBaggage.makeCurrent()) {
        orderService.createOrder("user-123", List.of("product-456"));
    }

    List<SpanData> spans = otelTesting.getSpans();

    // Downstream spans should have baggage values available
    // (your instrumentation should copy important baggage to span attributes)
    SpanData orderSpan = spans.stream()
        .filter(s -> s.getName().equals("order.create"))
        .findFirst()
        .orElseThrow();

    assertThat(orderSpan.getAttributes().get(
        AttributeKey.stringKey("tenant.id")
    )).isEqualTo("acme-corp");
}

Sampling Strategy Testing

Not all spans should be exported — at scale, 100% sampling is expensive. Test that your sampling configuration is working as intended:

@Test
void parentBasedSampling_samplesForcedTraces() {
    // Simulate an upstream service that forces sampling on
    TraceFlags sampledFlags = TraceFlags.getSampled();
    SpanContext upstreamCtx = SpanContext.createFromRemoteParent(
        TraceId.fromLongs(1L, 1L),
        SpanId.fromLong(1L),
        sampledFlags,
        TraceState.getDefault()
    );

    Context ctx = Context.root().with(Span.wrap(upstreamCtx));
    try (Scope scope = ctx.makeCurrent()) {
        orderService.createOrder("user-123", List.of("product-456"));
    }

    // All spans should be sampled because parent was sampled
    List<SpanData> spans = otelTesting.getSpans();
    assertThat(spans).isNotEmpty();
    spans.forEach(span ->
        assertThat(span.getSpanContext().isSampled()).isTrue()
    );
}

Semantic Conventions Compliance

OpenTelemetry defines semantic conventions — standard attribute names for common operations. Validating compliance ensures your spans are compatible with standard dashboards and queries:

func TestHTTPSpanFollowsSemanticConventions(t *testing.T) {
    recorder, _ := setupTestTracer(t)

    // Make a request to your service
    resp, _ := http.Get("http://localhost:8080/api/orders/123")
    defer resp.Body.Close()

    spans := recorder.Ended()
    serverSpan := findSpanByKind(spans, trace.SpanKindServer)
    if serverSpan == nil {
        t.Fatal("no server span")
    }

    attrs := spanAttrsToMap(serverSpan)

    // Required HTTP server attributes per OTel semantic conventions
    required := []string{
        "http.method",
        "http.route",      // Must be parameterized: /api/orders/:id, not /api/orders/123
        "http.status_code",
        "net.host.name",
    }
    for _, key := range required {
        if _, ok := attrs[key]; !ok {
            t.Errorf("missing required attribute: %s", key)
        }
    }

    // http.route must NOT contain the actual order ID (cardinality)
    route, _ := attrs["http.route"].(string)
    if route == "/api/orders/123" {
        t.Error("http.route contains actual ID — should be parameterized as /api/orders/:id")
    }
}

Running Tracing Tests in CI

Tracing tests split into two categories for CI:

Fast unit tests (in-memory exporter): Run on every commit. No external dependencies. Catch instrumentation bugs, missing spans, wrong attributes.

Integration tests (test collector): Run in CI on a dedicated stage with a Docker Compose or Kubernetes environment. Verify OTLP export, collector pipeline, and backend ingestion.

# .github/workflows/test.yaml
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tracing unit tests
        run: ./gradlew test --tests "*TracingTest"

  integration-tests:
    runs-on: ubuntu-latest
    services:
      otel-collector:
        image: otel/opentelemetry-collector-contrib:latest
        ports:
          - 4317:4317
          - 4318:4318
    steps:
      - uses: actions/checkout@v4
      - name: Run tracing integration tests
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4317
        run: ./gradlew integrationTest --tests "*TracingIntegrationTest"

Wrapping Up

Distributed tracing is only useful if the instrumentation is correct. Broken context propagation, missing spans, and wrong attributes fail silently — they don't break your service, they just make your traces useless when you actually need them.

The in-memory exporter pattern makes tracing instrumentation testable without any external infrastructure. The investment is low: a few dozen lines of test code. The payoff is confidence that your traces will actually help you when a production incident requires them.

Start with three tests for every service: one verifying the happy path emits the expected spans, one verifying errors are correctly marked as failures, and one verifying context propagation works correctly. Build from there.