Testing Strategy

Distributed Tracing in Testing: How to Use Traces to Debug Test Failures

HelpMeTest

HelpMeTest

— 8 min read
Distributed Tracing in Testing: How to Use Traces to Debug Test Failures

When a test fails in a distributed system, the error message tells you what failed — not where, why, or which of the twelve services involved was responsible. Distributed tracing in your test suite gives you the full execution path for every test run, turning hours of log-digging into a 30-second trace inspection.

Key Takeaways

Trace context must propagate through test calls. Without injecting W3C trace context headers in test requests, each service creates its own isolated trace and you cannot correlate spans across service boundaries.

In-memory span exporters make trace assertions fast. You do not need a running Jaeger instance in CI — an in-memory exporter collects spans locally and makes them queryable in test assertions.

Span assertions catch behavioral regressions that response assertions miss. A correct response produced by the wrong execution path — hitting the database instead of cache, calling an extra downstream service — is a performance regression waiting to become a correctness regression.

Trace IDs in test logs turn debugging into a one-step lookup. Logging the trace ID alongside every test failure lets developers jump straight to the trace in Jaeger without reproducing the failure.

Sampling must be 100% in test environments. Head-based sampling in production is necessary; in tests it is a bug — sampled-away spans make assertions non-deterministic.

The Distributed System Test Failure Problem

A test fails. The assertion says: Expected 201 Created, got 500 Internal Server Error. You look at the service logs and find: Error: upstream service timeout. You look at the upstream service logs and find nothing — the request may not have arrived, or it arrived and logged in a different format. Twenty minutes later you have found the problem: a database connection pool was exhausted in a third service two hops away.

This is the standard experience of debugging test failures in microservices, and it does not have to be. Distributed tracing gives you the complete execution path — every service, every database query, every external call — for any given request. Adding tracing to your test infrastructure turns distributed test debugging from archaeology into observation.

How Trace Context Propagation Works

The W3C Trace Context standard defines two HTTP headers that connect spans across service boundaries:

  • traceparent: carries the trace ID, parent span ID, and sampling flags
  • tracestate: carries vendor-specific trace state

When service A receives a request with a traceparent header and creates a child span, that span shares the same trace ID. Every service in the call chain that propagates these headers contributes spans to the same trace.

In tests, you need to inject this context explicitly:

// Without trace context — each service creates its own trace
await fetch('http://orders-service/orders', {
  method: 'POST',
  body: JSON.stringify({ items }),
});

// With trace context — all downstream spans join the same trace
const { context, propagation, trace } = require('@opentelemetry/api');

const testTracer = trace.getTracer('test-suite');
const span = testTracer.startSpan('test: create order');

const headers = {};
propagation.inject(trace.setSpan(context.active(), span), headers);

await fetch('http://orders-service/orders', {
  method: 'POST',
  headers: { ...headers, 'Content-Type': 'application/json' },
  body: JSON.stringify({ items }),
});

span.end();

Now every span created by every service handling this request — orders, inventory, payments, notifications — is linked under the same trace ID.

Setting Up In-Process Span Collection for Tests

Running Jaeger in CI just to collect test spans adds operational overhead. Instead, use OpenTelemetry's in-memory span exporter to collect spans locally during test execution:

// test/setup/otel.js
const {
  BasicTracerProvider,
  InMemorySpanExporter,
  SimpleSpanProcessor,
} = require('@opentelemetry/sdk-trace-base');
const { W3CTraceContextPropagator } = require('@opentelemetry/core');
const { context, propagation, trace } = require('@opentelemetry/api');

const spanExporter = new InMemorySpanExporter();
const provider = new BasicTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(spanExporter));
provider.register({
  propagator: new W3CTraceContextPropagator(),
});

module.exports = {
  getSpans: () => spanExporter.getFinishedSpans(),
  resetSpans: () => spanExporter.reset(),
  tracer: trace.getTracer('test-suite'),
};

For integration tests that actually start the service in-process, this works directly — spans created by service code are captured by the same exporter. For tests that call a separately running service, you need a test-specific OTLP endpoint that the service sends spans to.

# docker-compose.test.yml
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    ports:
      - "4318:4318"  # OTLP HTTP

  orders-service:
    build: .
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_TRACES_SAMPLER=always_on  # 100% sampling in tests
// test/helpers/spans.js — query spans from the collector
async function getSpansForTrace(traceId) {
  const response = await fetch(
    `http://localhost:16686/api/traces/${traceId}`
  );
  const data = await response.json();
  return data.data[0]?.spans ?? [];
}

Writing Span Assertions

With spans collected, you can assert on execution paths rather than just response payloads:

// test/integration/checkout.test.js
const { tracer, resetSpans, getSpans } = require('../setup/otel');
const { propagation, context, trace } = require('@opentelemetry/api');

describe('Checkout flow', () => {
  beforeEach(() => resetSpans());

  it('reserves inventory and creates payment intent atomically', async () => {
    const testSpan = tracer.startSpan('test: checkout');
    const headers = {};
    propagation.inject(
      trace.setSpan(context.active(), testSpan),
      headers
    );

    const response = await apiClient.post('/checkout', {
      headers,
      body: { cartId: 'cart-123', paymentMethodId: 'pm-456' },
    });

    testSpan.end();

    expect(response.status).toBe(200);

    // Assert on what happened internally
    const spans = getSpans();

    // Inventory reservation must happen before payment
    const inventorySpan = spans.find(s => s.name === 'inventory.reserve');
    const paymentSpan = spans.find(s => s.name === 'payment.createIntent');

    expect(inventorySpan).toBeDefined();
    expect(paymentSpan).toBeDefined();

    // Inventory starts before payment
    expect(inventorySpan.startTime).toBeLessThan(paymentSpan.startTime);

    // Payment must not start before inventory completes
    const inventoryEnd = inventorySpan.startTime + inventorySpan.duration;
    expect(paymentSpan.startTime).toBeGreaterThanOrEqual(inventoryEnd);

    // No external calls to notification service during checkout
    // (notifications are async, must not block checkout)
    const notificationSpans = spans.filter(s =>
      s.name.includes('notification')
    );
    expect(notificationSpans).toHaveLength(0);
  });
});

This test verifies the order of operations and the absence of unexpected calls. A refactoring that accidentally makes notifications synchronous would fail this test even if the response were still 200.

Debugging Test Failures with Trace IDs

The second major use of tracing in tests is not assertion — it is debugging. When a test fails, the trace ID gives you an immediate path to the full execution context.

// test/helpers/trace-reporter.js
class TraceReporter {
  constructor(jaegerUrl = 'http://localhost:16686') {
    this.jaegerUrl = jaegerUrl;
  }

  onTestFailed(test, traceId) {
    if (traceId) {
      console.error(
        `\n[TRACE] ${test.fullName}\n` +
        `  Jaeger: ${this.jaegerUrl}/trace/${traceId}\n` +
        `  TraceID: ${traceId}`
      );
    }
  }
}
// test/integration/orders.test.js
describe('Order creation', () => {
  let currentTraceId;
  let testSpan;

  beforeEach(() => {
    testSpan = tracer.startSpan('test run');
    currentTraceId = testSpan.spanContext().traceId;
  });

  afterEach(function() {
    if (this.currentTest.state === 'failed') {
      reporter.onTestFailed(this.currentTest, currentTraceId);
    }
    testSpan?.end();
  });

  it('creates order with correct total', async () => {
    const headers = {};
    propagation.inject(trace.setSpan(context.active(), testSpan), headers);

    const response = await apiClient.post('/orders', {
      headers,
      body: { items: [{ productId: 'p1', qty: 3, price: 10.00 }] },
    });

    expect(response.body.total).toBe(30.00);
  });
});

When this test fails, the output includes:

AssertionError: expected 29.97 to equal 30.00

[TRACE] Order creation creates order with correct total
  Jaeger: http://localhost:16686/trace/4bf92f3577b34da6a3ce929d0e0e4736
  TraceID: 4bf92f3577b34da6a3ce929d0e0e4736

Opening that Jaeger URL shows exactly which service computed the total, which database query it read, and whether a discount service was called unexpectedly. Debugging time: 30 seconds instead of 30 minutes.

Asserting on Span Attributes

Spans carry attributes that encode the context of each operation. Testing those attributes verifies that your instrumentation is correct and that the correct context is flowing through your system:

it('attaches user context to all downstream spans', async () => {
  const testSpan = tracer.startSpan('test: user context propagation');
  const headers = {};
  propagation.inject(trace.setSpan(context.active(), testSpan), headers);

  await apiClient.post('/orders', {
    headers: { ...headers, Authorization: 'Bearer user-token-for-user-999' },
    body: { items },
  });

  testSpan.end();

  const spans = getSpans();

  // Every span that touches user data must carry the user ID
  const userDataSpans = spans.filter(s =>
    s.name === 'pg.query' ||
    s.name === 'redis.get' ||
    s.name === 'inventory.check'
  );

  for (const span of userDataSpans) {
    expect(span.attributes['user.id']).toBe('999');
  }
});

This test catches a common security issue: user context failing to propagate through async boundaries, resulting in operations that execute without proper tenant isolation.

Jaeger and Zipkin Integration in CI

For teams running a full observability stack in CI, connecting your test spans to Jaeger or Zipkin is straightforward:

# .github/workflows/integration-tests.yml
services:
  jaeger:
    image: jaegertracing/all-in-one:1.52
    ports:
      - "16686:16686"  # UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
    env:
      COLLECTOR_OTLP_ENABLED: "true"
      SPAN_STORAGE_TYPE: "memory"
// test/setup/otel-ci.js — sends spans to Jaeger in CI
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { InMemorySpanExporter } = require('@opentelemetry/sdk-trace-base');

const exporter = process.env.CI
  ? new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' })
  : new InMemorySpanExporter();

module.exports = { exporter };

With Jaeger running in CI, failed test traces are preserved in Jaeger's memory store during the test run. You can query them via the Jaeger API and attach trace URLs to CI failure annotations, giving developers an instant link from a failed CI check to the full trace.

Sampling Strategy: Always 100% in Tests

In production, head-based sampling at 1-10% is essential for cost and performance. In tests, sampling anything less than 100% means some test runs will have no spans, making assertions non-deterministic. This is always a bug.

// Enforce always-on sampling in test environments
const { AlwaysOnSampler, ParentBasedSampler, TraceIdRatioBased } = require('@opentelemetry/sdk-trace-base');

const sdk = new NodeSDK({
  sampler: process.env.NODE_ENV === 'test'
    ? new AlwaysOnSampler()
    : new ParentBasedSampler({ root: new TraceIdRatioBased(0.01) }),
});

For services that read sampling config from environment variables, set explicitly in your test runner:

OTEL_TRACES_SAMPLER=always_on
OTEL_TRACES_SAMPLER_ARG=1.0

Testing Trace Context Across Async Boundaries

One of the trickiest problems in distributed systems is trace context loss through async operations. If a span is started synchronously but the actual work happens in an async callback, the trace context may not propagate:

// WRONG — context is lost across async boundary
function processOrder(orderId) {
  const span = tracer.startSpan('processOrder');
  setTimeout(() => {
    // This callback runs in a different async context
    // span is not automatically the active span here
    db.query('SELECT * FROM orders WHERE id = ?', [orderId]);
    span.end();
  }, 0);
}

// RIGHT — explicitly propagate context
function processOrder(orderId) {
  const span = tracer.startSpan('processOrder');
  const ctx = trace.setSpan(context.active(), span);

  context.with(ctx, () => {
    setTimeout(() => {
      // context.with() ensures the span is active in this callback
      db.query('SELECT * FROM orders WHERE id = ?', [orderId]);
      span.end();
    }, 0);
  });
}

Write a test that specifically verifies context propagation across your async patterns:

it('maintains trace context through async queue processing', async () => {
  const testSpan = tracer.startSpan('test: async context');
  const headers = {};
  propagation.inject(trace.setSpan(context.active(), testSpan), headers);

  // Submit order (synchronous response, async processing)
  const response = await apiClient.post('/orders', { headers, body: orderData });
  expect(response.status).toBe(202); // Accepted for async processing

  // Wait for async processing
  await waitForOrderProcessed(response.body.orderId);
  testSpan.end();

  const spans = getSpans();
  const processingSpan = spans.find(s => s.name === 'order.process');

  // Processing span must be part of the same trace
  expect(processingSpan.traceId).toBe(testSpan.spanContext().traceId);
});

HelpMeTest: Traces as Test Evidence

When HelpMeTest runs your test scenarios, it collects the full HTTP interaction — request, response, timing, and any correlation IDs in the response headers. If your services emit traceresponse headers (part of the W3C Trace Context spec), HelpMeTest surfaces the trace ID alongside the test result, letting you jump directly from a failed test scenario to the corresponding Jaeger trace with one click.

This closes the loop between synthetic monitoring (HelpMeTest running your scenarios on a schedule) and distributed debugging (Jaeger showing you exactly what happened during the failure).

Conclusion

Distributed tracing belongs in your test suite, not just in your observability stack. Propagating trace context through test requests connects your test assertions to the full execution path across every service involved. In-memory span exporters make this fast and CI-compatible. Span assertions catch behavioral regressions that response assertions miss. And trace IDs in failure output turn debugging from archaeology into observation.

The investment is an afternoon of setup. The return is test failures that tell you not just what broke, but exactly where and why.

Read more

Start now free