Testing

Asserting OpenTelemetry Spans in Integration Tests

HelpMeTest

HelpMeTest

— 4 min read
Asserting OpenTelemetry Spans in Integration Tests

Unit tests verify logic in isolation. End-to-end tests verify user-visible behavior. Neither tells you whether your distributed system's observability is correct — whether the traces you're emitting are complete, correctly named, and carrying the right attributes. Asserting on OpenTelemetry spans in integration tests fills this gap.

Why Test Traces

Traces break in subtle ways:

  • A new service version stops propagating context headers, silently dropping spans
  • A developer changes a span name that a Grafana dashboard depends on
  • A new code path creates spans without required attributes (user ID, tenant ID, request ID)
  • Sampling is misconfigured and 99% of traces are dropped

These aren't caught by functional tests. But they're testable — if you route spans to an in-memory exporter during tests, you can assert on them like any other output.

The In-Memory Exporter Pattern

The key is using OpenTelemetry's InMemorySpanExporter instead of sending spans to a backend during tests:

import {
  BasicTracerProvider,
  InMemorySpanExporter,
  SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-base';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';

const exporter = new InMemorySpanExporter();

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

// Your test setup
beforeEach(() => {
  exporter.reset();  // clear spans between tests
});

afterAll(() => {
  provider.shutdown();
});

Now every span created during tests is captured in exporter.getFinishedSpans().

Node.js / TypeScript Example

A service that traces HTTP requests:

// src/orders.ts
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('orders-service', '1.0.0');

export async function createOrder(userId: string, items: string[]) {
  return tracer.startActiveSpan('orders.create', async (span) => {
    span.setAttribute('user.id', userId);
    span.setAttribute('order.item_count', items.length);

    try {
      const orderId = await db.insert({ userId, items });
      span.setAttribute('order.id', orderId);
      span.setStatus({ code: SpanStatusCode.OK });
      return orderId;
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}

Test asserting the spans:

import { InMemorySpanExporter } from '@opentelemetry/sdk-trace-base';
import { SpanStatusCode } from '@opentelemetry/api';
import { createOrder } from '../src/orders';

describe('createOrder tracing', () => {
  let exporter: InMemorySpanExporter;

  beforeAll(() => {
    exporter = setupTestTracing();  // returns the exporter from your setup above
  });

  beforeEach(() => exporter.reset());

  it('creates a span with correct attributes', async () => {
    await createOrder('user-123', ['item-a', 'item-b']);

    const spans = exporter.getFinishedSpans();
    expect(spans).toHaveLength(1);

    const span = spans[0];
    expect(span.name).toBe('orders.create');
    expect(span.attributes['user.id']).toBe('user-123');
    expect(span.attributes['order.item_count']).toBe(2);
    expect(span.status.code).toBe(SpanStatusCode.OK);
  });

  it('records exception span on failure', async () => {
    await expect(createOrder('user-invalid', [])).rejects.toThrow();

    const spans = exporter.getFinishedSpans();
    const errorSpan = spans.find(s => s.status.code === SpanStatusCode.ERROR);

    expect(errorSpan).toBeDefined();
    expect(errorSpan!.events).toContainEqual(
      expect.objectContaining({ name: 'exception' })
    );
    expect(errorSpan!.status.message).toContain('validation');
  });
});

Testing Span Hierarchies (Parent-Child Relationships)

Distributed tracing is most valuable when spans are properly nested. Test the hierarchy:

async function processPayment(orderId: string) {
  return tracer.startActiveSpan('payment.process', async (parentSpan) => {
    parentSpan.setAttribute('order.id', orderId);

    // Child span for external API call
    await tracer.startActiveSpan('payment.gateway.charge', async (childSpan) => {
      childSpan.setAttribute('gateway', 'stripe');
      await stripe.charge(orderId);
      childSpan.end();
    });

    parentSpan.end();
  });
}

it('creates parent-child span relationship', async () => {
  await processPayment('order-456');

  const spans = exporter.getFinishedSpans();
  const parent = spans.find(s => s.name === 'payment.process');
  const child = spans.find(s => s.name === 'payment.gateway.charge');

  expect(parent).toBeDefined();
  expect(child).toBeDefined();

  // Child's parent ID should match parent's span ID
  expect(child!.parentSpanId).toBe(parent!.spanContext().spanId);

  // Both should be in the same trace
  expect(child!.spanContext().traceId).toBe(parent!.spanContext().traceId);
});

Python Example with pytest

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
import pytest

@pytest.fixture
def tracer_setup():
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    yield exporter
    exporter.clear()

def create_user(user_id: str, email: str):
    tracer = trace.get_tracer("user-service")
    with tracer.start_as_current_span("user.create") as span:
        span.set_attribute("user.id", user_id)
        span.set_attribute("user.email_domain", email.split("@")[1])
        # ... business logic
        return {"id": user_id}

def test_create_user_emits_correct_span(tracer_setup):
    create_user("u-001", "alice@example.com")

    spans = tracer_setup.get_finished_spans()
    assert len(spans) == 1

    span = spans[0]
    assert span.name == "user.create"
    assert span.attributes["user.id"] == "u-001"
    assert span.attributes["user.email_domain"] == "example.com"
    assert span.status.status_code.name == "OK"

Testing Context Propagation Across HTTP

For microservices, test that trace context flows through HTTP headers:

import { W3CTraceContextPropagator } from '@opentelemetry/core';
import { context, propagation, trace } from '@opentelemetry/api';

it('propagates trace context in outgoing HTTP headers', async () => {
  const capturedHeaders: Record<string, string> = {};

  // Intercept fetch
  jest.spyOn(global, 'fetch').mockImplementation(async (url, init) => {
    Object.assign(capturedHeaders, init?.headers || {});
    return new Response(JSON.stringify({ ok: true }));
  });

  await tracer.startActiveSpan('parent-operation', async (span) => {
    await callDownstreamService();  // internally uses fetch
    span.end();
  });

  // Verify W3C trace context headers were propagated
  expect(capturedHeaders['traceparent']).toMatch(/^00-[a-f0-9]{32}-[a-f0-9]{16}-\d{2}$/);
});

Asserting Semantic Conventions

OpenTelemetry defines semantic conventions for common attributes. Assert that your code follows them:

import { SemanticAttributes } from '@opentelemetry/semantic-conventions';

it('uses semantic conventions for HTTP spans', async () => {
  await httpClient.get('/api/users');

  const spans = exporter.getFinishedSpans();
  const httpSpan = spans.find(s => s.name.startsWith('HTTP'));

  expect(httpSpan!.attributes[SemanticAttributes.HTTP_METHOD]).toBe('GET');
  expect(httpSpan!.attributes[SemanticAttributes.HTTP_URL]).toContain('/api/users');
  expect(httpSpan!.attributes[SemanticAttributes.HTTP_STATUS_CODE]).toBe(200);
});

This prevents drift from standard attribute names that break dashboards and alerts.

Handling Async and Sampling

Flushing spans: With SimpleSpanProcessor, spans are exported synchronously. With BatchSpanProcessor (used in production), you need to flush before asserting:

await provider.forceFlush();
const spans = exporter.getFinishedSpans();

Sampling: If your production config uses sampling, the in-memory exporter may not capture all spans. Override sampling in tests:

import { AlwaysOnSampler } from '@opentelemetry/sdk-trace-base';

const provider = new NodeTracerProvider({
  sampler: new AlwaysOnSampler(),  // capture everything in tests
});

What This Catches in Practice

Teams that add span assertions regularly catch:

  • Missing end() calls (span never closes, causing memory leaks)
  • Wrong status codes (errors marked as OK)
  • Dropped parent context (child spans appear as root spans in Jaeger)
  • PII in attributes (email addresses, passwords accidentally logged as span attributes)
  • Broken instrumentation after library upgrades

Summary

Asserting on OpenTelemetry spans requires three things: an InMemorySpanExporter wired in during tests, exporter.reset() between test cases, and assertions on exporter.getFinishedSpans(). Test span names, attributes, status codes, parent-child relationships, and semantic convention compliance. The patterns work in Node.js, Python, Java, and any language with an OpenTelemetry SDK.

Read more