Testing OpenTelemetry SDK Instrumentation Correctness
When you add OpenTelemetry instrumentation to your codebase, you're writing code that needs to be tested like any other code. A function that creates a span, records attributes, and handles errors can have bugs. Without tests, you discover them in production when your traces are missing or incorrect.
This guide covers how to unit test OTEL SDK instrumentation code — the layer where developers write custom spans, record attributes, and add context to auto-instrumented libraries.
What Custom Instrumentation Needs Testing
Auto-instrumentation handles most of the basics — HTTP requests, database calls, and popular framework integrations are instrumented automatically by the OTEL SDK. What needs manual testing is your custom instrumentation:
Business operation spans: Spans you create for important business operations that aren't auto-instrumented (checkout.process, payment.verify, recommendation.generate).
Custom attributes: Attributes you add to spans to make traces useful — user ID, tenant ID, feature flags, business context.
Error recording: Custom error handling that records exceptions on spans.
Context enrichment: Adding baggage, span links, or parent context to spans created in background jobs or async operations.
Instrumentation libraries you write: If you wrap a third-party library with OTEL instrumentation, that wrapper needs tests.
Setting Up the Test Environment
The key dependency for instrumentation unit tests is the in-memory exporter:
// test-utils/setup-otel.js
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { InMemorySpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-node';
export function setupTestTracing() {
const exporter = new InMemorySpanExporter();
const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();
return {
exporter,
provider,
getSpans: () => exporter.getFinishedSpans(),
reset: () => exporter.reset(),
shutdown: () => provider.shutdown(),
};
}Use SimpleSpanProcessor in tests (not BatchSpanProcessor) — it exports synchronously when spans end, making assertions predictable without waiting.
Testing Span Creation
// services/checkout.js
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('checkout-service');
export async function processCheckout(order) {
const span = tracer.startSpan('checkout.process', {
attributes: {
'checkout.order_id': order.id,
'checkout.item_count': order.items.length,
'checkout.user_id': order.userId,
}
});
try {
const result = await chargePayment(order);
span.setAttribute('checkout.payment_method', result.method);
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (error) {
span.recordException(error);
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
throw error;
} finally {
span.end();
}
}// tests/services/checkout.instrumentation.test.js
import { setupTestTracing } from '../test-utils/setup-otel';
import { processCheckout } from '../../services/checkout';
import { SpanStatusCode, SpanKind } from '@opentelemetry/api';
describe('checkout instrumentation', () => {
let tracing;
beforeAll(() => { tracing = setupTestTracing(); });
afterAll(() => tracing.shutdown());
afterEach(() => tracing.reset());
describe('span creation', () => {
it('creates a span named checkout.process', async () => {
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['item-1'] });
const spans = tracing.getSpans();
expect(spans).toHaveLength(1);
expect(spans[0].name).toBe('checkout.process');
});
it('spans are of INTERNAL kind for business operations', async () => {
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['item-1'] });
const span = tracing.getSpans()[0];
expect(span.kind).toBe(SpanKind.INTERNAL);
});
it('span ends after the operation completes', async () => {
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['item-1'] });
const span = tracing.getSpans()[0];
expect(span.endTime).toBeDefined();
expect(span.endTime[0]).toBeGreaterThan(span.startTime[0]); // endTime > startTime
});
it('creates exactly one span per checkout (not one per item)', async () => {
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['item-1', 'item-2', 'item-3'] });
const checkoutSpans = tracing.getSpans()
.filter(s => s.name === 'checkout.process');
expect(checkoutSpans).toHaveLength(1); // Not 3
});
});
describe('attribute recording', () => {
it('records order ID attribute', async () => {
await processCheckout({ id: 'ord-test-123', userId: 'u1', items: ['item-1'] });
const span = tracing.getSpans()[0];
expect(span.attributes['checkout.order_id']).toBe('ord-test-123');
});
it('records item count as a number', async () => {
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['a', 'b', 'c'] });
const span = tracing.getSpans()[0];
const itemCount = span.attributes['checkout.item_count'];
expect(typeof itemCount).toBe('number');
expect(itemCount).toBe(3);
});
it('records payment method after successful payment', async () => {
// Mock the payment to return a specific method
jest.spyOn(paymentService, 'charge').mockResolvedValue({ method: 'visa' });
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['item-1'] });
const span = tracing.getSpans()[0];
expect(span.attributes['checkout.payment_method']).toBe('visa');
});
});
describe('status recording', () => {
it('sets OK status on successful checkout', async () => {
await processCheckout({ id: 'ord-1', userId: 'u1', items: ['item-1'] });
const span = tracing.getSpans()[0];
expect(span.status.code).toBe(SpanStatusCode.OK);
});
it('sets ERROR status when checkout fails', async () => {
await expect(
processCheckout({ id: 'ord-1', userId: 'u1', items: [] }) // empty cart fails
).rejects.toThrow();
const span = tracing.getSpans()[0];
expect(span.status.code).toBe(SpanStatusCode.ERROR);
expect(span.status.message).toBeTruthy();
});
it('records exception details on error', async () => {
await expect(
processCheckout({ id: 'ord-1', userId: 'u1', items: [] })
).rejects.toThrow('Cart is empty');
const span = tracing.getSpans()[0];
const exceptionEvents = span.events.filter(e => e.name === 'exception');
expect(exceptionEvents).toHaveLength(1);
expect(exceptionEvents[0].attributes['exception.type']).toBeDefined();
expect(exceptionEvents[0].attributes['exception.message']).toContain('Cart is empty');
});
it('ends span even when an exception is thrown', async () => {
await expect(
processCheckout({ id: 'ord-1', userId: 'u1', items: [] })
).rejects.toThrow();
// Span should still be in finished spans (ended in finally block)
const spans = tracing.getSpans();
expect(spans).toHaveLength(1);
expect(spans[0].endTime).toBeDefined();
});
});
});Testing Context Propagation in Custom Code
// services/background-jobs.js
import { context, trace } from '@opentelemetry/api';
export function scheduleBackgroundJob(parentContext, jobData) {
// Context must be explicitly passed to async callbacks
const boundContext = context.with(parentContext, async () => {
const tracer = trace.getTracer('background-jobs');
const span = tracer.startSpan('background.job.process');
try {
await processJob(jobData);
span.setStatus({ code: SpanStatusCode.OK });
} finally {
span.end();
}
});
return boundContext;
}describe('background job context propagation', () => {
it('background job span is child of parent span', async () => {
const spans = await captureSpans(async () => {
// Create parent span
const parentSpan = tracer.startSpan('parent.operation');
const parentContext = trace.setSpan(context.active(), parentSpan);
await scheduleBackgroundJob(parentContext, { type: 'email' });
parentSpan.end();
});
const parentSpan = spans.find(s => s.name === 'parent.operation');
const jobSpan = spans.find(s => s.name === 'background.job.process');
expect(parentSpan).toBeDefined();
expect(jobSpan).toBeDefined();
// Job span should be a child of parent span
expect(jobSpan.parentSpanId).toBe(parentSpan.spanContext().spanId);
// Both should have the same trace ID
expect(jobSpan.spanContext().traceId).toBe(parentSpan.spanContext().traceId);
});
it('background job without context creates new trace', async () => {
const spans = await captureSpans(async () => {
// No parent context passed
await scheduleBackgroundJob(context.active(), { type: 'cleanup' });
});
const jobSpan = spans.find(s => s.name === 'background.job.process');
expect(jobSpan.parentSpanId).toBeUndefined(); // root span
});
});Testing Instrumentation Libraries
If you write a wrapper around a third-party library:
// lib/redis-instrumented.js
export class InstrumentedRedis {
constructor(client) {
this.client = client;
this.tracer = trace.getTracer('redis-instrumentation');
}
async get(key) {
const span = this.tracer.startSpan('redis.get', {
kind: SpanKind.CLIENT,
attributes: {
'db.system': 'redis',
'db.operation': 'GET',
'db.redis.key': key,
}
});
try {
const value = await this.client.get(key);
span.setAttribute('db.redis.hit', value !== null);
return value;
} catch (error) {
span.recordException(error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw error;
} finally {
span.end();
}
}
}// tests/lib/redis-instrumented.test.js
describe('InstrumentedRedis', () => {
let mockRedis;
let instrumentedRedis;
let tracing;
beforeAll(() => {
tracing = setupTestTracing();
mockRedis = { get: jest.fn() };
instrumentedRedis = new InstrumentedRedis(mockRedis);
});
afterAll(() => tracing.shutdown());
afterEach(() => { tracing.reset(); jest.clearAllMocks(); });
it('creates a CLIENT span for redis GET', async () => {
mockRedis.get.mockResolvedValue('cached-value');
await instrumentedRedis.get('my-key');
const span = tracing.getSpans()[0];
expect(span.name).toBe('redis.get');
expect(span.kind).toBe(SpanKind.CLIENT);
});
it('records cache hit/miss attribute', async () => {
mockRedis.get.mockResolvedValue(null); // cache miss
await instrumentedRedis.get('missing-key');
const span = tracing.getSpans()[0];
expect(span.attributes['db.redis.hit']).toBe(false);
});
it('does not record key value — only key name', async () => {
mockRedis.get.mockResolvedValue('secret-session-data');
await instrumentedRedis.get('session:user123');
const span = tracing.getSpans()[0];
const spanJson = JSON.stringify(span);
// Value should never appear in span data
expect(spanJson).not.toContain('secret-session-data');
// Key name is OK
expect(span.attributes['db.redis.key']).toBe('session:user123');
});
});Common Instrumentation Bugs the Tests Catch
The tests above are specifically designed to catch real bugs:
Span not ended on exception (tested by "ends span even when exception thrown"): The most common instrumentation bug. If you don't end the span in a finally block, the span leaks and never appears in your traces.
Item-count bug (tested by "creates exactly one span per checkout"): Instrumentation inside a loop creates multiple spans when there should be one.
Attribute set before value available (tested by "records payment method after successful payment"): Setting an attribute before the value exists results in undefined or empty attribute.
Context not propagated to async code (tested by context propagation tests): Async callbacks without explicit context binding create disconnected traces.
Exception recorded but status not set (tested by error status tests): recordException and setStatus(ERROR) are separate operations. Recording the exception without setting error status means the span appears successful.
Summary
Testing OTEL SDK instrumentation is unit testing your observability code. The patterns are the same as any unit test: set up, exercise, assert.
The assertions to prioritize:
- Span exists with the correct name and kind
- Required attributes are set with correct values and types
- Status is set correctly (OK on success, ERROR on failure)
- Exceptions are recorded with
recordException - Spans always end (even on exception)
- Context is propagated to async/background operations
Write these tests when you write the instrumentation. The feedback loop is fast — an in-memory exporter lets you verify span output in milliseconds without any external dependencies.
