Testing

OpenTelemetry in CI/CD: Continuous Observability Testing Pipelines

HelpMeTest

HelpMeTest

— 5 min read
OpenTelemetry in CI/CD: Continuous Observability Testing Pipelines

Most teams think of OpenTelemetry as a production concern — you instrument your app, deploy it, and then your observability stack tells you what's happening. CI/CD is where you validate code; OTEL is where you debug production. These two worlds don't meet.

This is a missed opportunity. When you integrate OTEL testing into CI/CD, you catch an entire class of production incident before it happens: the kind where your app is broken but your observability stack doesn't tell you because the instrumentation broke in the last deploy.

This guide covers how to build continuous observability testing into your CI/CD pipeline using OpenTelemetry.

Why Observability Needs CI/CD Testing

Observability regressions happen in ways that are hard to notice:

Instrumentation breaks silently: A developer refactors a service, renames a function, or changes how a library is imported. The OTEL instrumentation that was working stops firing. Spans disappear from traces. Metrics go silent. Production alerting based on those metrics stops working.

Span attributes get dropped: A change in how context is propagated means that span attributes that dashboards rely on (user ID, tenant ID, request type) stop being attached. Dashboards show empty data or misleading aggregations.

Trace propagation breaks: A new service, a new middleware, or a library upgrade breaks W3C trace context propagation. Distributed traces fragment into unconnected spans. Debugging production incidents becomes much harder.

Cardinality explosion: A developer adds a dynamic attribute (like a full URL or a user input) to a span. This explodes cardinality in your metrics backend and can cause billing spikes or performance degradation.

None of these cause immediate test failures. They cause silent degradation of your observability — the kind you discover when you're trying to debug a production incident and your traces are missing.

Architecture: Testing OTEL in CI/CD

The approach is to run your service in CI with a local OTEL collector and assert on the telemetry it emits:

[Your Service] → [OTEL Collector (in-memory)] → [Test Assertions]

The collector captures all spans, metrics, and logs emitted during a test run. Your test assertions query the collector to verify that expected telemetry was emitted correctly.

Setting Up the Test Collector

Use the OTEL collector with the debug exporter or write your own in-memory exporter for tests:

// test-utils/otel-collector.js
import { NodeSDK } from '@opentelemetry/sdk-node';
import { InMemorySpanExporter } from '@opentelemetry/sdk-trace-node';
import { InMemoryMetricExporter } from '@opentelemetry/sdk-metrics';

export class TestCollector {
  constructor() {
    this.spanExporter = new InMemorySpanExporter();
    this.metricExporter = new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE);
  }
  
  setup() {
    this.sdk = new NodeSDK({
      traceExporter: this.spanExporter,
      // metrics exporter if needed
    });
    this.sdk.start();
  }
  
  getSpans() {
    return this.spanExporter.getFinishedSpans();
  }
  
  getSpansByName(name) {
    return this.getSpans().filter(span => span.name === name);
  }
  
  getSpansByAttribute(key, value) {
    return this.getSpans().filter(span => 
      span.attributes[key] === value
    );
  }
  
  reset() {
    this.spanExporter.reset();
  }
  
  async shutdown() {
    await this.sdk.shutdown();
  }
}

Test Structure

// tests/observability/checkout.otel.test.js
import { TestCollector } from '../test-utils/otel-collector';
import { processCheckout } from '../../services/checkout';

describe('checkout observability', () => {
  let collector;
  
  beforeAll(() => {
    collector = new TestCollector();
    collector.setup();
  });
  
  afterAll(() => collector.shutdown());
  afterEach(() => collector.reset());
  
  it('emits a span for checkout processing', async () => {
    await processCheckout({ userId: 'u123', items: ['item-1'] });
    
    const spans = collector.getSpansByName('checkout.process');
    expect(spans).toHaveLength(1);
  });
  
  it('includes required span attributes', async () => {
    await processCheckout({ userId: 'u123', items: ['item-1', 'item-2'] });
    
    const span = collector.getSpansByName('checkout.process')[0];
    
    expect(span.attributes['user.id']).toBe('u123');
    expect(span.attributes['checkout.item_count']).toBe(2);
    expect(span.attributes['service.name']).toBeDefined();
  });
  
  it('records error spans when checkout fails', async () => {
    await expect(processCheckout({ userId: 'u123', items: [] }))
      .rejects.toThrow();
    
    const spans = collector.getSpansByName('checkout.process');
    expect(spans[0].status.code).toBe(SpanStatusCode.ERROR);
    expect(spans[0].status.message).toContain('empty cart');
  });
});

Integrating into GitHub Actions

# .github/workflows/observability.yml
name: Observability Tests

on: [push, pull_request]

jobs:
  otel-tests:
    runs-on: ubuntu-latest
    
    services:
      # Optional: run a real OTEL collector for integration tests
      otel-collector:
        image: otel/opentelemetry-collector-contrib:latest
        ports:
          - 4317:4317
          - 4318:4318
    
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - run: npm ci
      
      - name: Run unit observability tests
        run: npm run test:otel
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318
      
      - name: Run integration observability tests
        run: npm run test:otel:integration
        env:
          OTEL_SERVICE_NAME: test-service
          OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318
      
      - name: Verify no cardinality explosions
        run: node scripts/check-span-cardinality.js

Testing Trace Propagation in CI

The most impactful observability test you can add to CI is verifying that trace context propagates correctly across service boundaries:

// tests/observability/trace-propagation.test.js
import { context, trace } from '@opentelemetry/api';
import { W3CTraceContextPropagator } from '@opentelemetry/core';

describe('trace context propagation', () => {
  it('propagates trace context through HTTP headers', async () => {
    const tracer = trace.getTracer('test');
    let capturedHeaders = {};
    
    // Simulate the outgoing HTTP call
    const span = tracer.startSpan('parent-operation');
    context.with(trace.setSpan(context.active(), span), () => {
      const propagator = new W3CTraceContextPropagator();
      propagator.inject(context.active(), capturedHeaders, {
        set: (headers, key, value) => { headers[key] = value; }
      });
    });
    span.end();
    
    // Verify traceparent header was set correctly
    expect(capturedHeaders['traceparent']).toBeDefined();
    expect(capturedHeaders['traceparent']).toMatch(
      /^00-[a-f0-9]{32}-[a-f0-9]{16}-[0-9]{2}$/
    );
  });
  
  it('extracts parent context from incoming requests', async () => {
    const parentTraceId = 'abc123' + '0'.repeat(26); // 32 hex chars
    const parentSpanId = 'def456' + '0'.repeat(10);   // 16 hex chars
    
    const headers = {
      'traceparent': `00-${parentTraceId}-${parentSpanId}-01`
    };
    
    const propagator = new W3CTraceContextPropagator();
    const extractedContext = propagator.extract(context.active(), headers, {
      get: (headers, key) => headers[key],
      keys: (headers) => Object.keys(headers)
    });
    
    const spanContext = trace.getSpanContext(extractedContext);
    expect(spanContext?.traceId).toBe(parentTraceId);
  });
});

Detecting Cardinality Explosions

High cardinality span attributes can cost you significantly in production observability billing. Add a CI check:

// scripts/check-span-cardinality.js
import { TestCollector } from '../test-utils/otel-collector';
import { runFullTestSuite } from '../test-utils/run-integration-tests';

const MAX_UNIQUE_VALUES_PER_ATTRIBUTE = 100;

async function checkCardinality() {
  const collector = new TestCollector();
  collector.setup();
  
  await runFullTestSuite();
  
  const spans = collector.getSpans();
  const attributeCardinality = {};
  
  for (const span of spans) {
    for (const [key, value] of Object.entries(span.attributes)) {
      if (!attributeCardinality[key]) {
        attributeCardinality[key] = new Set();
      }
      attributeCardinality[key].add(String(value));
    }
  }
  
  const violations = Object.entries(attributeCardinality)
    .filter(([, values]) => values.size > MAX_UNIQUE_VALUES_PER_ATTRIBUTE)
    .map(([key, values]) => ({ key, uniqueValues: values.size }));
  
  if (violations.length > 0) {
    console.error('HIGH CARDINALITY ATTRIBUTES DETECTED:');
    violations.forEach(({ key, uniqueValues }) => {
      console.error(`  ${key}: ${uniqueValues} unique values (limit: ${MAX_UNIQUE_VALUES_PER_ATTRIBUTE})`);
    });
    process.exit(1);
  }
  
  console.log('Cardinality check passed.');
  await collector.shutdown();
}

checkCardinality();

Performance Impact Testing

OTEL instrumentation adds overhead. Test that it doesn't add unacceptable latency:

// tests/observability/performance.test.js
describe('OTEL instrumentation performance overhead', () => {
  const MAX_OVERHEAD_MS = 5; // 5ms max overhead
  
  it('adds minimal overhead to checkout processing', async () => {
    // Baseline: run without instrumentation
    const baseline = await measureTime(() => processCheckoutUninstrumented());
    
    // With instrumentation
    const instrumented = await measureTime(() => processCheckout());
    
    const overhead = instrumented - baseline;
    
    expect(overhead).toBeLessThan(MAX_OVERHEAD_MS);
  });
});

async function measureTime(fn, iterations = 100) {
  const times = [];
  for (let i = 0; i < iterations; i++) {
    const start = performance.now();
    await fn();
    times.push(performance.now() - start);
  }
  return times.reduce((a, b) => a + b) / times.length;
}

The Observability Regression Gate

Define a required CI gate for observability: the deployment can't proceed if observability regresses:

# Required status checks in branch protection:
# - otel-unit-tests
# - otel-integration-tests  
# - otel-cardinality-check
# - otel-propagation-check

When you add these to required status checks, any PR that breaks instrumentation fails CI and can't merge. The developer gets immediate feedback: "your change broke trace propagation in checkout service."

What to Test vs. Skip

Test in every PR:

  • Spans exist for critical business operations
  • Required span attributes are present
  • Error spans are emitted on failure paths
  • Trace context propagation for cross-service calls

Test nightly:

  • Full span coverage across all endpoints
  • Cardinality analysis across realistic test workloads
  • Performance overhead measurements

Don't test:

  • That the OTEL SDK itself works correctly (it's a library with its own tests)
  • That your observability backend (Datadog, Honeycomb, Jaeger) ingests correctly (test at the boundary, not the backend)
  • Every possible attribute combination (too many; test critical ones)

Summary

Observability testing in CI/CD catches a class of production incidents that no other testing approach catches: silent instrumentation failures that leave you blind during an outage.

The minimum viable setup:

  1. Add an in-memory span exporter for unit tests
  2. Assert that key business operations emit spans with required attributes
  3. Test trace context propagation across service boundaries
  4. Add a cardinality check before merge

Once this infrastructure is in place, adding coverage for new features is minimal work: add assertions for the spans the new feature should emit. CI catches regressions automatically. Your observability stack stays healthy across every deploy.

Read more

Start now free