Testing

Testing OpenTelemetry Metrics and Alerting: Validate Before Production

HelpMeTest

HelpMeTest

— 5 min read
Testing OpenTelemetry Metrics and Alerting: Validate Before Production

Your OTEL metrics instrumentation is only as good as your confidence that it measures what it claims to measure. A checkout success counter that double-counts, a latency histogram with wrong boundaries, or an alert rule that fires too late (or never) — these are observability bugs that cause real production incidents.

Testing metrics and alerting is different from testing traces. Traces are about individual requests; metrics are about aggregated behavior over time. The testing approach needs to match.

What Can Go Wrong With OTEL Metrics

Counter semantics errors: A checkout.success counter increments once per item instead of once per order. Your checkout success rate dashboard shows 5x the actual rate.

Histogram bucket misconfiguration: Your API latency histogram has buckets at [10ms, 50ms, 100ms, 500ms, 1s]. Your P99 latency is 800ms. The histogram can only tell you it's somewhere between 500ms and 1s — not useful for SLO compliance.

Gauge staleness: A gauge metric that should update on every request stops updating after a configuration change. Your dashboards show stale values. Alerting based on the gauge stops working.

Missing dimensions: You add a service.region attribute to new metrics but forget to add it to existing metrics. Cross-region comparisons on dashboards break.

Alert query bugs: Your alert fires count(errors) > 100. Your error counter resets every deployment. The alert fires after every deploy regardless of error rate.

Unit mismatches: Your latency metric records in milliseconds. Your alert threshold is set to > 1 assuming seconds. Alert never fires.

Setting Up Metrics Testing

The OTEL SDK provides in-memory metric exporters that work for unit testing:

// test-utils/metrics-collector.js
import { MeterProvider } from '@opentelemetry/sdk-metrics';
import { InMemoryMetricExporter } from '@opentelemetry/sdk-metrics';
import { AggregationTemporality } from '@opentelemetry/sdk-metrics';

export class MetricsTestCollector {
  constructor() {
    this.exporter = new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE);
    this.meterProvider = new MeterProvider();
    this.meterProvider.addMetricReader(
      new PeriodicExportingMetricReader({
        exporter: this.exporter,
        exportIntervalMillis: 100, // Short interval for tests
      })
    );
  }
  
  getMeter(name) {
    return this.meterProvider.getMeter(name);
  }
  
  async flush() {
    await this.meterProvider.forceFlush();
  }
  
  getMetrics() {
    return this.exporter.getMetrics();
  }
  
  getMetricByName(name) {
    return this.getMetrics()
      .flatMap(rm => rm.scopeMetrics)
      .flatMap(sm => sm.metrics)
      .find(m => m.descriptor.name === name);
  }
  
  reset() {
    this.exporter.reset();
  }
  
  async shutdown() {
    await this.meterProvider.shutdown();
  }
}

Testing Counter Semantics

// tests/observability/metrics.test.js
import { MetricsTestCollector } from '../test-utils/metrics-collector';
import { CheckoutService } from '../../services/checkout';

describe('checkout metrics', () => {
  let collector;
  let checkout;
  
  beforeAll(() => {
    collector = new MetricsTestCollector();
    checkout = new CheckoutService(collector.getMeter('checkout'));
  });
  
  afterAll(() => collector.shutdown());
  afterEach(() => collector.reset());
  
  it('increments success counter once per order', async () => {
    // Order with 3 items should count as 1 successful checkout
    await checkout.process({ items: ['a', 'b', 'c'] });
    
    await collector.flush();
    
    const metric = collector.getMetricByName('checkout.success.total');
    expect(metric).toBeDefined();
    
    // Should be 1, not 3
    const dataPoint = metric.dataPoints[0];
    expect(dataPoint.value).toBe(1);
  });
  
  it('records correct attributes on success counter', async () => {
    await checkout.process({ items: ['a'], region: 'us-east-1' });
    
    await collector.flush();
    
    const metric = collector.getMetricByName('checkout.success.total');
    const dataPoint = metric.dataPoints[0];
    
    expect(dataPoint.attributes['checkout.region']).toBe('us-east-1');
    expect(dataPoint.attributes['checkout.item_count']).toBeDefined();
  });
  
  it('increments error counter on failure', async () => {
    await expect(
      checkout.process({ items: [] }) // empty cart
    ).rejects.toThrow();
    
    await collector.flush();
    
    const errorMetric = collector.getMetricByName('checkout.error.total');
    expect(errorMetric?.dataPoints[0]?.value).toBeGreaterThan(0);
    
    // Success counter should NOT increment on failure
    const successMetric = collector.getMetricByName('checkout.success.total');
    expect(successMetric?.dataPoints[0]?.value ?? 0).toBe(0);
  });
});

Testing Histogram Bucket Coverage

Histogram bucket configuration is easy to get wrong — and when it's wrong, you can't compute accurate percentiles:

describe('latency histogram configuration', () => {
  it('has appropriate buckets for API latency', () => {
    const histogram = meter.createHistogram('api.latency', {
      description: 'API request latency',
      unit: 'ms',
      // Boundaries should cover your typical latency range with sufficient resolution
      advice: { explicitBucketBoundaries: [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000] }
    });
    
    // Record some synthetic latencies
    const testLatencies = [7, 15, 45, 80, 200, 450, 750, 1200];
    testLatencies.forEach(lat => histogram.record(lat));
    
    await collector.flush();
    
    const metric = collector.getMetricByName('api.latency');
    const { dataPoints } = metric;
    
    // Verify that our P99 case (1200ms) falls in a meaningful bucket
    // Not in "infinity bucket" that tells us nothing
    const buckets = dataPoints[0].value.buckets;
    const bucket1500 = buckets.boundaries.indexOf(1000); // closest upper bound
    
    // The 1200ms value should be captured before the "5000ms" bucket
    // If it's in the last bucket, our resolution is too coarse for SLO tracking
    expect(bucket1500).not.toBe(buckets.boundaries.length - 1);
  });
  
  it('records latency in milliseconds not seconds', async () => {
    // This tests for a common unit confusion bug
    const startTime = Date.now();
    await apiCall();
    const duration = Date.now() - startTime;
    
    await collector.flush();
    
    const metric = collector.getMetricByName('api.latency');
    const recorded = metric.dataPoints[0].value.sum;
    
    // Recorded value should be in milliseconds (close to actual duration)
    // If it's in seconds, it would be ~1000x smaller
    expect(recorded).toBeGreaterThan(duration * 0.5);
    expect(recorded).toBeLessThan(duration * 2);
  });
});

Testing Alert Rules

Alert rules written in PromQL, ClickHouse SQL, or your backend's query language should be tested separately from instrumentation:

// tests/observability/alerts.test.js
import { AlertRule } from '../test-utils/alert-tester';

describe('error rate alert', () => {
  it('fires when error rate exceeds 5% over 5 minutes', async () => {
    const alert = new AlertRule(`
      rate(http_server_errors_total[5m]) 
      / 
      rate(http_server_requests_total[5m]) 
      > 0.05
    `);
    
    // Simulate 1000 requests with 60 errors (6% error rate)
    const series = generateTimeSeries({
      errors: 60,
      total: 1000,
      windowMinutes: 5
    });
    
    const result = await alert.evaluate(series);
    expect(result.firing).toBe(true);
  });
  
  it('does not fire for transient spike within tolerance', async () => {
    const alert = new AlertRule(`
      rate(http_server_errors_total[5m]) 
      / 
      rate(http_server_requests_total[5m]) 
      > 0.05
      FOR 2m  // Must be sustained for 2 minutes
    `);
    
    // 30 second spike of 10% errors, then returns to 1%
    const series = generateTransientSpike({
      spikeRate: 0.10,
      spikeDurationSeconds: 30,
      baselineRate: 0.01,
      windowMinutes: 5
    });
    
    const result = await alert.evaluate(series);
    expect(result.firing).toBe(false); // Transient, not sustained
  });
  
  it('recovers correctly after error rate drops', async () => {
    const alert = new AlertRule(`...`);
    
    // Fire state
    const firingResult = await alert.evaluate(highErrorSeries);
    expect(firingResult.firing).toBe(true);
    
    // Recovery state
    const recoveredResult = await alert.evaluate(normalSeries);
    expect(recoveredResult.firing).toBe(false);
    expect(recoveredResult.resolved).toBe(true);
  });
});

Integration: Testing Metrics End-to-End

Unit testing metrics tells you the instrumentation code is correct. An integration test tells you the full pipeline works:

// tests/integration/metrics-pipeline.test.js
describe('metrics pipeline integration', () => {
  it('metrics emitted by app are queryable in Prometheus', async () => {
    // Hit the endpoint 10 times
    for (let i = 0; i < 10; i++) {
      await request(app).get('/api/products').expect(200);
    }
    
    // Wait for metrics to be scraped
    await wait(15000); // Prometheus scrape interval
    
    // Query Prometheus
    const result = await prometheusQuery(
      'increase(http_server_requests_total{path="/api/products"}[1m])'
    );
    
    // Should show approximately 10 requests
    expect(parseFloat(result.value)).toBeGreaterThan(8);
    expect(parseFloat(result.value)).toBeLessThan(12);
  });
});

Metric Cardinality Governance

High-cardinality metrics are a common production incident. Add cardinality tests:

describe('metric cardinality', () => {
  it('http_server_requests_total does not include user ID as label', async () => {
    // Make requests as different users
    await request(app).get('/api').set('Authorization', 'Bearer user1-token');
    await request(app).get('/api').set('Authorization', 'Bearer user2-token');
    await request(app).get('/api').set('Authorization', 'Bearer user3-token');
    
    await collector.flush();
    
    const metric = collector.getMetricByName('http_server_requests_total');
    const labelKeys = Object.keys(metric.dataPoints[0].attributes);
    
    // user_id must not be a label — it would create unbounded cardinality
    expect(labelKeys).not.toContain('user_id');
    expect(labelKeys).not.toContain('user.id');
  });
  
  it('error metrics use code not full error message', async () => {
    // Different error messages shouldn't create different label values
    await triggerError('Database connection timeout after 30s');
    await triggerError('Database connection timeout after 31s');
    
    await collector.flush();
    
    const metric = collector.getMetricByName('application.errors.total');
    
    // Should use error.type not error.message
    const labelKeys = Object.keys(metric.dataPoints[0].attributes);
    expect(labelKeys).toContain('error.type');
    expect(labelKeys).not.toContain('error.message');
  });
});

Summary

Testing OpenTelemetry metrics and alerting prevents a specific class of production failure: the outage where your monitoring system doesn't alert because the metrics are wrong.

The essential tests:

  1. Counter semantics — verify counters count what they claim (units, per-event vs per-item)
  2. Histogram bucket coverage — verify your latency buckets support your SLO resolution
  3. Attribute correctness — verify required dimensions are present, cardinality dimensions are absent
  4. Alert query accuracy — verify alert rules fire and recover correctly
  5. End-to-end pipeline — verify emitted metrics reach your backend

Write these tests when you write the instrumentation. A metric that's never tested is a metric that can silently mislead you about your production system.

Read more

Start now free