Testing

Testing OpenTelemetry Sampling Strategies: Head, Tail, and Adaptive Sampling

HelpMeTest

HelpMeTest

— 6 min read
Testing OpenTelemetry Sampling Strategies: Head, Tail, and Adaptive Sampling

Sampling is one of the most consequential decisions in an observability setup — and one of the least tested. When sampling is misconfigured, you lose exactly the traces you most need: errors, slow requests, and anomalies. Or you over-sample and blow your observability budget.

Testing your sampling strategy should be a required part of your observability setup. This guide covers how.

Why Sampling Needs Testing

Sampling configuration bugs are invisible until you need the traces that got dropped:

Under-sampling errors: Your sampler is configured to sample 1% of requests. You have a bug affecting 0.1% of transactions. You're unlikely to capture any traces of that bug until you either change the sampling rate or get very lucky.

Tail sampler misconfiguration: Your tail sampler should keep all traces with errors. A misconfiguration means error traces are being dropped at the collector. Developers file bugs about not being able to find error traces in production.

Sampling flag propagation failures: Your service samples at 10%. A downstream service always samples 100%. The result: all spans from the downstream service are sampled, but they're disconnected from the parent traces because the parent context says "not sampled."

Adaptive sampler drift: Your adaptive sampler is tuned for 10,000 requests/hour. Traffic doubles. The sampler adapts, but now you're missing entire categories of requests because the rate-based cutoffs are wrong.

Types of Sampling in OpenTelemetry

Head sampling: Decision made at the start of a trace, at the root span. Fast, low overhead. Problem: you decide without knowing what will happen in the trace (whether it'll be slow or contain errors).

Types:

  • AlwaysOnSampler: Sample everything (use only in development)
  • AlwaysOffSampler: Sample nothing (use for disabling)
  • TraceIdRatioBased: Sample X% based on trace ID hash (deterministic)
  • ParentBased: Follow the parent's sampling decision; apply your own for root spans

Tail sampling: Decision made after the trace completes. You see the full trace before deciding. Better for capturing errors and slow traces. Requires a sampling processor in the OTEL collector.

Adaptive/dynamic sampling: Adjusts sampling rate based on traffic volume, error rate, or other signals. Complex to configure and test.

Testing Head Sampling

Testing TraceIdRatioBased Sampling

// tests/observability/sampling.test.js
import { TraceIdRatioBased, AlwaysOnSampler } from '@opentelemetry/sdk-trace-base';
import { SamplingDecision } from '@opentelemetry/sdk-trace-base';

describe('TraceIdRatioBased sampler', () => {
  it('samples approximately the configured ratio', () => {
    const ratio = 0.1; // 10%
    const sampler = new TraceIdRatioBased(ratio);
    
    const ITERATIONS = 10000;
    let sampledCount = 0;
    
    for (let i = 0; i < ITERATIONS; i++) {
      // Generate realistic trace IDs
      const traceId = Array.from({ length: 32 }, () => 
        Math.floor(Math.random() * 16).toString(16)
      ).join('');
      
      const result = sampler.shouldSample(
        { getValue: () => undefined }, // context
        traceId,
        'test-span',
        0, // SpanKind.INTERNAL
        {},
        []
      );
      
      if (result.decision === SamplingDecision.RECORD_AND_SAMPLED) {
        sampledCount++;
      }
    }
    
    const actualRatio = sampledCount / ITERATIONS;
    // Allow 20% relative error (1% absolute at 10% ratio)
    expect(actualRatio).toBeGreaterThan(ratio * 0.8);
    expect(actualRatio).toBeLessThan(ratio * 1.2);
  });
  
  it('is deterministic for the same trace ID', () => {
    const sampler = new TraceIdRatioBased(0.5);
    const traceId = 'abc123def456789012345678901234ab';
    
    const result1 = sampler.shouldSample(
      { getValue: () => undefined }, traceId, 'span', 0, {}, []
    );
    const result2 = sampler.shouldSample(
      { getValue: () => undefined }, traceId, 'span', 0, {}, []
    );
    
    // Same trace ID must always get the same decision
    expect(result1.decision).toBe(result2.decision);
  });
});

Testing ParentBased Sampling

ParentBased sampling is critical to get right for distributed tracing:

describe('ParentBased sampler behavior', () => {
  const ROOT_SAMPLER = new TraceIdRatioBased(0.5);
  const sampler = new ParentBased({ root: ROOT_SAMPLER });
  
  it('respects parent sampling=true decision', () => {
    // Simulate a parent that was sampled
    const parentContext = createContextWithSampledParent();
    
    const result = sampler.shouldSample(parentContext, 'trace-id', 'child-span', 0, {}, []);
    
    // Child must be sampled if parent was sampled
    expect(result.decision).toBe(SamplingDecision.RECORD_AND_SAMPLED);
  });
  
  it('respects parent sampling=false decision', () => {
    // Simulate a parent that was NOT sampled
    const parentContext = createContextWithUnsampledParent();
    
    const result = sampler.shouldSample(parentContext, 'trace-id', 'child-span', 0, {}, []);
    
    // Child must NOT be sampled if parent was not sampled
    expect(result.decision).toBe(SamplingDecision.DROP);
  });
  
  it('applies root sampler when there is no parent', () => {
    const noParentContext = context.active(); // no parent span
    
    // With 50% ratio, result will vary — test many times and verify distribution
    let sampledCount = 0;
    for (let i = 0; i < 1000; i++) {
      const traceId = randomTraceId();
      const result = sampler.shouldSample(noParentContext, traceId, 'root-span', 0, {}, []);
      if (result.decision === SamplingDecision.RECORD_AND_SAMPLED) sampledCount++;
    }
    
    expect(sampledCount).toBeGreaterThan(350); // ~50%, allowing variance
    expect(sampledCount).toBeLessThan(650);
  });
});

Testing Tail Sampling in the OTEL Collector

Tail sampling happens in the OTEL collector, not the SDK. Testing it requires spinning up a collector with your configuration and sending test data:

# config/otel-collector-test.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      
      - name: slow-traces-policy
        type: latency
        latency: { threshold_ms: 1000 }
      
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [logging]

Testing this configuration:

// tests/observability/tail-sampling.test.js
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { CollectorTestHarness } from '../test-utils/collector-harness';

describe('tail sampling policies', () => {
  let collector;
  
  beforeAll(async () => {
    collector = new CollectorTestHarness('./config/otel-collector-test.yaml');
    await collector.start();
  });
  
  afterAll(() => collector.stop());
  
  it('keeps error traces regardless of sampling rate', async () => {
    // Send 100 successful traces + 10 error traces
    await sendTraces({ count: 100, hasError: false });
    await sendTraces({ count: 10, hasError: true });
    
    await wait(15000); // Wait for tail sampling decision window
    
    const exportedTraces = collector.getExportedTraces();
    const errorTraces = exportedTraces.filter(t => t.hasError);
    
    // All 10 error traces should be kept
    expect(errorTraces).toHaveLength(10);
  });
  
  it('keeps slow traces above latency threshold', async () => {
    await sendTraces({ count: 50, latencyMs: 500 }); // below threshold
    await sendTraces({ count: 10, latencyMs: 1500 }); // above threshold
    
    await wait(15000);
    
    const slowTraces = collector.getExportedTraces()
      .filter(t => t.duration > 1000);
    
    expect(slowTraces).toHaveLength(10);
  });
  
  it('samples remaining traces at configured percentage', async () => {
    // Send 1000 normal, fast, non-error traces
    await sendTraces({ count: 1000, latencyMs: 100, hasError: false });
    
    await wait(15000);
    
    const exportedCount = collector.getExportedTraces().length;
    
    // 5% of 1000 = 50, allow 20% variance
    expect(exportedCount).toBeGreaterThan(30);
    expect(exportedCount).toBeLessThan(70);
  });
});

Testing Sampling With Integration Tests

The most practical test: verify that your application's actual sampling behavior in a realistic test scenario is correct:

// tests/integration/sampling-behavior.test.js
describe('application sampling behavior', () => {
  it('captures 100% of error requests', async () => {
    const spans = [];
    
    // Intercept all exported spans
    jest.spyOn(exporter, 'export').mockImplementation((s) => {
      spans.push(...s);
    });
    
    // Send 100 requests that trigger errors
    for (let i = 0; i < 100; i++) {
      await request(app).post('/checkout').send({ items: [] }); // empty cart error
    }
    
    await flushSpans();
    
    const errorSpans = spans.filter(s => s.status.code === SpanStatusCode.ERROR);
    
    // With error-based tail sampling, we expect to see error spans
    expect(errorSpans.length).toBeGreaterThan(0);
    expect(errorSpans.every(s => s.name === 'checkout.process')).toBe(true);
  });
});

Common Sampling Misconfiguration Tests

These are the specific misconfiguration patterns worth testing for:

describe('sampling misconfiguration detection', () => {
  it('does not sample error spans when parent is not sampled', async () => {
    // THIS IS THE BUG: If your sampler doesn't properly handle
    // parent=not-sampled + child has error, you might lose error data
    
    const spans = await runWithUnsampledParent(async () => {
      throw new Error('expected error');
    });
    
    // Error spans from unsampled roots should be dropped (unless using error-based policies)
    // This test documents the behavior — change expectation based on your policy
    const errorSpans = spans.filter(s => s.status.code === SpanStatusCode.ERROR);
    
    if (USE_ERROR_BASED_TAIL_SAMPLING) {
      expect(errorSpans).toHaveLength(1); // tail sampler rescues it
    } else {
      expect(errorSpans).toHaveLength(0); // parent decision is respected
    }
  });
  
  it('sampling decision is consistent for all spans in a trace', async () => {
    const allSpans = [];
    
    for (let i = 0; i < 100; i++) {
      const spans = await captureSpans(() => processRequest());
      const traceId = spans[0]?.spanContext().traceId;
      
      // All spans in the same trace should have the same sampling decision
      const sampledFlags = spans.map(s => s.spanContext().traceFlags);
      const unique = new Set(sampledFlags);
      
      expect(unique.size).toBe(1); // all spans have same flag within a trace
    }
  });
});

Building a Sampling Test Report

For teams that need documentation of their sampling behavior (compliance, cost analysis):

// scripts/sampling-analysis.js
async function analyzeSamplingBehavior() {
  const scenarios = [
    { name: 'Normal traffic', generate: () => sendNormalRequests(1000) },
    { name: 'Error traffic', generate: () => sendErrorRequests(100) },
    { name: 'Slow traffic', generate: () => sendSlowRequests(100) },
  ];
  
  const report = {};
  
  for (const scenario of scenarios) {
    const collector = new TestCollector();
    await scenario.generate();
    
    const spans = collector.getSpans();
    report[scenario.name] = {
      generated: scenario.count,
      exported: spans.length,
      samplingRate: `${((spans.length / scenario.count) * 100).toFixed(1)}%`,
      errorsCaptured: spans.filter(s => s.status.code === SpanStatusCode.ERROR).length,
    };
  }
  
  console.table(report);
}

Summary

Testing your OpenTelemetry sampling strategy catches configuration bugs before they affect production observability. The key tests:

  1. Head sampler accuracy: Verify TraceIdRatioBased samples at the configured rate within statistical tolerance
  2. ParentBased correctness: Verify child spans follow parent sampling decisions
  3. Tail sampler policies: Verify error and latency policies capture the right traces
  4. Consistency: All spans in a trace share the same sampling decision
  5. Cardinality: Your sampler doesn't create high-cardinality attributes

Add these to your CI pipeline. When sampling breaks, you want to know before production — not during an outage when the traces you need aren't there.

Read more

Start now free