Testing

Grafana Tempo for Testing: Query Traces Programmatically in CI

HelpMeTest

16 May 2026 — 4 min read

Grafana Tempo is a cost-efficient, horizontally scalable trace backend built for the Grafana observability stack. It works well as a test-environment trace store because it's simpler to configure than Jaeger with external storage, integrates naturally with Prometheus and Loki, and supports TraceQL — a purpose-built query language for traces.

Why Tempo in a Test Environment

If you're already using Grafana for metrics (Prometheus) and logs (Loki), Tempo completes the three-pillar observability stack. Your test assertions can then use a consistent query style across all three:

Metrics → PromQL
Logs → LogQL
Traces → TraceQL

One Grafana datasource setup, three assertion surfaces.

Running Tempo Locally

# docker-compose.observability.yml
version: '3.8'
services:
  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo-local.yaml:/etc/tempo.yaml
      - tempo-data:/tmp/tempo
    ports:
      - "3200:3200"   # Tempo HTTP API
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml

volumes:
  tempo-data:

# tempo-local.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal

compactor:
  compaction:
    block_retention: 1h  # Short retention for tests — keep storage small

# grafana-datasources.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
      serviceMap:
        datasourceUid: prometheus

Sending Traces to Tempo

// tracing.js
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
});

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces \
OTEL_SERVICE_NAME=order-service \
node -r ./tracing.js server.js

Querying Tempo in Tests

Tempo exposes two query interfaces:

/api/search — search by service name, span name, tags, and time range
/api/traces/{traceId} — retrieve a specific trace

Search API

// test-utils/tempo.js
const TEMPO_API = 'http://localhost:3200';

async function searchTraces({ service, spanName, afterMs, limit = 10 }) {
  const params = new URLSearchParams({
    minDuration: '0ms',
    limit: limit.toString(),
    start: Math.floor(afterMs / 1000).toString(),
    end: Math.floor(Date.now() / 1000).toString(),
  });
  
  if (service) params.set('tags', `service.name=${service}`);
  if (spanName) params.set('name', spanName);
  
  const res = await fetch(`${TEMPO_API}/api/search?${params}`);
  const json = await res.json();
  return json.traces || [];
}

async function getTrace(traceId) {
  const res = await fetch(`${TEMPO_API}/api/traces/${traceId}`);
  return res.json();
}

module.exports = { searchTraces, getTrace };

Using TraceQL

Tempo's TraceQL is a structured query language — more expressive than tag filters:

async function searchWithTraceQL(query, afterMs) {
  const params = new URLSearchParams({
    q: query,
    start: Math.floor(afterMs / 1000).toString(),
    end: Math.floor(Date.now() / 1000).toString(),
    limit: '10',
  });
  
  const res = await fetch(`${TEMPO_API}/api/search?${params}`);
  const json = await res.json();
  return json.traces || [];
}

// Example TraceQL queries:
// Find traces with errors
await searchWithTraceQL('{ status = error }', startMs);

// Find slow traces
await searchWithTraceQL('{ duration > 500ms }', startMs);

// Find traces from a specific service that hit the orders endpoint
await searchWithTraceQL('{ .service.name = "order-service" && name = "POST /orders" }', startMs);

// Find traces with a specific attribute
await searchWithTraceQL('{ .order.item_count > 5 }', startMs);

Integration Test Examples

const { test, expect } = require('@playwright/test');
const { searchTraces, getTrace, searchWithTraceQL } = require('./test-utils/tempo');

test('order service — happy path trace validation', async ({ page }) => {
  const startMs = Date.now();

  // Act
  const res = await fetch('http://localhost:3001/orders', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ userId: 'u1', sku: 'WIDGET-A', qty: 2 }),
  });
  expect(res.status).toBe(201);
  
  // Wait for async export
  await new Promise(r => setTimeout(r, 1000));

  // Assert — find trace for this request
  const traces = await searchWithTraceQL(
    '{ .service.name = "order-service" && name = "POST /orders" && status != error }',
    startMs
  );
  
  expect(traces.length).toBeGreaterThan(0);
  
  // Get full trace
  const fullTrace = await getTrace(traces[0].traceID);
  const spans = fullTrace.batches.flatMap(b => 
    b.scopeSpans.flatMap(ss => ss.spans)
  );
  
  const serviceNames = fullTrace.batches.map(b => 
    b.resource.attributes.find(a => a.key === 'service.name')?.value?.stringValue
  ).filter(Boolean);
  
  expect(serviceNames).toContain('inventory-service');
  expect(serviceNames).toContain('payment-service');
});

test('no error traces for valid orders (TraceQL)', async () => {
  const startMs = Date.now();
  
  await fetch('http://localhost:3001/orders', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ userId: 'u1', sku: 'WIDGET-A', qty: 1 }),
  });
  
  await new Promise(r => setTimeout(r, 1000));
  
  const errorTraces = await searchWithTraceQL(
    '{ .service.name = "order-service" && status = error }',
    startMs
  );
  
  expect(errorTraces).toHaveLength(0);
});

test('performance: order traces complete within 500ms (TraceQL)', async () => {
  const startMs = Date.now();
  
  await fetch('http://localhost:3001/orders', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ userId: 'u1', sku: 'WIDGET-A', qty: 1 }),
  });
  
  await new Promise(r => setTimeout(r, 1000));
  
  const slowTraces = await searchWithTraceQL(
    '{ .service.name = "order-service" && name = "POST /orders" && duration > 500ms }',
    startMs
  );
  
  expect(slowTraces).toHaveLength(0); // No slow traces — SLO met
});

Tempo vs Jaeger for Test Environments

Aspect	Tempo	Jaeger
Query language	TraceQL (rich)	URL params (basic)
Grafana integration	Native	Via datasource
Default storage	Local filesystem	In-memory
Docker simplicity	Slightly more config	Single env var
OTLP support	Native	Native (with flag)
Best when	Already using Grafana stack	Standalone trace exploration

Resetting Tempo Between Test Runs

Since Tempo writes to local filesystem (/tmp/tempo), you can reset it by restarting the container:

docker restart tempo
sleep 3  <span class="hljs-comment"># wait for Tempo to be ready

Or use a unique block_retention short enough that old data expires automatically (e.g., 5m in local dev).

CI Pipeline

# .github/workflows/integration.yml
services:
  tempo:
    image: grafana/tempo:latest
    options: >-
      -v ${{ github.workspace }}/tempo-ci.yaml:/etc/tempo.yaml
    ports:
      - 3200:3200
      - 4318:4318

# tempo-ci.yaml
server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        http:
storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal
compactor:
  compaction:
    block_retention: 10m

Summary

Grafana Tempo is worth considering for test environments when:

Your observability stack already uses Prometheus + Loki + Grafana
You want TraceQL for expressive trace queries in assertions
You need a cost-efficient backend that doesn't require external storage (Cassandra/Elasticsearch)

The setup is slightly more involved than Jaeger's single-container mode, but the payoff is TraceQL — which makes trace assertions more readable and precise than URL parameter filtering.

Both Tempo and Jaeger are solid choices. Pick based on the rest of your observability stack.