TraceTest: Write Tests Against OpenTelemetry Traces

TraceTest: Write Tests Against OpenTelemetry Traces

When an API call triggers a chain of microservice calls, database queries, and cache lookups, a test that only checks the final HTTP response is testing almost nothing. The response was 200 OK — but did the service actually query the database, or return a stale cache hit? Did it call the payment service before the inventory check, or after? Did a database query run 47 times instead of once?

TraceTest answers these questions by letting you write test assertions against OpenTelemetry distributed traces, not just HTTP responses.

What Trace-Based Testing Means

In traditional API testing, you send a request and assert on the response. In trace-based testing, you send a request and assert on the entire execution trace — the sequence of spans that represents every operation your system performed to handle that request.

A span might represent:

  • An HTTP call to a downstream service
  • A database query
  • A cache read or write
  • A message published to a queue
  • A function call within your application

TraceTest triggers an HTTP request, waits for the distributed trace to complete in your tracing backend (Jaeger, Tempo, or an OpenTelemetry Collector), then lets you write assertions against any attribute of any span in that trace.

Installation and Setup

TraceTest runs as a Docker service alongside your application. The minimal setup requires a docker-compose.yml that connects TraceTest to your tracing backend:

version: "3"
services:
  tracetest:
    image: kubeshop/tracetest:latest
    ports:
      - "11633:11633"
    volumes:
      - ./tracetest-config.yml:/app/config.yml
    depends_on:
      - jaeger

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "4317:4317"

The configuration file tells TraceTest where to find traces:

# tracetest-config.yml
traceStore:
  type: jaeger
  jaeger:
    endpoint: jaeger:16686

Your application must already be instrumented with OpenTelemetry and sending traces to the same backend. TraceTest doesn't add instrumentation — it consumes the traces your application already produces.

Writing a Trace-Based Test

Tests are defined in YAML. A test specifies a trigger (the HTTP request to send), a polling profile (how long to wait for the trace), and a set of assertions:

type: Test
spec:
  id: orders-create-test
  name: Create order - full trace assertions
  trigger:
    type: http
    httpRequest:
      method: POST
      url: http://localhost:8080/api/orders
      headers:
        - key: Content-Type
          value: application/json
      body: '{"item":"widget","qty":3}'
  specs:
    - selector: span[tracetest.span.type="http" name="POST /api/orders"]
      assertions:
        - attr:http.status_code = 201
    - selector: span[tracetest.span.type="database"]
      assertions:
        - attr:db.system = "postgresql"
        - attr:db.statement contains "INSERT INTO orders"
    - selector: span[tracetest.span.type="http" name="POST /inventory/reserve"]
      assertions:
        - attr:http.status_code = 200

This test verifies three things:

  1. The HTTP endpoint returned 201
  2. An INSERT into the orders table happened
  3. The inventory service was called and returned 200

None of this is visible in the HTTP response alone.

Span Selectors

TraceTest's span selector syntax is similar to CSS selectors but applies to trace spans. You can select spans by type, name, attribute values, or their relationship to other spans:

# All database spans
span[tracetest.span.type="database"]

# HTTP spans with a specific method
span[tracetest.span.type="http" http.method="GET"]

# Spans that are children of a specific span
span[tracetest.span.type="database"] follows span[name="processOrder"]

Assertions can check any attribute on the selected spans:

assertions:
  # Exact match
  - attr:http.status_code = 200
  # Contains
  - attr:db.statement contains "WHERE user_id ="
  # Comparison
  - attr:http.response_content_length < 10000
  # Count — number of matching spans
  - count(span[tracetest.span.type="database"]) <= 5

The count() assertion is particularly valuable — it catches N+1 query patterns that look fine in responses but destroy production performance.

Running Tests from the CLI

TraceTest has a CLI for running tests in CI pipelines:

# Install
curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash

# Configure the CLI to point at your TraceTest server
tracetest configure --endpoint http://localhost:11633

# Run a test file
tracetest run test -f ./tests/create-order.yml

# Run all tests in a directory
tracetest run test -f ./tests/

The CLI exits with a non-zero code on failure, making it straightforward to gate CI pipelines:

# .github/workflows/test.yml
- name: Run TraceTest suite
  run: |
    tracetest run test -f ./tracetest/

Test output shows which assertions passed and failed, with the actual span attribute values:

✅ span[tracetest.span.type="http"] http.status_code = 201
✅ span[tracetest.span.type="database"] db.statement contains "INSERT INTO orders"
❌ span[tracetest.span.type="http" name="POST /inventory/reserve"]
   Expected: http.status_code = 200
   Got:      http.status_code = 503

Test Suites and Chained Tests

TraceTest supports test suites that chain multiple tests together, passing outputs between them:

type: TestSuite
spec:
  id: order-lifecycle
  name: Full order lifecycle
  steps:
    - id: create-order
    - id: check-order-status
    - id: cancel-order

You can extract values from one test's trace and use them as inputs to the next test:

# In create-order test
outputs:
  - name: ORDER_ID
    selector: span[name="createOrder"]
    value: attr:app.order.id
# In check-order-status test
trigger:
  httpRequest:
    url: http://localhost:8080/api/orders/${var:ORDER_ID}

This enables end-to-end scenario testing where each step verifies not just that it returned the right response, but that the right distributed operations happened.

When Trace-Based Testing Pays Off

Trace-based testing has setup overhead — your application needs OpenTelemetry instrumentation and a working tracing backend. For simple monolithic services with straightforward database interactions, the ROI is low. For microservice architectures, the ROI is high:

Where it excels:

  • Verifying that service calls happen in the correct order (authorization before data access, etc.)
  • Catching N+1 query regressions introduced by ORM changes
  • Confirming that async events are published with correct attributes
  • Testing cache hit/miss behavior
  • Verifying distributed transaction consistency

Where it doesn't help:

  • Services without OpenTelemetry instrumentation
  • Testing pure business logic (unit tests are better)
  • Checking UI behavior

Complementing TraceTest with Continuous Monitoring

TraceTest runs at test time, against a test environment. For production visibility, you need runtime monitoring. HelpMeTest provides continuous API health checks that run 24/7 against your live endpoints, alerting on failures before users report them. The combination gives you both trace-level correctness verification at deploy time and production monitoring that catches issues that only surface under real load.

If you're already running OpenTelemetry in production and want to start writing tests that actually verify your distributed system's behavior — not just its HTTP facade — TraceTest is the most direct path there.

Read more

Start now free