Logging

Datadog Log Management for Testing: Assertions, Analytics, and Anomaly Detection

HelpMeTest

HelpMeTest

— 8 min read
Datadog Log Management for Testing: Assertions, Analytics, and Anomaly Detection

Most teams treat logs as a post-hoc debugging tool: something breaks, you search logs to find out why. That is a valid use, but it captures only a fraction of what log data can do. Datadog Log Management enables a fundamentally different workflow: log-based assertions in automated tests, anomaly detection that catches problems before users notice, and live tail debugging that eliminates the round-trip between "something is wrong" and "here is the evidence." This guide covers how to wire up each of these for testing and observability.

Log Ingestion Architecture

Before using logs for testing, you need reliable ingestion. Datadog supports several ingestion paths:

Datadog Agent (recommended for most setups) — the Agent tails log files and forwards them with automatic parsing:

# /etc/datadog-agent/conf.d/your-app.d/conf.yaml
logs:
  - type: file
    path: /var/log/your-app/*.log
    service: your-app
    source: nodejs
    tags:
      - env:production

Docker/Kubernetes — the Agent auto-discovers containers and collects stdout/stderr logs when you add labels:

# Kubernetes pod annotation
annotations:
  ad.datadoghq.com/app.logs: '[{"source": "nodejs", "service": "checkout"}]'

Direct API ingestion — for serverless or environments where the Agent is not practical:

curl -X POST "https://http-intake.logs.datadoghq.com/api/v2/logs" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: <your_api_key>" \
  -d '[{
    "ddsource": "nodejs",
    "ddtags": "env:production,version:1.4.0",
    "hostname": "web-01",
    "message": "Order created successfully",
    "service": "checkout",
    "order_id": "ord_123",
    "amount": 4999
  }]'

Log forwarding from your application logger — using Winston (Node.js) with the Datadog transport:

const winston = require('winston');
const { createLogger, format, transports } = winston;

const logger = createLogger({
  format: format.combine(
    format.timestamp(),
    format.json()
  ),
  transports: [
    new transports.Console(),
    new transports.Http({
      host: 'http-intake.logs.datadoghq.com',
      path: `/api/v2/logs?dd-api-key=${process.env.DD_API_KEY}`,
      ssl: true
    })
  ]
});

For Python, the ddtrace library integrates with the standard logging module and automatically injects trace IDs into log records when log injection is enabled.

Structured Logging is Non-Negotiable

Unstructured log lines like "User 12345 placed order for $49.99 at 2024-01-15T10:30:00Z" are readable but not queryable. You cannot efficiently filter by user ID, order amount, or time range without parsing.

Structured logs (JSON) make every field a first-class queryable attribute:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "info",
  "message": "Order placed",
  "service": "checkout",
  "user_id": "usr_12345",
  "order_id": "ord_67890",
  "amount_cents": 4999,
  "payment_method": "card",
  "dd.trace_id": "4821859359786038914",
  "dd.span_id": "7235941211107093236"
}

In Datadog Log Explorer, amount_cents > 10000 is now a valid filter. So is payment_method:card AND @order_id:ord_67890. The dd.trace_id fields link directly to the APM trace — one click from a log line to the full distributed trace.

Invest time defining a structured log schema for your application. Fields like user_id, request_id, order_id, and error.type should be consistent across all services. Inconsistent field naming (some services use userId, others use user_id, others use uid) makes cross-service queries painful.

Log-Based Assertions in Tests

The most direct use of logs in testing is asserting that expected log events were emitted during a test run. This is particularly valuable for:

  • Background jobs — no HTTP response to assert on; the only observable output is what the job logs
  • Event-driven systems — verifying that a published event was consumed and processed
  • Audit trails — confirming that security-relevant actions were logged correctly
  • Third-party integrations — verifying that the right payload was sent to an external service

Approach 1: Log Capture in Unit Tests

Capture logs in-process for unit and integration tests:

// Node.js with Winston
const { Writable } = require('stream');

function createLogCapture() {
  const logs = [];
  const stream = new Writable({
    write(chunk, encoding, callback) {
      logs.push(JSON.parse(chunk.toString()));
      callback();
    }
  });
  return { logs, stream };
}

test('order creation emits audit log', async () => {
  const { logs, stream } = createLogCapture();
  logger.add(new transports.Stream({ stream }));

  await createOrder({ userId: 'usr_123', amount: 4999 });

  const auditLog = logs.find(l => l.message === 'Order placed');
  expect(auditLog).toBeDefined();
  expect(auditLog.user_id).toBe('usr_123');
  expect(auditLog.amount_cents).toBe(4999);
});

Approach 2: Datadog Logs API in End-to-End Tests

For end-to-end tests running against a real environment, query Datadog's Logs API to assert on emitted logs:

const axios = require('axios');

async function waitForLog({ query, timeFrom, timeTo, maxWaitMs = 10000 }) {
  const deadline = Date.now() + maxWaitMs;
  while (Date.now() < deadline) {
    const response = await axios.post(
      'https://api.datadoghq.com/api/v2/logs/events/search',
      {
        filter: {
          query,
          from: timeFrom,
          to: timeTo,
        },
        page: { limit: 1 }
      },
      {
        headers: {
          'DD-API-KEY': process.env.DD_API_KEY,
          'DD-APPLICATION-KEY': process.env.DD_APP_KEY,
        }
      }
    );
    if (response.data.data.length > 0) {
      return response.data.data[0];
    }
    await new Promise(r => setTimeout(r, 1000));
  }
  throw new Error(`Log not found within ${maxWaitMs}ms: ${query}`);
}

test('payment processing logs successful charge', async () => {
  const testOrderId = `test-${Date.now()}`;
  const testStart = new Date().toISOString();

  await createOrder({ orderId: testOrderId, amount: 4999 });

  const log = await waitForLog({
    query: `service:checkout @order_id:${testOrderId} @message:"Payment processed"`,
    timeFrom: testStart,
    timeTo: new Date(Date.now() + 30000).toISOString(),
    maxWaitMs: 15000
  });

  expect(log.attributes['@charge_id']).toMatch(/^ch_/);
  expect(log.attributes['@amount_cents']).toBe(4999);
});

This pattern is particularly useful when testing asynchronous workflows: trigger an action, then poll the logs API until the expected log appears (or the timeout expires).

Log Parsing and Pipelines

Raw log lines need to become structured attributes. Datadog's log processing pipelines handle this transformation.

Navigate to Logs → Configuration → Pipelines and create a pipeline for your service:

Grok Parser — extract structured fields from unstructured log lines using patterns:

rule_name %{date("yyyy-MM-dd HH:mm:ss"):timestamp} \[%{word:level}\] %{data:message}

JSON Parser — automatically extract all keys from a JSON log line as attributes (no configuration needed if your logs are already JSON).

Arithmetic Processor — compute derived metrics from log fields:

@response_time_ms / 1000 → @response_time_seconds

Lookup Processor — enrich logs with data from a reference table (e.g., map error_code to error_description).

Remapper — normalize field names across services:

@userId → @user_id
@orderId → @order_id

Pipeline processing happens at ingestion time. The original raw log is preserved; the processed attributes are indexed alongside it. This means you can fix a broken parser and reprocess historical logs without losing data.

Log-Based Monitors and Anomaly Detection

Threshold Monitor

Alert when a specific log pattern exceeds a rate:

  1. Navigate to Monitors → New Monitor → Logs
  2. Search query: service:checkout status:error
  3. Alert threshold: more than 10 occurrences in 5 minutes
  4. Evaluation window: rolling 5 minutes

This creates a basic error rate alert. Make it more specific to reduce false positives:

service:checkout status:error @error.type:PaymentDeclinedException

Alerting only on PaymentDeclinedException (not all errors) means you page on a specific, actionable failure rather than every transient network hiccup.

Anomaly Detection on Log Volume

Static thresholds require guessing what "too many errors" means. Anomaly detection learns the pattern:

  1. Search query: service:checkout status:error
  2. Alert condition: Anomaly — alert when count deviates from baseline by more than 2 standard deviations
  3. Algorithm: agile (adapts quickly to trend changes)

This approach catches sudden spikes (deploy broke something) and slow creep (gradual degradation over hours) without requiring you to guess what the normal rate is.

New Error Detection

This is one of the most valuable log monitors: alert when an error type appears that has never been seen before:

  1. Search query: service:checkout status:error
  2. Group by: @error.type
  3. Alert condition: new value detected

When a previously-unseen error type first appears in your logs, you get an immediate alert. This catches entire error categories that threshold-based monitors would miss until the volume grew large enough.

Live Tail for Real-Time Debugging

During a test run or a production incident, Live Tail streams logs in real time without indexing them. Navigate to Logs → Live Tail and filter:

service:checkout env:staging

You see every log line from your checkout service in staging as they are emitted. This is invaluable during:

  • Integration test debugging — watching the exact sequence of log events during a failing test
  • Deployment monitoring — the 5 minutes after a deploy where you watch for error spikes
  • Performance investigation — seeing slow queries and timeouts as they happen rather than after the fact

Live Tail has no retention — it is a real-time stream only. But for debugging, this is exactly what you want: no query lag, no indexing delay, immediate feedback.

Combine Live Tail with grep-style filtering using the search syntax:

service:checkout @user_id:usr_12345 status:error

This shows only errors for a specific user — useful when a customer reports a problem and you want to trace exactly what happened to their session.

Log Analytics for Test Coverage Gaps

An indirect but powerful use of logs: identify user behaviors in production that your test suite does not cover.

Run a query in Logs → Analytics that shows the most common actions in your application:

service:checkout
group by: @action
aggregate: count
sort: count desc

If action:apply_discount_code appears 50,000 times per day and you have no tests covering discount code application, that is a coverage gap with real business impact.

Similarly, look at the most common error types:

service:checkout status:error
group by: @error.type
aggregate: count

The top error types should correspond to test cases in your suite. If PaymentMethodNotSupported appears 200 times per day but there is no test for it, users are hitting it regularly without it being caught in CI.

Metric Generation from Logs

Datadog can generate custom metrics from log attributes — turning log data into time-series metrics without code changes.

Navigate to Logs → Generate Metrics and create a metric from your payment logs:

  • Filter: service:checkout @message:"Payment processed"
  • Metric type: Count → checkout.payment.success.count
  • Group by: @payment_method, @currency

This generates a metric you can use in dashboards and monitors. The advantage over application-emitted metrics: if you need a new metric, you can create it retroactively from existing logs rather than waiting for a code deployment.

For example, create checkout.order.amount_cents as a distribution metric from @amount_cents in your order logs. You can then compute revenue per hour, p95 order value, and average order size — all from logs rather than a separate analytics pipeline.

Indexes, Retention, and Cost Control

Logs can be expensive at scale. Datadog's indexing system lets you control costs by routing different log types to different retention tiers:

High-value logs (production errors, security events, audit logs) → long retention (15–30 days), fully indexed.

Medium-value logs (info-level application events) → medium retention (7 days), partial indexing.

Noise (health checks, static asset requests) → excluded from indexing entirely using exclusion filters.

Set up exclusion filters under Logs → Configuration → Indexes:

# Exclude health check noise
service:loadbalancer @http.url_details.path:/health

# Exclude debug logs in production
service:* env:production status:debug

Exclusion filters apply before indexing, so excluded logs do not count toward your bill but are still available in Live Tail for real-time debugging.

Archive to S3 — even excluded logs can be archived to S3 at low cost for compliance purposes:

# Logs → Archives → New Archive
destination: AWS S3
bucket: my-log-archive-bucket
path: datadog-logs/{year}/{month}/{day}/{hour}/

Archived logs can be rehydrated back into Datadog for a time-limited investigation window — you pay for the rehydration compute but avoid paying for permanent indexing of everything.

Correlating Logs with Tests in Practice

The most powerful workflow: combine log-based assertions with end-to-end tests.

When a test creates an order with a unique test_order_id, every log line associated with that order (checkout, payment, fulfillment services) can be retrieved from the Logs API and asserted on. This gives you distributed system observability inside your test assertions — not just "the API returned 200" but "the payment service logged a successful charge, the fulfillment service logged a shipment queued, and no error logs appeared for this order."

Tools like HelpMeTest that run end-to-end functional tests pair naturally with Datadog log monitoring: when a functional test fails, the first debugging step is checking what the application logged during that test's execution — and with trace ID injection enabled, every log line from that test run is directly linkable to the failing APM trace.

Summary

Datadog log management for testing enables:

  1. Structured ingestion — JSON logs with consistent field naming across all services
  2. Log-based assertions — polling the Logs API in end-to-end tests to verify expected events were emitted
  3. Live Tail debugging — real-time log streaming during test runs and deployments
  4. Anomaly detection — catch error rate spikes and new error types automatically
  5. Metric generation — derive business metrics from log data retroactively
  6. Log analytics — identify coverage gaps by finding high-frequency production behaviors with no test coverage

Start with structured logging and trace ID injection. The rest of the capabilities build on having good log data — garbage in, garbage out applies here as much as anywhere.

Read more

Start now free