CI/CD

Datadog CI/CD Testing Integration: Test Visibility, Flaky Test Detection, and Pipeline Analytics

HelpMeTest

HelpMeTest

— 7 min read
Datadog CI/CD Testing Integration: Test Visibility, Flaky Test Detection, and Pipeline Analytics

Slow CI pipelines, flaky tests, and unknown test coverage gaps are invisible taxes on engineering velocity. You feel them as "the build is red again" conversations and 40-minute pipeline waits, but without data you cannot quantify them or prioritize fixing them. Datadog CI Visibility gives you that data: test run history, failure trends, flaky test identification, and pipeline performance analytics — all queryable and alertable like any other observability metric.

What CI Visibility Actually Measures

Datadog CI Visibility operates at two levels:

Test visibility — individual test results: which tests passed, which failed, how long each took, what the failure message was, and whether a test has a history of intermittent failures. Data flows from your test runner (Jest, pytest, JUnit, RSpec, etc.) to Datadog via a test reporter library.

Pipeline visibility — entire pipeline runs: stage durations, queue wait times, failure rates by pipeline, and comparisons between branches. Data flows from your CI platform (GitHub Actions, GitLab CI, Jenkins, CircleCI, etc.) via a Datadog integration or the datadog-ci CLI.

Both are queryable in the CI section of Datadog alongside your APM, logs, and metrics — so you can correlate "tests started failing after this deploy" without switching tools.

Instrumenting Tests

Jest (JavaScript/TypeScript)

npm install --save-dev jest-circus dd-trace

Configure Jest to use the Datadog reporter in jest.config.js:

module.exports = {
  testEnvironment: 'node',
  reporters: [
    'default',
    ['dd-trace/jest/reporter', {
      service: 'my-app-tests',
    }]
  ]
};

Set environment variables before running tests:

export DD_API_KEY=<your_api_key>
export DD_CIVISIBILITY_AGENTLESS_ENABLED=true
export DD_ENV=ci
export DD_SERVICE=my-app-tests

npx jest

In agentless mode, results are sent directly to the Datadog intake API — no agent required. This is the typical setup for ephemeral CI runners.

pytest (Python)

pip install pytest-datadog-ci
DD_API_KEY=<your_api_key> \
DD_CIVISIBILITY_AGENTLESS_ENABLED=true \
DD_SERVICE=my-api-tests \
DD_ENV=ci \
pytest --ddtrace

The --ddtrace flag enables CI Visibility. Every test generates a span with its duration, outcome, error message, and the test file/line number.

JUnit XML (Language-Agnostic)

If your test runner produces JUnit XML, you can upload the results directly regardless of language:

datadog-ci junit upload \
  --service my-app \
  --env ci \
  ./test-results/*.xml

This covers frameworks like Go's gotestsum, Maven/Gradle in Java, and any other runner that produces the standard JUnit XML format. The upload approach adds CI Visibility to existing pipelines with minimal changes.

RSpec (Ruby)

gem install ddtrace
DD_API_KEY=<your_api_key> \
DD_CIVISIBILITY_AGENTLESS_ENABLED=true \
DD_SERVICE=my-app-tests \
bundle exec rspec --format RspecJunitFormatter --out rspec.xml
datadog-ci junit upload --service my-app ./rspec.xml

GitHub Actions Integration

Full Pipeline Visibility

Add the Datadog CI integration to your GitHub Actions workflow:

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        env:
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_CIVISIBILITY_AGENTLESS_ENABLED: 'true'
          DD_SERVICE: 'my-app-tests'
          DD_ENV: 'ci'
          DD_GIT_REPOSITORY_URL: ${{ github.server_url }}/${{ github.repository }}
          DD_GIT_COMMIT_SHA: ${{ github.sha }}
          DD_GIT_BRANCH: ${{ github.ref_name }}
          DD_GIT_COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
          DD_GIT_COMMIT_AUTHOR_NAME: ${{ github.event.head_commit.author.name }}
          DD_GIT_COMMIT_AUTHOR_EMAIL: ${{ github.event.head_commit.author.email }}
        run: npx jest

The DD_GIT_* variables attach the test results to the specific commit and branch. This is what enables the commit-level view in Datadog: for any commit, see which tests ran, which failed, and how performance compared to the previous commit on the same branch.

Synthetic Tests in CI

To block a deployment on synthetic test failure, add a step after your unit tests:

      - name: Run Datadog Synthetics
        run: |
          npx @datadog/datadog-ci synthetics run-tests \
            --config ./.datadog/synthetics.json \
            --tunnel
        env:
          DATADOG_API_KEY: ${{ secrets.DD_API_KEY }}
          DATADOG_APP_KEY: ${{ secrets.DD_APP_KEY }}

.datadog/synthetics.json:

{
  "testSearchQuery": "tag:env:staging",
  "global": {
    "startUrl": "https://staging.example.com"
  },
  "failOnCriticalErrors": true,
  "failOnMissingTests": true,
  "defaultTestOverrides": {
    "allowInsecureCertificates": true
  }
}

testSearchQuery selects tests by tag, so you can run a targeted subset for staging vs production. The --tunnel flag creates an encrypted tunnel from Datadog's test infrastructure to your ephemeral staging environment without requiring a public URL or a long-lived private location.

Jenkins Integration

Declarative Pipeline

pipeline {
    agent any

    environment {
        DD_API_KEY = credentials('datadog-api-key')
        DD_CIVISIBILITY_AGENTLESS_ENABLED = 'true'
        DD_SERVICE = 'my-app-tests'
        DD_ENV = 'ci'
    }

    stages {
        stage('Test') {
            steps {
                sh 'npm ci'
                sh 'npx jest --reporters=default --reporters=jest-junit'
            }
            post {
                always {
                    sh '''
                        npx @datadog/datadog-ci junit upload \
                          --service ${DD_SERVICE} \
                          --env ${DD_ENV} \
                          ./junit.xml
                    '''
                }
            }
        }
    }
}

The post { always { ... } } block ensures results are uploaded even when tests fail — which is when you most need the data.

Jenkins Plugin

The official Datadog Jenkins plugin (datadog-plugin) provides pipeline visibility at the stage level without manual instrumentation. Install it via the Jenkins Plugin Manager, configure your API key, and pipeline data automatically flows to Datadog.

With the plugin, each stage in your Declarative or Scripted pipeline appears in Datadog's pipeline view with its duration, status, and failure reason. You can then build dashboards showing average stage durations over time, identifying which stages have been getting slower across commits.

Flaky Test Detection

A flaky test is one that sometimes passes and sometimes fails without any code changes. Flaky tests destroy trust in your CI pipeline — when the build is red, developers spend time investigating whether it is a real failure or a flake. Eventually they start ignoring red builds entirely.

Datadog detects flaky tests automatically. The detection algorithm:

  1. Collects the last N runs of each test across all branches and pipelines
  2. Identifies tests that switched from pass to fail (or vice versa) in consecutive runs without a commit between them
  3. Tags those tests with is_flaky:true in the CI Visibility explorer

In the Tests → Flaky Tests view, you see:

  • Which tests are currently flaky
  • Flakiness rate (what percentage of runs fail)
  • First detected date (how long it has been flaky)
  • The branches and environments where flakiness is observed

This gives you an actionable prioritized list. A test that is flaky 40% of the time and runs on every PR is costing far more engineering time than one that is flaky 5% of the time and only runs nightly.

Early Flake Detection (EFD)

EFD takes a more proactive approach: when a new test is introduced, Datadog runs it multiple times in the same CI job to check for flakiness before it ships:

DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=true npx jest

If the new test fails in any of the repeated runs, CI marks it as potentially flaky and surfaces this in the PR. You catch flakes at introduction rather than after they have been eroding pipeline trust for weeks.

Test Impact Analysis

Running your full test suite on every commit is a sound practice — until it takes 20 minutes. Test impact analysis selects only the tests that are relevant to the changed code:

DD_CIVISIBILITY_ITR_ENABLED=true npx jest

Datadog tracks which test files exercise which source files (using code coverage data collected during previous runs). When a commit changes src/checkout/payment.js, only tests that previously covered that file are selected for execution. Tests covering unrelated code are skipped.

The time savings vary by codebase structure, but teams typically see 30–70% reduction in test execution time for incremental changes. Full test runs still execute on main branch merges or on a schedule to catch indirect dependencies.

Pipeline Analytics and Dashboards

With pipeline visibility enabled, you have metric data for every pipeline run. Build dashboards around:

Pipeline reliability:

sum:ci.pipeline.run{pipeline_name:deploy-production,status:failed} / 
sum:ci.pipeline.run{pipeline_name:deploy-production} * 100

This gives you the failure rate of your production deploy pipeline over time. Alerting when this exceeds 15% catches systemic issues (broken tests, infrastructure problems) before they become cultural problems (developers disabling tests).

Pipeline duration trends:

avg:ci.stage.duration{pipeline_name:ci,stage_name:test} by {branch}.rollup(avg, 3600)

Plotting average test stage duration by week reveals whether your test suite is getting slower over time. A 10% slowdown per month compounds into a 2× slowdown in 7 months — by which point developers are actively skipping tests to save time.

Queue wait times:

avg:ci.pipeline.queue_time{pipeline_name:ci}

High queue wait times mean you are resource-constrained on runners. This metric makes the case for adding more CI capacity — with numbers, not anecdotes.

Test Coverage Integration

Datadog does not replace dedicated coverage tools (Istanbul, coverage.py, JaCoCo), but it surfaces coverage data from your existing tools:

# For JavaScript with Istanbul/nyc
nyc npx jest
datadog-ci coverage upload --service my-app ./coverage/lcov.info

Coverage data in Datadog is then queryable alongside test results. You can see coverage by test file, track coverage trends over commits, and configure alerts when overall coverage drops below a threshold.

Real-World Configuration: Monorepo

For a monorepo with multiple services, maintain per-service test configuration:

# .github/workflows/test.yml
jobs:
  test-api:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx jest packages/api
        env:
          DD_SERVICE: 'api-tests'
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_CIVISIBILITY_AGENTLESS_ENABLED: 'true'
          DD_ENV: ci

  test-frontend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx jest packages/frontend
        env:
          DD_SERVICE: 'frontend-tests'
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_CIVISIBILITY_AGENTLESS_ENABLED: 'true'
          DD_ENV: ci

Using distinct DD_SERVICE values per package lets you filter tests and flakiness reports by service in Datadog, rather than having all tests from a 40-package monorepo in one undifferentiated view.

Connecting CI Testing with Production Observability

The most powerful outcome of Datadog CI Visibility is the ability to correlate CI metrics with production behavior. When a slow test was introduced, you can check whether APM showed a corresponding slowdown in production after that commit shipped. When tests start failing on the release branch, you can check whether the same service shows elevated error rates in staging.

This context shortens the debugging loop: instead of "tests are failing, let me figure out why from scratch," you have "tests started failing at commit X, APM shows latency increased at deploy Y for the same service, here is the trace."

For teams building end-to-end test coverage, tools like HelpMeTest complement Datadog CI Visibility by providing always-on functional tests that run continuously in production — catching regressions that unit and integration tests in CI miss because they never hit real production data.

Summary

Datadog CI Visibility gives engineering teams:

  1. Test history and trends — which tests fail most, how often, for how long
  2. Flaky test detection — automatic identification of unreliable tests before they erode pipeline trust
  3. Pipeline analytics — duration, queue time, and reliability metrics for every pipeline and stage
  4. Test impact analysis — run only relevant tests to cut cycle times by 30–70%
  5. GitHub Actions / Jenkins / GitLab integration — first-class support for the most common CI platforms
  6. Synthetic test blocking — prevent deploys when end-to-end tests fail

The prerequisite for all of this is instrumenting your test runners. Start there: add the Datadog reporter to your Jest or pytest configuration, push API keys as CI secrets, and run a build. Within a single sprint you will have enough data to identify your top 3 flaky tests and your slowest pipeline stage — two concrete improvements that compound over time.

Read more

Start now free