Testim CI Integration: Running AI Tests in Your Pipeline
Testim tests recorded in the dashboard are only useful if they run automatically on every code change. Getting Testim wired into CI requires the CLI runner, an API token, and some pipeline configuration. This post covers the complete setup for GitHub Actions and Jenkins, including parallel execution, branch-specific configurations, and failure notifications.
Installing the Testim CLI
The Testim CLI is distributed as an npm package:
npm install -g @testim/testim-cliFor CI environments where you want a pinned version and don't want to install globally:
npm install --save-dev @testim/testim-cliThen reference it via npx testim or add it to your package.json scripts:
{
"scripts": {
"test:e2e": "testim --token $TESTIM_TOKEN --project $TESTIM_PROJECT_ID"
}
}Verify the installation works locally before wiring it into CI:
testim --versionAPI Token Setup
In the Testim dashboard, navigate to Settings > CLI/API Tokens. Create a new token scoped to your project. Token permissions:
- Run only: Can trigger test runs and read results. Cannot create or delete tests.
- Admin: Full access. Avoid using admin tokens in CI — if your pipeline configuration leaks, run-only tokens limit the blast radius.
Store the token as an environment secret in your CI system, not in your repository. The variable name used in Testim's own documentation is TESTIM_TOKEN.
You also need your project ID, visible in the dashboard URL when you're inside a project: https://app.testim.io/#/project/YOUR_PROJECT_ID/...
Running Tests via CLI
Basic invocation to run all tests in a project:
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--base-url "https://staging.yourapp.com"Run tests with a specific label (the primary way to scope test runs):
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--label "smoke" \
--base-url "https://staging.yourapp.com"Run a specific test by name:
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--name "User can complete checkout" \
--base-url "https://staging.yourapp.com"Run a test suite (a saved collection of tests):
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--suite "regression" \
--base-url "https://staging.yourapp.com"The CLI exits with code 0 if all tests pass and non-zero if any test fails. CI systems interpret exit codes automatically, so a failing test will fail the pipeline step.
Test Result Output Formats
By default, the CLI outputs a summary to stdout. For CI integration you usually want machine-readable output:
JUnit XML (works with most CI systems, Slack integrations, test analytics dashboards):
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--junit-report-filename "testim-results.xml"JSON report:
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--report-file "testim-results.json"For GitHub Actions, the JUnit XML output can be consumed by test reporter actions to display pass/fail results inline on pull requests.
GitHub Actions Workflow
A complete workflow file that runs Testim tests on pull requests and pushes to main:
name: E2E Tests
on:
push:
branches: [main, develop]
pull_request:
branches: [main, develop]
jobs:
e2e:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install Testim CLI
run: npm install -g @testim/testim-cli
- name: Run Testim smoke tests
env:
TESTIM_TOKEN: ${{ secrets.TESTIM_TOKEN }}
TESTIM_PROJECT_ID: ${{ secrets.TESTIM_PROJECT_ID }}
run: |
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--label "smoke" \
--base-url "${{ vars.STAGING_URL }}" \
--junit-report-filename "testim-results.xml" \
--retries 1
- name: Publish test results
uses: mikepenz/action-junit-report@v4
if: always()
with:
report_paths: 'testim-results.xml'
check_name: 'Testim E2E Results'
- name: Upload test artifacts on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: testim-results
path: testim-results.xml
retention-days: 7The --retries 1 flag tells Testim to retry each failing test once before marking it as failed. This filters out intermittent network-related failures without hiding real bugs.
Branch-Specific Test Configurations
Different branches often need different test configurations — feature branches might run only smoke tests while main runs the full regression suite:
- name: Determine test label
id: test-config
run: |
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
echo "label=regression" >> $GITHUB_OUTPUT
echo "base_url=https://staging.yourapp.com" >> $GITHUB_OUTPUT
else
echo "label=smoke" >> $GITHUB_OUTPUT
echo "base_url=https://preview-${{ github.event.pull_request.number }}.yourapp.com" >> $GITHUB_OUTPUT
fi
- name: Run Testim tests
env:
TESTIM_TOKEN: ${{ secrets.TESTIM_TOKEN }}
TESTIM_PROJECT_ID: ${{ secrets.TESTIM_PROJECT_ID }}
run: |
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--label "${{ steps.test-config.outputs.label }}" \
--base-url "${{ steps.test-config.outputs.base_url }}"This pattern assumes you have preview deployments per PR. If not, always point at a shared staging environment and just vary the label.
Jenkins Pipeline Setup
A Jenkinsfile using declarative syntax:
pipeline {
agent { label 'linux' }
environment {
TESTIM_TOKEN = credentials('testim-token')
TESTIM_PROJECT_ID = credentials('testim-project-id')
}
options {
timeout(time: 30, unit: 'MINUTES')
}
stages {
stage('Install') {
steps {
sh 'npm install -g @testim/testim-cli'
}
}
stage('E2E Tests') {
steps {
sh """
testim \\
--token "${TESTIM_TOKEN}" \\
--project "${TESTIM_PROJECT_ID}" \\
--label "smoke" \\
--base-url "${STAGING_URL}" \\
--junit-report-filename "testim-results.xml" \\
--retries 1
"""
}
post {
always {
junit 'testim-results.xml'
archiveArtifacts artifacts: 'testim-results.xml', allowEmptyArchive: true
}
}
}
}
post {
failure {
emailext(
subject: "E2E Tests Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
body: "Build URL: ${env.BUILD_URL}",
to: 'qa-team@yourcompany.com'
)
}
}
}Jenkins' built-in JUnit publisher picks up the XML report automatically and tracks test trend history across builds.
Parallel Execution
Testim's cloud execution engine runs tests in parallel by default when you have enough tests and the right plan. The CLI --parallel flag controls the degree of parallelism:
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--label "regression" \
--parallel 5 \
--base-url "https://staging.yourapp.com"This runs up to 5 tests concurrently. The practical limit depends on your Testim plan's concurrent session allowance.
For GitHub Actions matrix-based parallelism (splitting tests across multiple runners):
strategy:
matrix:
shard: [1, 2, 3, 4]
steps:
- name: Run Testim shard
run: |
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--label "regression" \
--shard "${{ matrix.shard }}/4" \
--base-url "https://staging.yourapp.com"The --shard N/TOTAL flag divides the test suite into equal partitions and runs only the partition for the current runner. This is useful when you want to keep wall-clock time down without paying for more Testim concurrent sessions.
Failure Notification Setup
Beyond pipeline-level email notifications, Testim integrates directly with Slack and other channels for per-test failure notifications:
Slack via webhook (in the CLI):
testim \
--token "$TESTIM_TOKEN" \
--project "$TESTIM_PROJECT_ID" \
--slack-webhook "$SLACK_WEBHOOK_URL" \
--label "smoke"This sends a formatted Slack message when any test in the run fails, including a link directly to the failed test in the Testim dashboard.
Testim dashboard notifications: In Project Settings > Notifications, you can configure email and Slack alerts triggered by test runs that finish with failures. These trigger regardless of how the run was initiated — CLI, scheduled run, or API call.
PagerDuty integration: For production monitoring test runs (scheduled tests hitting production), Testim's PagerDuty integration can open incidents directly. Configure the integration in the Testim dashboard and pass the --integrations pagerduty flag in your CLI invocation for production runs.
Scheduled Test Runs
For production smoke tests that run on a schedule rather than triggered by code changes, use Testim's built-in scheduler in the dashboard (Project Settings > Scheduled Runs) rather than CI cron jobs. The dashboard scheduler handles retries, sends notifications through configured channels, and keeps run history separate from CI-triggered runs so you can track production health trends independently.
If you prefer cron in CI, add a workflow with on: schedule: in GitHub Actions and point --base-url at your production URL. Use a read-only test user with no write permissions when running against production.
Summary
Testim CI integration is straightforward once you have the CLI installed and a token configured. The key decisions are: which labels map to which branches, what degree of parallelism your plan supports, and which output format your CI system's test reporting expects. For GitHub Actions, the JUnit reporter gives you inline PR annotations. For Jenkins, the built-in JUnit publisher handles trend tracking. The --retries flag is worth enabling in all CI runs to filter genuine flakiness from network noise.