TestComplete CI/CD Integrations: Jenkins, Azure DevOps, and More

TestComplete CI/CD Integrations: Jenkins, Azure DevOps, and More

TestComplete tests run headlessly from the command line, which means they can plug into any CI/CD pipeline that can execute Windows batch or PowerShell commands. The practical challenge is the Windows-only runtime, license management at scale, and publishing test results in a format your CI tool understands. This guide covers the real integration work — not just "add a build step."

The CI Architecture

Before configuration details, understand the execution model:

TestComplete is the full IDE — for authoring tests, viewing results, and debugging. It's what QA engineers use at their desks. License cost is per seat.

TestExecute is a runtime-only license for CI. It runs TestComplete projects without the IDE. Cheaper than TestComplete per node, designed for unattended execution. For CI pipelines, you install TestExecute on agents, not TestComplete.

SmartBear License Server manages floating licenses across nodes. When a build starts, the agent checks out a license; when it finishes, the license returns to the pool. Without a license server, you need one fixed license per machine.

Windows Agent Requirements

TestComplete/TestExecute only runs on Windows. This has real implications for your CI setup:

  • Jenkins: Windows build agents required for TestComplete jobs. Your Jenkins controller can be Linux; the agents that run TestComplete tests must be Windows.
  • Azure DevOps: Use Windows-hosted agents (Microsoft-hosted or self-hosted Windows VMs).
  • GitHub Actions: Use windows-latest runner or self-hosted Windows runners.
  • GitLab CI: Windows runners (self-hosted; GitLab.com doesn't offer Windows shared runners for free tiers).

If your existing agents are Linux, you'll need dedicated Windows nodes for TestComplete. This is a real infrastructure investment — factor it in.

Installing TestExecute on CI Agents

On each Windows build agent:

  1. Download TestExecute from SmartBear's website (same version as the TestComplete used to author the tests — version mismatch causes runtime errors).
  2. Install silently:
# Silent install of TestExecute
.\TestExecuteInstaller.exe /S /D="C:\Program Files\SmartBear\TestExecute 15"
  1. Activate the license. For license server:
# Point TestExecute to the license server
TestExecute.exe /licenseserver:licenseserver.internal.com /activate

For single-machine licenses, activate via the SmartBear License Manager UI on first run.

  1. Install browser extensions on the agent (for web tests):
# Chrome extension — install via policy or manually
# Edge extension — same

For headless CI agents (no display), use a virtual display or configure tests to run in headless-compatible mode. Chrome headless mode is supported.

Command-Line Execution Reference

The core command for running TestComplete/TestExecute in CI:

TestExecute.exe "C:\Tests\MyProject.pjs" ^
  /run ^
  /exit ^
  /SilentMode ^
  /exportlog:"C:\Results\log.xml" ^
  /p:"ProjectName" ^
  /test:"TestSuite\TestItem"

Key parameters:

Parameter Purpose
/run Execute tests immediately on launch
/exit Close after execution completes
/SilentMode Suppress dialogs that block unattended execution
/exportlog:"path" Export results to XML or MHT
/p:"ProjectName" Target a specific project in the suite
/test:"Path" Target a specific test item (slash-separated)
/p:var:Name=Value Set a project variable at runtime
/timeout:minutes Maximum execution time before forced exit

Exit codes:

  • 0 — all tests passed
  • 1 — one or more tests failed
  • 2 — TestComplete/TestExecute failed to start (license, config error)

Jenkins Integration

Freestyle Job

  1. In Jenkins, create a new Freestyle project
  2. Add a Execute Windows batch command build step:
cd C:\Tests
TestExecute.exe "MyProject.pjs" /run /exit /SilentMode /exportlog:"%WORKSPACE%\results\log.xml"
if %ERRORLEVEL% NEQ 0 exit /b %ERRORLEVEL%
  1. Add a Post-build action: Publish JUnit test results. TestExecute exports JUnit XML when you use:
TestExecute.exe "MyProject.pjs" /run /exit /SilentMode /exportlog:"%WORKSPACE%\results\junit.xml" /ExportLogFormat:JUnit

Then set the JUnit XML path to results/junit.xml in the post-build action.

Pipeline Job (Jenkinsfile)

pipeline {
    agent {
        label 'windows-testcomplete'  // must be a Windows agent with TestExecute
    }

    environment {
        TEST_PROJECT = 'C:\\Tests\\MyProject.pjs'
        RESULTS_DIR = "${WORKSPACE}\\results"
        TESTEXECUTE = 'C:\\Program Files\\SmartBear\\TestExecute 15\\Bin\\TestExecute.exe'
    }

    stages {
        stage('Prepare') {
            steps {
                bat "if not exist ${RESULTS_DIR} mkdir ${RESULTS_DIR}"
            }
        }

        stage('Run Tests') {
            steps {
                bat """
                    "${TESTEXECUTE}" "${TEST_PROJECT}" ^
                    /run /exit /SilentMode ^
                    /exportlog:"${RESULTS_DIR}\\junit.xml" ^
                    /ExportLogFormat:JUnit ^
                    /p:var:BaseURL="${params.BASE_URL}"
                """
            }
        }
    }

    post {
        always {
            junit "${RESULTS_DIR}\\junit.xml"
            archiveArtifacts artifacts: "${RESULTS_DIR}\\**", allowEmptyArchive: true
        }
    }
}

Parameterized Builds

Pass environment-specific values at runtime:

parameters {
    string(name: 'BASE_URL', defaultValue: 'https://staging.example.com', description: 'Target URL')
    choice(name: 'BROWSER', choices: ['Chrome', 'Firefox', 'Edge'], description: 'Browser to test')
    string(name: 'TEST_SUITE', defaultValue: 'RegressionSuite', description: 'Test suite to run')
}

Then in the batch command:

TestExecute.exe "Project.pjs" /run /exit /SilentMode ^
  /p:var:BaseURL="%BASE_URL%" ^
  /p:var:Browser="%BROWSER%" ^
  /test:"%TEST_SUITE%"

Azure DevOps Integration

YAML Pipeline

trigger:
  - main
  - develop

pool:
  vmImage: 'windows-2022'  # or self-hosted Windows agent

variables:
  testProject: 'tests\MyProject.pjs'
  resultsDir: '$(Build.ArtifactStagingDirectory)\results'
  testExecutePath: 'C:\Program Files\SmartBear\TestExecute 15\Bin\TestExecute.exe'

stages:
  - stage: AutomatedTests
    displayName: 'Run TestComplete Tests'
    jobs:
      - job: TestExecution
        displayName: 'Execute Test Suite'
        steps:
          - task: PowerShell@2
            displayName: 'Create results directory'
            inputs:
              script: |
                New-Item -ItemType Directory -Force -Path "$(resultsDir)"

          - task: CmdLine@2
            displayName: 'Run TestComplete Tests'
            inputs:
              script: |
                "$(testExecutePath)" "$(testProject)" ^
                  /run /exit /SilentMode ^
                  /exportlog:"$(resultsDir)\results.xml" ^
                  /ExportLogFormat:JUnit ^
                  /p:var:BaseURL="$(BaseURL)"

          - task: PublishTestResults@2
            displayName: 'Publish Test Results'
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '$(resultsDir)\results.xml'
              failTaskOnFailedTests: true
            condition: always()

          - task: PublishBuildArtifacts@1
            displayName: 'Publish Test Artifacts'
            inputs:
              PathtoPublish: '$(resultsDir)'
              ArtifactName: 'TestResults'
            condition: always()

Variable Groups for Environment Config

Store environment-specific values in Azure DevOps Variable Groups:

  1. Go to Pipelines → Library → Variable Groups
  2. Create groups: staging-config, production-config
  3. Add variables: BaseURL, TestUserEmail, TestUserPassword
  4. Link the variable group in your pipeline:
variables:
  - group: staging-config

Mark sensitive variables (passwords) as Secret — they'll be masked in logs and not accessible via the API.

Self-Hosted Windows Agents

For teams with dedicated hardware or VMs:

  1. Create an agent pool in Azure DevOps (Project Settings → Agent Pools)
  2. Download the agent package to your Windows machine
  3. Configure and register:
.\config.cmd --url https://dev.azure.com/yourorg --auth pat --token YOUR_PAT --pool "TestComplete-Agents" --agent "tc-agent-01"
.\run.cmd
  1. Install TestExecute and browser extensions on the agent machine
  2. In your pipeline, target the pool: pool: { name: 'TestComplete-Agents' }

GitHub Actions Integration

name: TestComplete Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: windows-latest  # or self-hosted Windows runner

    steps:
      - uses: actions/checkout@v4

      - name: Install TestExecute
        run: |
          # Download and install TestExecute silently
          # Assumes installer is pre-deployed to the runner, or downloaded here
          Start-Process -FilePath ".\TestExecuteSetup.exe" -ArgumentList "/S" -Wait
        shell: pwsh

      - name: Activate License
        run: |
          & "C:\Program Files\SmartBear\TestExecute 15\Bin\TestExecute.exe" /licenseserver:${{ secrets.LICENSE_SERVER }} /activate
        shell: pwsh

      - name: Run Tests
        run: |
          $result = Start-Process -FilePath "C:\Program Files\SmartBear\TestExecute 15\Bin\TestExecute.exe" `
            -ArgumentList '"${{ github.workspace }}\tests\Project.pjs" /run /exit /SilentMode /exportlog:"${{ github.workspace }}\results\junit.xml" /ExportLogFormat:JUnit' `
            -Wait -PassThru
          exit $result.ExitCode
        shell: pwsh

      - name: Publish Test Results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: TestComplete Results
          path: results/junit.xml
          reporter: java-junit

      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: results/

For self-hosted runners (recommended for license cost management):

jobs:
  test:
    runs-on: [self-hosted, windows, testcomplete]

Add the testcomplete label to runners that have TestExecute installed.

JUnit XML Export and Result Publishing

TestComplete doesn't natively produce JUnit XML, but the /ExportLogFormat:JUnit flag (TestComplete 14.4+) generates a JUnit-compatible format. For older versions, transform the native XML log:

TestComplete native log (XML) structure:

<TestLog>
  <Node NodeType="TestItem" Name="LoginTests" Status="0">
    <Node NodeType="Checkpoint" Name="Verify dashboard visible" Status="0"/>
    <Node NodeType="Checkpoint" Name="Verify welcome message" Status="3"/>
  </Node>
</TestLog>

Status values: 0 = pass, 1 = warning, 2 = error, 3 = fail.

Transform to JUnit with PowerShell:

# Convert TestComplete XML log to JUnit XML
param(
    [string]$InputPath,
    [string]$OutputPath
)

$xml = [xml](Get-Content $InputPath)
$tests = $xml.SelectNodes("//Node[@NodeType='Checkpoint']")

$suiteXml = New-Object System.Xml.XmlDocument
$suite = $suiteXml.CreateElement("testsuite")
$suite.SetAttribute("name", "TestComplete")
$suite.SetAttribute("tests", $tests.Count)

foreach ($test in $tests) {
    $case = $suiteXml.CreateElement("testcase")
    $case.SetAttribute("name", $test.Name)
    
    if ($test.Status -eq "3" -or $test.Status -eq "2") {
        $failure = $suiteXml.CreateElement("failure")
        $failure.SetAttribute("message", "Test failed")
        $case.AppendChild($failure) | Out-Null
    }
    
    $suite.AppendChild($case) | Out-Null
}

$suiteXml.AppendChild($suite) | Out-Null
$suiteXml.Save($OutputPath)

Parallel Test Execution

TestComplete tests run sequentially within a single instance. For parallelism, run multiple TestExecute instances concurrently against different test suites.

Jenkins Parallel Stages

stage('Parallel Tests') {
    parallel {
        stage('Suite A') {
            agent { label 'tc-agent-01' }
            steps {
                bat 'TestExecute.exe "Project.pjs" /run /test:"SuiteA" /exit /SilentMode /exportlog:"results\\suiteA.xml" /ExportLogFormat:JUnit'
            }
        }
        stage('Suite B') {
            agent { label 'tc-agent-02' }
            steps {
                bat 'TestExecute.exe "Project.pjs" /run /test:"SuiteB" /exit /SilentMode /exportlog:"results\\suiteB.xml" /ExportLogFormat:JUnit'
            }
        }
    }
}

Each agent needs a TestExecute license checked out from the license server. Plan your license pool accordingly — running 4 agents in parallel requires 4 concurrent TestExecute licenses.

Splitting Tests by Tag

TestComplete doesn't have built-in tag filtering. Implement a split strategy by creating separate Test Item configurations in the project:

  • SuiteA — smoke tests, fast
  • SuiteB — regression, slow
  • SuiteC — data-driven, parameterized

Then target them via /test:"SuiteA" on different CI agents.

Managing Project Variables per Environment

Avoid hardcoding environment-specific values in tests. Use project variables and override them at runtime:

In TestComplete — define project variables:

  • BaseURL = https://staging.example.com (default)
  • APIKey = test-key (default)
  • TestEmail = testuser@example.com (default)

In CI — override via /p:var::

TestExecute.exe "Project.pjs" /run /exit /SilentMode ^
  /p:var:BaseURL="https://prod.example.com" ^
  /p:var:APIKey="%PROD_API_KEY%" ^
  /p:var:TestEmail="%PROD_TEST_EMAIL%"

Sensitive values (API keys, passwords) should come from your CI secret store — never hardcoded in pipeline YAML or batch scripts.

Troubleshooting CI Failures

"License checkout failed": The license server is unreachable from the build agent, or all licenses are in use. Check network access to the license server, and verify the license count is sufficient for concurrent builds.

"Browser not found": The Chrome/Firefox version on the agent doesn't have a compatible TestComplete extension. Ensure the extension is installed and enabled for all users (system-wide, not just the current user account).

"Cannot open project": The project file path is wrong, or the project references resources (data files, scripts) that don't exist at the expected paths on the agent. Use relative paths in the project and ensure the repository structure is preserved.

Tests pass locally but fail in CI: Usually a timing issue — CI agents are slower than developer machines. Increase wait timeouts, or check for display resolution differences that affect element recognition.

Process "TestExecute.exe" hangs: A dialog box appeared and TestComplete didn't suppress it. Ensure /SilentMode is in the command and that all test scripts use try/catch to handle unexpected popups.

Integrating with Test Management Tools

TestComplete integrates with several test management platforms:

Zephyr Scale (Jira): Use the SmartBear Zephyr Scale plugin. After test execution, results sync to Jira test cycles automatically.

TestRail: Export JUnit XML from TestComplete, then use TestRail's JUnit import feature or the TestRail API to push results.

Azure Test Plans: Native integration — test results from the JUnit XML publish task in Azure Pipelines appear in Azure Test Plans automatically.

Xray (Jira): Import JUnit XML via Xray's REST API or the Xray Jenkins plugin.

Summary

TestComplete's CI integration is straightforward once you accept the Windows runtime constraint and plan your license infrastructure. The command-line interface is clean, JUnit export works with all major CI result publishing tools, and parallel execution is achievable with multiple agents.

The main investments: Windows build agents, TestExecute licenses, and browser extension management. Get those right, and TestComplete fits naturally into enterprise CI pipelines alongside your existing Java, .NET, or Node.js build processes.

Read more

Start now free