TestComplete CI/CD Integrations: Jenkins, Azure DevOps, and More
TestComplete tests run headlessly from the command line, which means they can plug into any CI/CD pipeline that can execute Windows batch or PowerShell commands. The practical challenge is the Windows-only runtime, license management at scale, and publishing test results in a format your CI tool understands. This guide covers the real integration work — not just "add a build step."
The CI Architecture
Before configuration details, understand the execution model:
TestComplete is the full IDE — for authoring tests, viewing results, and debugging. It's what QA engineers use at their desks. License cost is per seat.
TestExecute is a runtime-only license for CI. It runs TestComplete projects without the IDE. Cheaper than TestComplete per node, designed for unattended execution. For CI pipelines, you install TestExecute on agents, not TestComplete.
SmartBear License Server manages floating licenses across nodes. When a build starts, the agent checks out a license; when it finishes, the license returns to the pool. Without a license server, you need one fixed license per machine.
Windows Agent Requirements
TestComplete/TestExecute only runs on Windows. This has real implications for your CI setup:
- Jenkins: Windows build agents required for TestComplete jobs. Your Jenkins controller can be Linux; the agents that run TestComplete tests must be Windows.
- Azure DevOps: Use Windows-hosted agents (Microsoft-hosted or self-hosted Windows VMs).
- GitHub Actions: Use
windows-latestrunner or self-hosted Windows runners. - GitLab CI: Windows runners (self-hosted; GitLab.com doesn't offer Windows shared runners for free tiers).
If your existing agents are Linux, you'll need dedicated Windows nodes for TestComplete. This is a real infrastructure investment — factor it in.
Installing TestExecute on CI Agents
On each Windows build agent:
- Download TestExecute from SmartBear's website (same version as the TestComplete used to author the tests — version mismatch causes runtime errors).
- Install silently:
# Silent install of TestExecute
.\TestExecuteInstaller.exe /S /D="C:\Program Files\SmartBear\TestExecute 15"- Activate the license. For license server:
# Point TestExecute to the license server
TestExecute.exe /licenseserver:licenseserver.internal.com /activateFor single-machine licenses, activate via the SmartBear License Manager UI on first run.
- Install browser extensions on the agent (for web tests):
# Chrome extension — install via policy or manually
# Edge extension — sameFor headless CI agents (no display), use a virtual display or configure tests to run in headless-compatible mode. Chrome headless mode is supported.
Command-Line Execution Reference
The core command for running TestComplete/TestExecute in CI:
TestExecute.exe "C:\Tests\MyProject.pjs" ^
/run ^
/exit ^
/SilentMode ^
/exportlog:"C:\Results\log.xml" ^
/p:"ProjectName" ^
/test:"TestSuite\TestItem"Key parameters:
| Parameter | Purpose |
|---|---|
/run |
Execute tests immediately on launch |
/exit |
Close after execution completes |
/SilentMode |
Suppress dialogs that block unattended execution |
/exportlog:"path" |
Export results to XML or MHT |
/p:"ProjectName" |
Target a specific project in the suite |
/test:"Path" |
Target a specific test item (slash-separated) |
/p:var:Name=Value |
Set a project variable at runtime |
/timeout:minutes |
Maximum execution time before forced exit |
Exit codes:
0— all tests passed1— one or more tests failed2— TestComplete/TestExecute failed to start (license, config error)
Jenkins Integration
Freestyle Job
- In Jenkins, create a new Freestyle project
- Add a Execute Windows batch command build step:
cd C:\Tests
TestExecute.exe "MyProject.pjs" /run /exit /SilentMode /exportlog:"%WORKSPACE%\results\log.xml"
if %ERRORLEVEL% NEQ 0 exit /b %ERRORLEVEL%- Add a Post-build action: Publish JUnit test results. TestExecute exports JUnit XML when you use:
TestExecute.exe "MyProject.pjs" /run /exit /SilentMode /exportlog:"%WORKSPACE%\results\junit.xml" /ExportLogFormat:JUnitThen set the JUnit XML path to results/junit.xml in the post-build action.
Pipeline Job (Jenkinsfile)
pipeline {
agent {
label 'windows-testcomplete' // must be a Windows agent with TestExecute
}
environment {
TEST_PROJECT = 'C:\\Tests\\MyProject.pjs'
RESULTS_DIR = "${WORKSPACE}\\results"
TESTEXECUTE = 'C:\\Program Files\\SmartBear\\TestExecute 15\\Bin\\TestExecute.exe'
}
stages {
stage('Prepare') {
steps {
bat "if not exist ${RESULTS_DIR} mkdir ${RESULTS_DIR}"
}
}
stage('Run Tests') {
steps {
bat """
"${TESTEXECUTE}" "${TEST_PROJECT}" ^
/run /exit /SilentMode ^
/exportlog:"${RESULTS_DIR}\\junit.xml" ^
/ExportLogFormat:JUnit ^
/p:var:BaseURL="${params.BASE_URL}"
"""
}
}
}
post {
always {
junit "${RESULTS_DIR}\\junit.xml"
archiveArtifacts artifacts: "${RESULTS_DIR}\\**", allowEmptyArchive: true
}
}
}Parameterized Builds
Pass environment-specific values at runtime:
parameters {
string(name: 'BASE_URL', defaultValue: 'https://staging.example.com', description: 'Target URL')
choice(name: 'BROWSER', choices: ['Chrome', 'Firefox', 'Edge'], description: 'Browser to test')
string(name: 'TEST_SUITE', defaultValue: 'RegressionSuite', description: 'Test suite to run')
}Then in the batch command:
TestExecute.exe "Project.pjs" /run /exit /SilentMode ^
/p:var:BaseURL="%BASE_URL%" ^
/p:var:Browser="%BROWSER%" ^
/test:"%TEST_SUITE%"Azure DevOps Integration
YAML Pipeline
trigger:
- main
- develop
pool:
vmImage: 'windows-2022' # or self-hosted Windows agent
variables:
testProject: 'tests\MyProject.pjs'
resultsDir: '$(Build.ArtifactStagingDirectory)\results'
testExecutePath: 'C:\Program Files\SmartBear\TestExecute 15\Bin\TestExecute.exe'
stages:
- stage: AutomatedTests
displayName: 'Run TestComplete Tests'
jobs:
- job: TestExecution
displayName: 'Execute Test Suite'
steps:
- task: PowerShell@2
displayName: 'Create results directory'
inputs:
script: |
New-Item -ItemType Directory -Force -Path "$(resultsDir)"
- task: CmdLine@2
displayName: 'Run TestComplete Tests'
inputs:
script: |
"$(testExecutePath)" "$(testProject)" ^
/run /exit /SilentMode ^
/exportlog:"$(resultsDir)\results.xml" ^
/ExportLogFormat:JUnit ^
/p:var:BaseURL="$(BaseURL)"
- task: PublishTestResults@2
displayName: 'Publish Test Results'
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(resultsDir)\results.xml'
failTaskOnFailedTests: true
condition: always()
- task: PublishBuildArtifacts@1
displayName: 'Publish Test Artifacts'
inputs:
PathtoPublish: '$(resultsDir)'
ArtifactName: 'TestResults'
condition: always()Variable Groups for Environment Config
Store environment-specific values in Azure DevOps Variable Groups:
- Go to Pipelines → Library → Variable Groups
- Create groups:
staging-config,production-config - Add variables:
BaseURL,TestUserEmail,TestUserPassword - Link the variable group in your pipeline:
variables:
- group: staging-configMark sensitive variables (passwords) as Secret — they'll be masked in logs and not accessible via the API.
Self-Hosted Windows Agents
For teams with dedicated hardware or VMs:
- Create an agent pool in Azure DevOps (Project Settings → Agent Pools)
- Download the agent package to your Windows machine
- Configure and register:
.\config.cmd --url https://dev.azure.com/yourorg --auth pat --token YOUR_PAT --pool "TestComplete-Agents" --agent "tc-agent-01"
.\run.cmd- Install TestExecute and browser extensions on the agent machine
- In your pipeline, target the pool:
pool: { name: 'TestComplete-Agents' }
GitHub Actions Integration
name: TestComplete Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
jobs:
test:
runs-on: windows-latest # or self-hosted Windows runner
steps:
- uses: actions/checkout@v4
- name: Install TestExecute
run: |
# Download and install TestExecute silently
# Assumes installer is pre-deployed to the runner, or downloaded here
Start-Process -FilePath ".\TestExecuteSetup.exe" -ArgumentList "/S" -Wait
shell: pwsh
- name: Activate License
run: |
& "C:\Program Files\SmartBear\TestExecute 15\Bin\TestExecute.exe" /licenseserver:${{ secrets.LICENSE_SERVER }} /activate
shell: pwsh
- name: Run Tests
run: |
$result = Start-Process -FilePath "C:\Program Files\SmartBear\TestExecute 15\Bin\TestExecute.exe" `
-ArgumentList '"${{ github.workspace }}\tests\Project.pjs" /run /exit /SilentMode /exportlog:"${{ github.workspace }}\results\junit.xml" /ExportLogFormat:JUnit' `
-Wait -PassThru
exit $result.ExitCode
shell: pwsh
- name: Publish Test Results
uses: dorny/test-reporter@v1
if: always()
with:
name: TestComplete Results
path: results/junit.xml
reporter: java-junit
- name: Upload Artifacts
uses: actions/upload-artifact@v4
if: always()
with:
name: test-results
path: results/For self-hosted runners (recommended for license cost management):
jobs:
test:
runs-on: [self-hosted, windows, testcomplete]Add the testcomplete label to runners that have TestExecute installed.
JUnit XML Export and Result Publishing
TestComplete doesn't natively produce JUnit XML, but the /ExportLogFormat:JUnit flag (TestComplete 14.4+) generates a JUnit-compatible format. For older versions, transform the native XML log:
TestComplete native log (XML) structure:
<TestLog>
<Node NodeType="TestItem" Name="LoginTests" Status="0">
<Node NodeType="Checkpoint" Name="Verify dashboard visible" Status="0"/>
<Node NodeType="Checkpoint" Name="Verify welcome message" Status="3"/>
</Node>
</TestLog>Status values: 0 = pass, 1 = warning, 2 = error, 3 = fail.
Transform to JUnit with PowerShell:
# Convert TestComplete XML log to JUnit XML
param(
[string]$InputPath,
[string]$OutputPath
)
$xml = [xml](Get-Content $InputPath)
$tests = $xml.SelectNodes("//Node[@NodeType='Checkpoint']")
$suiteXml = New-Object System.Xml.XmlDocument
$suite = $suiteXml.CreateElement("testsuite")
$suite.SetAttribute("name", "TestComplete")
$suite.SetAttribute("tests", $tests.Count)
foreach ($test in $tests) {
$case = $suiteXml.CreateElement("testcase")
$case.SetAttribute("name", $test.Name)
if ($test.Status -eq "3" -or $test.Status -eq "2") {
$failure = $suiteXml.CreateElement("failure")
$failure.SetAttribute("message", "Test failed")
$case.AppendChild($failure) | Out-Null
}
$suite.AppendChild($case) | Out-Null
}
$suiteXml.AppendChild($suite) | Out-Null
$suiteXml.Save($OutputPath)Parallel Test Execution
TestComplete tests run sequentially within a single instance. For parallelism, run multiple TestExecute instances concurrently against different test suites.
Jenkins Parallel Stages
stage('Parallel Tests') {
parallel {
stage('Suite A') {
agent { label 'tc-agent-01' }
steps {
bat 'TestExecute.exe "Project.pjs" /run /test:"SuiteA" /exit /SilentMode /exportlog:"results\\suiteA.xml" /ExportLogFormat:JUnit'
}
}
stage('Suite B') {
agent { label 'tc-agent-02' }
steps {
bat 'TestExecute.exe "Project.pjs" /run /test:"SuiteB" /exit /SilentMode /exportlog:"results\\suiteB.xml" /ExportLogFormat:JUnit'
}
}
}
}Each agent needs a TestExecute license checked out from the license server. Plan your license pool accordingly — running 4 agents in parallel requires 4 concurrent TestExecute licenses.
Splitting Tests by Tag
TestComplete doesn't have built-in tag filtering. Implement a split strategy by creating separate Test Item configurations in the project:
SuiteA— smoke tests, fastSuiteB— regression, slowSuiteC— data-driven, parameterized
Then target them via /test:"SuiteA" on different CI agents.
Managing Project Variables per Environment
Avoid hardcoding environment-specific values in tests. Use project variables and override them at runtime:
In TestComplete — define project variables:
BaseURL=https://staging.example.com(default)APIKey=test-key(default)TestEmail=testuser@example.com(default)
In CI — override via /p:var::
TestExecute.exe "Project.pjs" /run /exit /SilentMode ^
/p:var:BaseURL="https://prod.example.com" ^
/p:var:APIKey="%PROD_API_KEY%" ^
/p:var:TestEmail="%PROD_TEST_EMAIL%"Sensitive values (API keys, passwords) should come from your CI secret store — never hardcoded in pipeline YAML or batch scripts.
Troubleshooting CI Failures
"License checkout failed": The license server is unreachable from the build agent, or all licenses are in use. Check network access to the license server, and verify the license count is sufficient for concurrent builds.
"Browser not found": The Chrome/Firefox version on the agent doesn't have a compatible TestComplete extension. Ensure the extension is installed and enabled for all users (system-wide, not just the current user account).
"Cannot open project": The project file path is wrong, or the project references resources (data files, scripts) that don't exist at the expected paths on the agent. Use relative paths in the project and ensure the repository structure is preserved.
Tests pass locally but fail in CI: Usually a timing issue — CI agents are slower than developer machines. Increase wait timeouts, or check for display resolution differences that affect element recognition.
Process "TestExecute.exe" hangs: A dialog box appeared and TestComplete didn't suppress it. Ensure /SilentMode is in the command and that all test scripts use try/catch to handle unexpected popups.
Integrating with Test Management Tools
TestComplete integrates with several test management platforms:
Zephyr Scale (Jira): Use the SmartBear Zephyr Scale plugin. After test execution, results sync to Jira test cycles automatically.
TestRail: Export JUnit XML from TestComplete, then use TestRail's JUnit import feature or the TestRail API to push results.
Azure Test Plans: Native integration — test results from the JUnit XML publish task in Azure Pipelines appear in Azure Test Plans automatically.
Xray (Jira): Import JUnit XML via Xray's REST API or the Xray Jenkins plugin.
Summary
TestComplete's CI integration is straightforward once you accept the Windows runtime constraint and plan your license infrastructure. The command-line interface is clean, JUnit export works with all major CI result publishing tools, and parallel execution is achievable with multiple agents.
The main investments: Windows build agents, TestExecute licenses, and browser extension management. Get those right, and TestComplete fits naturally into enterprise CI pipelines alongside your existing Java, .NET, or Node.js build processes.