Ranorex CI/CD Integration: Running GUI Tests in Jenkins, Azure DevOps, and GitHub Actions

Running Ranorex tests in CI is not as simple as pointing a script at your test suite and pressing go. GUI automation requires a display, a licensed runner, and some thought about parallelism. This guide covers the actual setup — command-line execution, environment requirements, and pipeline configuration for Jenkins, Azure DevOps, and GitHub Actions.

Prerequisites for CI Execution

Before touching your pipeline config, make sure the CI agent has:

1. Windows OS — Ranorex only runs on Windows (desktop test execution requires the native Windows automation APIs)
2. Ranorex Runtime license — Studio licenses don't cover CI execution; you need a separate Runtime license for each concurrent execution node
3. Application under test installed — for desktop apps, the AUT must be installed on the CI agent
4. Visual C++ Redistributable — required by Ranorex's runtime
5. Browser drivers — if running web tests, ChromeDriver or GeckoDriver must be present and match the installed browser version

For virtual or headless environments (cloud CI agents, Docker): Ranorex GUI tests require an actual display. Use a virtual display solution — see the virtual display section below.

The Command-Line Runner

All Ranorex test execution in CI goes through the command-line runner — RanorexRunner.exe for pre-built test suites, or directly via the compiled test executable.

Build the test suite first in Ranorex Studio (or via MSBuild), then invoke the output:

# Basic execution
MyTestSuite.exe /rf:TestResults /ro:report.rxzlog

# With suite selection
MyTestSuite.exe /ts:"SmokeTests" /rf:TestResults /ro:smoke-report.rxzlog

# With test case filter
MyTestSuite.exe /tc:"LoginTest,CheckoutTest" /rf:TestResults

# JUnit XML output (for CI test result parsers)
MyTestSuite.exe /rf:TestResults /ro:results.xml /reportformat:junit

# With data binding override
MyTestSuite.exe /db:"DataSource=production-data.xlsx"

Key flags:

• /rf — report folder (output directory)
• /ro — report output filename
• /ts — test suite filter (run only matching suites)
• /tc — test case filter (comma-separated)
• /reportformat:junit — outputs JUnit XML instead of Ranorex HTML
• /db — override the data binding for data-driven tests
• /pa — set parameters passed into the test at runtime

Building in CI

If you're storing source files in version control (not compiled binaries), build before executing:

# MSBuild (available on Windows agents with Visual Studio or Build Tools)
msbuild MyTestSuite.sln /p:Configuration=Release /p:Platform="Any CPU"

# Then execute
.\MyTestSuite\bin\Release\MyTestSuite.exe /rf:TestResults /reportformat:junit

For teams that want clean builds every run, the build step adds a few minutes. For teams that commit compiled binaries, skip the build step — but you'll need a discipline around keeping compiled output in sync with source.

Jenkins Setup

Prerequisites on Jenkins agent: Windows agent with Ranorex Runtime license, installed application, browser drivers.

Pipeline (Jenkinsfile):

pipeline {
    agent { label 'windows-ranorex' }
    
    stages {
        stage('Build Tests') {
            steps {
                bat 'msbuild MyTestSuite.sln /p:Configuration=Release'
            }
        }
        
        stage('Run Tests') {
            steps {
                bat '''
                    MyTestSuite\\bin\\Release\\MyTestSuite.exe ^
                        /rf:TestResults ^
                        /reportformat:junit ^
                        /ro:test-results.xml
                '''
            }
            post {
                always {
                    junit 'TestResults/test-results.xml'
                    archiveArtifacts artifacts: 'TestResults/**', allowEmptyArchive: true
                }
            }
        }
    }
}

Collecting results: The junit step in Jenkins reads the JUnit XML and displays test results in the build summary. Failed tests appear as failures in the build report.

Screenshots: Ranorex captures screenshots per action by default. Archive the TestResults folder to preserve them — archiveArtifacts above covers this.

Azure DevOps Setup

Pipeline (azure-pipelines.yml):

trigger:
  branches:
    include:
      - main

pool:
  name: 'Windows-Ranorex-Agents'  # self-hosted Windows pool

steps:
  - task: MSBuild@1
    displayName: 'Build Ranorex test suite'
    inputs:
      solution: 'MyTestSuite.sln'
      configuration: 'Release'

  - script: |
      MyTestSuite\bin\Release\MyTestSuite.exe ^
        /rf:$(Build.ArtifactStagingDirectory)\TestResults ^
        /reportformat:junit ^
        /ro:test-results.xml
    displayName: 'Run Ranorex tests'

  - task: PublishTestResults@2
    displayName: 'Publish test results'
    condition: always()
    inputs:
      testResultsFormat: 'JUnit'
      testResultsFiles: '$(Build.ArtifactStagingDirectory)/TestResults/test-results.xml'
      testRunTitle: 'Ranorex GUI Tests'

  - task: PublishBuildArtifacts@1
    displayName: 'Archive test artifacts'
    condition: always()
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)/TestResults'
      ArtifactName: 'test-results'

Test reports in Azure DevOps: PublishTestResults makes results visible in the Tests tab of the build. Failures show up with test names, duration, and error messages.

Variable groups: Store environment-specific parameters (URLs, credentials) in Azure DevOps variable groups and pass them to tests via /pa:varName=value flags.

GitHub Actions Setup

GitHub Actions hosted runners are Ubuntu-based by default — they won't run Ranorex. You need a self-hosted Windows runner.

Self-hosted runner setup:

1. Go to your repo → Settings → Actions → Runners → New self-hosted runner
2. Follow the Windows installation instructions
3. Install Ranorex Runtime license on the runner machine

Workflow (.github/workflows/ranorex-tests.yml):

name: Ranorex GUI Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: self-hosted  # Windows runner with Ranorex

    steps:
      - uses: actions/checkout@v4

      - name: Build test suite
        run: msbuild MyTestSuite.sln /p:Configuration=Release
        shell: cmd

      - name: Run Ranorex tests
        run: |
          MyTestSuite\bin\Release\MyTestSuite.exe `
            /rf:TestResults `
            /reportformat:junit `
            /ro:test-results.xml
        shell: pwsh

      - name: Publish test results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: 'Ranorex Test Results'
          path: 'TestResults/test-results.xml'
          reporter: 'java-junit'

      - name: Upload test artifacts
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: TestResults/

Virtual Display for Headless Environments

Ranorex tests open real windows. On a headless CI agent or RDP session that disconnects, tests will fail because there's no display to render to.

Option 1: Keep an RDP session open — the simplest approach for dedicated Windows agents. Connect via RDP before starting the build and don't disconnect (minimize, don't close). Crude but effective.

Option 2: Virtual desktop (recommended) — Use a service that runs the GUI in a virtual desktop that persists regardless of RDP state:

# Start a virtual desktop session before tests (Windows)
# Using SysinternalsSuite Desktops or similar
# Or configure the CI agent as a Windows service that logs in as a specific user

Option 3: Windows Server with Desktop Experience — Ensure the CI agent runs Windows Server with Desktop Experience (not Core) and is configured to always have a desktop available.

For Docker/containers: Ranorex desktop tests cannot run in Linux containers. Windows containers with Desktop Experience support are technically possible but rarely used — the complexity isn't worth it for most teams.

Parallel Execution

Each parallel test run requires its own Runtime license. Ranorex doesn't support in-process parallelism; parallel execution means multiple CI agents each running a separate test suite.

Splitting test suites for parallel execution:

// Jenkins parallel stages
stage('Parallel Tests') {
    parallel {
        stage('Suite A') {
            agent { label 'ranorex-agent-1' }
            steps {
                bat 'MyTestSuite.exe /ts:"SuiteA" /rf:ResultsA /reportformat:junit /ro:results-a.xml'
            }
        }
        stage('Suite B') {
            agent { label 'ranorex-agent-2' }
            steps {
                bat 'MyTestSuite.exe /ts:"SuiteB" /rf:ResultsB /reportformat:junit /ro:results-b.xml'
            }
        }
    }
}

Merge JUnit results after parallel stages for a unified report. Most CI tools support multiple JUnit XML files in the publish step.

Common CI Failures and Fixes

"No license found" error:

• Runtime license not installed on the CI agent
• License server unreachable (for floating licenses) — check network connectivity
• License in use by another concurrent run — add more runtime licenses or stagger runs

"Cannot create window" or display errors:

• No desktop session available — set up a virtual desktop or persistent logon session
• Run under a service account that has "Log on as a service" and desktop access rights

Element not found / timeout:

• Application not fully loaded before test action — increase step timeouts in Ranorex Studio settings
• Application version mismatch between dev and CI environments

Build fails, tests not run:

• MSBuild version mismatch — ensure the CI agent has the same Visual Studio Build Tools version as the development machine
• Missing NuGet packages — run nuget restore before MSBuild

Test Result Trending

Most CI tools track test result history across builds. Jenkins (Blue Ocean), Azure DevOps (Test Plans), and GitHub Actions (with the dorny/test-reporter action) all provide trend views when you consistently publish JUnit results.

For more detailed analytics — flakiness rates, slowest tests, failure frequency by test — consider publishing results to a test management tool like qTest or TestRail in addition to the CI-native reports. Ranorex's JUnit output is compatible with both.