Testing

Squish CI Integration: Running GUI Tests in Jenkins and Azure DevOps

HelpMeTest

HelpMeTest

— 4 min read
Squish CI Integration: Running GUI Tests in Jenkins and Azure DevOps

Squish is built for scripted, repeatable GUI test execution, which makes it a good fit for CI/CD pipelines — in principle. In practice, getting Squish running reliably in CI requires solving several infrastructure problems: license server availability, display requirements for desktop GUI applications, parallel execution architecture, and test result integration. This post covers the practical setup for Jenkins and Azure DevOps.

The squishrunner Command Line

squishrunner is Squish's command-line test execution interface. It connects to a running squishserver, executes test suites or individual test cases, and produces output including a JUnit-compatible XML report.

A basic squishrunner invocation:

squishrunner \
  --host localhost \
  --port 4322 \
  --testsuite /path/to/suite \
  --testcase tst_login_happy_path \
  --reportgen junit,/tmp/squish-results.xml \
  --exitCodeOnFail 1

Breaking this down:

  • --host and --port: The address of the squishserver process. In CI, this is typically localhost if squishserver runs on the same agent, or a remote address if you're using a centralized Squish server setup.
  • --testsuite: Path to the test suite directory.
  • --testcase: Run a specific test case. Omit to run all test cases in the suite.
  • --reportgen junit,/path/to/output.xml: Generate JUnit XML output at the specified path.
  • --exitCodeOnFail 1: Exit with code 1 if any test fails, which CI systems interpret as a build failure.

For running multiple test cases, you can specify --testcase multiple times, or run the entire suite by omitting it. You can also run multiple suites by calling squishrunner multiple times in sequence.

License Server Configuration

Squish requires a license to run. In a developer workstation setup, the license is typically stored locally. In CI environments, you need either a floating license server or a CI-specific license configuration.

Squish supports a floating license model through a license server (squishserver can act as a license distributor in some configurations, or you use a dedicated license server). CI agents check out a license when running tests and release it when done.

Configuration for CI agents to use a license server:

export SQUISH_LICENSEKEY_PATH=/path/to/license/file
# or for a floating license server:
export SQUISH_SERVER_LICENSE_HOST=license-server-hostname
export SQUISH_SERVER_LICENSE_PORT=49344

If your CI runs more parallel jobs than you have license seats, jobs queue on license checkout. This is usually the constraint on parallelism rather than hardware capacity.

For organizations with a single-seat or small number of seats, configure CI to serialize Squish test jobs using pipeline resource locking rather than letting jobs pile up on license waits.

Jenkins Pipeline Setup

A Jenkins pipeline stage for Squish tests:

pipeline {
    agent { label 'squish-agent' }
    
    stages {
        stage('Start Squish Server') {
            steps {
                sh 'squishserver --daemon'
                sh 'sleep 3' // brief pause for server startup
            }
        }
        
        stage('Run Squish Tests') {
            steps {
                sh """
                    squishrunner \
                      --host localhost \
                      --port 4322 \
                      --testsuite ${WORKSPACE}/tests/suite_regression \
                      --reportgen junit,${WORKSPACE}/squish-results.xml \
                      --exitCodeOnFail 1
                """
            }
            post {
                always {
                    junit 'squish-results.xml'
                }
            }
        }
        
        stage('Stop Squish Server') {
            steps {
                sh 'squishserver --stop'
            }
        }
    }
}

The junit post step publishes the JUnit XML to Jenkins' test reporting system, making pass/fail visible in the build results, trend charts, and failure detail views.

The agent label squish-agent restricts this pipeline to Jenkins agents where Squish is installed and display requirements are met. Not all CI agents in a pool need Squish — label-based routing sends Squish jobs only to capable agents.

Azure DevOps Integration

Azure DevOps doesn't have a native Squish task, but the pipeline YAML approach with shell steps works cleanly:

pool:
  name: Squish Agents
  
steps:
- bash: |
    squishserver --daemon
    sleep 3
  displayName: 'Start Squish Server'

- bash: |
    squishrunner \
      --host localhost \
      --port 4322 \
      --testsuite $(Build.SourcesDirectory)/tests/suite_regression \
      --reportgen junit,$(Agent.TempDirectory)/squish-results.xml \
      --exitCodeOnFail 1
  displayName: 'Run Squish Tests'

- task: PublishTestResults@2
  condition: always()
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(Agent.TempDirectory)/squish-results.xml'
    testRunTitle: 'Squish GUI Tests'
  displayName: 'Publish Test Results'

- bash: squishserver --stop
  condition: always()
  displayName: 'Stop Squish Server'

The condition: always() on the publish and stop steps ensures results are published and the server is stopped even if tests fail — important for keeping the agent in a clean state for subsequent runs.

The pool: name: Squish Agents targets a specific agent pool where Squish is installed. Azure DevOps self-hosted agents in this pool need Squish installed, licensed, and the squishrunner/squishserver binaries on the PATH.

Handling Display Requirements on Headless Agents

This is the most common CI problem with GUI test automation. Desktop GUI applications need a display to render into. Linux CI agents typically don't have a physical display, which causes application launch to fail.

The solution is a virtual display using Xvfb (X Virtual Framebuffer):

# Start Xvfb on display :99
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99

# Now start squishserver and run squishrunner with the display set
squishserver --daemon
squishrunner --host localhost --port 4322 --testsuite ...

A cleaner wrapper using xvfb-run:

xvfb-run -a -s "-screen 0 1920x1080x24" squishrunner \
  --host localhost \
  --port 4322 \
  --testsuite /path/to/suite \
  --reportgen junit,/tmp/results.xml \
  --exitCodeOnFail 1

xvfb-run -a automatically selects an available display number, avoiding conflicts when multiple CI jobs run on the same machine.

For applications that use OpenGL or hardware-accelerated rendering, Xvfb may not provide sufficient display capabilities. In those cases, a virtual machine with a proper display driver (using tools like VirtualGL + TurboVNC) or a Windows CI agent with a remote desktop session are alternatives.

On Windows CI agents, a common issue is that interactive service processes can't access the desktop session used by the CI agent service. Running the Jenkins agent (or Azure DevOps agent) as an interactive process rather than a service resolves most Windows display issues.

Parallel Test Execution

Running multiple Squish test suites concurrently reduces total execution time. Parallelism in Squish requires:

  1. Multiple license seats (each concurrent runner needs a license)
  2. Multiple CI agent machines (or separate squishserver instances on one machine with different ports)
  3. Pipeline configuration to distribute suites across agents

Jenkins parallel pipeline:

stage('Squish Tests') {
    parallel {
        stage('Suite A') {
            agent { label 'squish-agent-1' }
            steps {
                sh 'squishrunner --testsuite suite_a --reportgen junit,results_a.xml'
            }
            post { always { junit 'results_a.xml' } }
        }
        stage('Suite B') {
            agent { label 'squish-agent-2' }
            steps {
                sh 'squishrunner --testsuite suite_b --reportgen junit,results_b.xml'
            }
            post { always { junit 'results_b.xml' } }
        }
    }
}

Each parallel branch runs on a separate labeled agent. The results are published independently and Jenkins aggregates them into the overall build test report.

Test Result Archiving

Beyond JUnit XML for the current run, archiving Squish's native .squishreport files preserves detailed per-assertion results and embedded screenshots that the JUnit format doesn't capture.

squishrunner \
  --reportgen junit,${WORKSPACE}/squish-junit.xml \
  --reportgen xml2,${WORKSPACE}/squish-report.xml \
  --exitCodeOnFail 1

The xml2 format produces Squish's detailed XML report. Archive this as a build artifact for later investigation. Jenkins' archiveArtifacts step and Azure DevOps' PublishBuildArtifacts task handle archiving.

For failures that require visual investigation, screenshots captured during test execution are embedded in the Squish report or written to a configurable directory. Archiving the screenshot directory alongside the report file gives investigators the visual context needed to diagnose failures without re-running the test.

Read more

Start now free