Gauge

Gauge CI/CD Integration: GitHub Actions, Jenkins, and Docker

HelpMeTest

HelpMeTest

— 8 min read
Gauge CI/CD Integration: GitHub Actions, Jenkins, and Docker

Gauge integrates naturally with any CI system: it exits with a non-zero code on failure, writes HTML reports to a configurable directory, and supports environment-specific configuration via the --env flag. GitHub Actions, Jenkins, and Docker each need a slightly different setup — mainly around browser dependencies for UI tests and report artifact handling. This post covers production-ready configurations for all three.

Key Takeaways

  • Gauge exits with code 1 on any test failure — CI systems pick this up automatically
  • Use --env ci to activate CI-specific properties (headless mode, staging URLs, etc.)
  • HTML reports can be uploaded as CI artifacts and retained for historical comparison
  • Docker-based execution ensures identical environments across developer machines and CI
  • Use --retry-failed in CI to handle flaky tests without manual re-runs
  • JUnit XML output (via the xml-report plugin) integrates with Jenkins test result tracking

What CI Needs from a Test Framework

A CI system fundamentally needs three things from a test runner:

  1. Exit code: non-zero on failure, zero on success
  2. Output: machine-readable results (JUnit XML) or human-readable reports
  3. Repeatability: the same test, on the same code, produces the same result

Gauge satisfies all three out of the box. The main work in CI setup is environment configuration, dependency installation, and report management.

Installing Gauge in CI

Most CI environments start from a clean Linux image on every run. You need to install Gauge and its plugins as part of the pipeline.

Linux (Debian/Ubuntu — most common in CI)

# Add Gauge apt repository
curl -SsL https://downloads.gauge.org/stable | sh

# Alternatively, download and install directly:
wget https://github.com/getgauge/gauge/releases/latest/download/gauge-linux.x86_64.zip
unzip gauge-linux.x86_64.zip -d /usr/local/bin/

# Install plugins (example: Java + HTML report + XML report)
gauge install java
gauge install html-report
gauge install xml-report
gauge install screenshot

Cache the Gauge home directory (~/.gauge) in CI to avoid re-downloading plugins on every run.

GitHub Actions

Basic Workflow

# .github/workflows/gauge-tests.yml
name: Gauge Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  GAUGE_HOME: ${{ github.workspace }}/.gauge-home

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Java 21
        uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: 'maven'

      - name: Cache Gauge plugins
        uses: actions/cache@v4
        with:
          path: ~/.gauge
          key: gauge-plugins-${{ runner.os }}-${{ hashFiles('manifest.json') }}
          restore-keys: |
            gauge-plugins-${{ runner.os }}-

      - name: Install Gauge
        run: |
          curl -SsL https://downloads.gauge.org/stable | sh
          gauge --version

      - name: Install Gauge plugins
        run: |
          gauge install java
          gauge install html-report
          gauge install xml-report
          gauge install screenshot

      - name: Install Chrome
        uses: browser-actions/setup-chrome@v1

      - name: Build project
        run: mvn compile test-compile -q

      - name: Run Gauge tests
        run: gauge run --env ci --parallel --n 4 specs/
        env:
          BASE_URL: ${{ secrets.STAGING_BASE_URL }}
          GAUGE_HTML_REPORT_THEME_PATH: ${{ github.workspace }}/custom-theme

      - name: Upload HTML report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: gauge-html-report-${{ github.run_number }}
          path: reports/html-report/
          retention-days: 30

      - name: Upload XML report (for test summary)
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: gauge-xml-report
          path: reports/xml-report/

      - name: Publish test results
        if: always()
        uses: EnricoMi/publish-unit-test-result-action@v2
        with:
          files: reports/xml-report/**/*.xml

Adding Retry for Flaky Tests

      - name: Run Gauge tests with retry
        run: |
          gauge run --env ci --parallel --n 4 specs/ || \
          gauge run --env ci --only-failures --retry-failed 2 specs/

The --retry-failed N flag retries each failing scenario up to N times before marking it as failed. Combined with --only-failures, this re-runs only the specs that failed in the first run.

Pull Request Commenting

Post a summary of test results as a PR comment using the XML report:

      - name: Parse test results
        if: always()
        id: test-results
        run: |
          TOTAL=$(python3 -c "
          import xml.etree.ElementTree as ET
          import glob
          total, failed = 0, 0
          for f in glob.glob('reports/xml-report/**/*.xml', recursive=True):
              tree = ET.parse(f)
              root = tree.getroot()
              total += int(root.get('tests', 0))
              failed += int(root.get('failures', 0)) + int(root.get('errors', 0))
          print(f'{total},{failed}')
          ")
          echo "summary=$TOTAL" >> $GITHUB_OUTPUT

      - name: Comment on PR
        if: always() && github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '## Gauge Test Results\n\nReport uploaded as artifact `gauge-html-report-${{ github.run_number }}`.'
            })

Jenkins

Declarative Pipeline

// Jenkinsfile
pipeline {
    agent {
        docker {
            image 'gauge-test-runner:latest'
            args '-v /tmp:/tmp --shm-size=2g'
        }
    }
    
    environment {
        GAUGE_HOME = "${WORKSPACE}/.gauge"
        BASE_URL   = credentials('staging-base-url')
        HEADLESS   = 'true'
    }
    
    options {
        timeout(time: 45, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '10'))
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Build') {
            steps {
                sh 'mvn compile test-compile -q'
            }
        }
        
        stage('Smoke Tests') {
            steps {
                sh 'gauge run --tags smoke --env ci specs/'
            }
            post {
                always {
                    junit 'reports/xml-report/**/*.xml'
                }
            }
        }
        
        stage('Full Regression') {
            when {
                anyOf {
                    branch 'main'
                    branch 'release/*'
                }
            }
            steps {
                sh 'gauge run --parallel --n 6 --env ci specs/'
            }
            post {
                always {
                    junit 'reports/xml-report/**/*.xml'
                    publishHTML(target: [
                        allowMissing: false,
                        alwaysLinkToLastBuild: true,
                        keepAll: true,
                        reportDir: 'reports/html-report',
                        reportFiles: 'index.html',
                        reportName: 'Gauge Test Report',
                        reportTitles: "Build ${BUILD_NUMBER}"
                    ])
                }
                failure {
                    // Archive screenshots on failure
                    archiveArtifacts artifacts: 'reports/html-report/**/*.png', 
                                     allowEmptyArchive: true
                    
                    emailext(
                        to: 'qa-team@example.com',
                        subject: "Gauge Tests Failed - Build ${BUILD_NUMBER}",
                        body: """
                            Build ${BUILD_NUMBER} failed.
                            Check report: ${BUILD_URL}Gauge_20Test_20Report/
                        """
                    )
                }
            }
        }
    }
    
    post {
        always {
            cleanWs()
        }
    }
}

Jenkins with Matrix (Sharded Execution)

pipeline {
    agent any
    
    stages {
        stage('Test') {
            matrix {
                axes {
                    axis {
                        name 'SHARD'
                        values '1', '2', '3', '4'
                    }
                }
                stages {
                    stage('Run Shard') {
                        agent {
                            docker { image 'gauge-test-runner:latest' }
                        }
                        steps {
                            sh """
                                SPECS=\$(find specs/ -name '*.spec' | sort)
                                TOTAL=\$(echo "\$SPECS" | wc -l)
                                SIZE=\$(( (\$TOTAL + 3) / 4 ))
                                START=\$(( (${SHARD} - 1) * \$SIZE + 1 ))
                                SHARD_SPECS=\$(echo "\$SPECS" | sed -n "\${START},\$(( \$START + \$SIZE - 1 ))p" | tr '\n' ' ')
                                gauge run --env ci \$SHARD_SPECS
                            """
                        }
                        post {
                            always {
                                junit "reports/xml-report/**/*.xml"
                                archiveArtifacts "reports/html-report/**"
                            }
                        }
                    }
                }
            }
        }
    }
}

Docker

Running Gauge in Docker ensures consistent environments and eliminates "works on my machine" problems.

Base Dockerfile

# Dockerfile.gauge-runner
FROM eclipse-temurin:21-jdk-jammy

# Install system dependencies for Chrome
RUN apt-get update && apt-get install -y \
    curl \
    wget \
    unzip \
    gnupg \
    ca-certificates \
    fonts-liberation \
    libasound2 \
    libatk-bridge2.0-0 \
    libatk1.0-0 \
    libcups2 \
    libdbus-1-3 \
    libdrm2 \
    libgbm1 \
    libgtk-3-0 \
    libnspr4 \
    libnss3 \
    libxcomposite1 \
    libxdamage1 \
    libxfixes3 \
    libxkbcommon0 \
    libxrandr2 \
    xdg-utils \
    --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

# Install Google Chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google.list \
    && apt-get update \
    && apt-get install -y google-chrome-stable \
    && rm -rf /var/lib/apt/lists/*

# Install Gauge
RUN curl -SsL https://downloads.gauge.org/stable | sh

# Install Gauge plugins
RUN gauge install java && \
    gauge install html-report && \
    gauge install xml-report && \
    gauge install screenshot

# Set working directory
WORKDIR /app

# Pre-install Maven dependencies (cache layer)
COPY pom.xml .
RUN mvn dependency:go-offline -q

# Copy project files
COPY . .

# Build
RUN mvn compile test-compile -q

# Default command
CMD ["gauge", "run", "--env", "ci", "specs/"]

Build and run:

# Build the image
docker build -f Dockerfile.gauge-runner -t gauge-test-runner:latest .

# Run tests
docker run --rm \
  -e BASE_URL=https://staging.example.com \
  -e HEADLESS=true \
  -v $(pwd)/reports:/app/reports \
  gauge-test-runner:latest

# Run specific tags
docker run --rm \
  -e BASE_URL=https://staging.example.com \
  -v $(pwd)/reports:/app/reports \
  gauge-test-runner:latest \
  gauge run --tags smoke --env ci specs/

Docker Compose for Local Testing

# docker-compose.test.yml
version: '3.8'

services:
  app:
    image: myapp:latest
    ports:
      - "3000:3000"
    environment:
      NODE_ENV: test
      DATABASE_URL: postgres://test:test@db:5432/testdb
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 5s
      timeout: 3s
      retries: 10

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: testdb
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U test"]
      interval: 5s
      timeout: 3s
      retries: 10

  gauge:
    build:
      context: .
      dockerfile: Dockerfile.gauge-runner
    depends_on:
      app:
        condition: service_healthy
    environment:
      BASE_URL: http://app:3000
      API_URL: http://app:3000/api
      HEADLESS: "true"
    volumes:
      - ./reports:/app/reports
    command: gauge run --parallel --n 4 --env ci specs/

Run with:

docker-compose -f docker-compose.test.yml up --exit-code-from gauge

The --exit-code-from gauge flag makes docker-compose up exit with the gauge service's exit code — non-zero if tests failed.

Report Management

HTML Report Configuration

Configure the HTML report via environment variables or env/ci/default.properties:

# env/ci/default.properties
gauge_reports_dir=reports
overwrite_reports=true
GAUGE_HTML_REPORT_THEME_PATH=

The overwrite_reports=true setting means each run replaces the previous report. Set to false (or remove) in environments where you want to retain history — though for CI, upload reports as artifacts and let CI manage retention.

XML Report for JUnit-Compatible Systems

Install the xml-report plugin and it automatically generates JUnit-compatible XML:

gauge install xml-report

The XML output goes to reports/xml-report/. Jenkins, GitHub Actions (via publish-unit-test-result-action), GitLab, and most other CI systems can parse this format.

Aggregating Reports from Parallel Shards

When running sharded tests across multiple CI jobs, you get separate report directories. Merge them in a final job:

# GitHub Actions: merge reports from matrix jobs
  merge-reports:
    needs: test
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Download all shard reports
        uses: actions/download-artifact@v4
        with:
          path: all-reports/
          pattern: gauge-report-shard-*

      - name: Upload combined reports
        uses: actions/upload-artifact@v4
        with:
          name: gauge-combined-report
          path: all-reports/

True HTML report merging requires the Gauge aggregate functionality or a post-processing script — the XML reports are easier to merge since they are plain files that CI systems parse themselves.

Configuring Gauge for CI

CI-Specific Properties

# env/ci/default.properties

# Application URLs
BASE_URL=https://staging.example.com
API_URL=https://api.staging.example.com

# Browser settings
HEADLESS=true
BROWSER=chrome
SCREENSHOT_ON_FAILURE=true

# Timeouts (longer in CI due to slower environments)
PAGE_LOAD_TIMEOUT=30000
ELEMENT_TIMEOUT=15000

# Reporting
gauge_reports_dir=reports
overwrite_reports=true

Gauge Configuration in gauge.properties

# gauge.properties (project root)

# Increase for slower CI environments
gauge_clear_state_level=scenario

# Runner connection timeout (ms)
runner_connection_timeout=300000
plugin_connection_timeout=10000

# Runner request timeout (ms)
runner_request_timeout=300000

Failure Handling Strategies

Retry Configuration

# Retry individual failing scenarios up to 2 times
gauge run --retry-failed 2 --env ci specs/

# Run, then retry all failures once
gauge run --env ci specs/ ; gauge run --only-failures --env ci specs/

Marking Flaky Tests

Tag known-flaky tests and exclude them from the main run, running them separately with retries:

# Main run: exclude flaky tests
gauge run --tags "!flaky" --env ci specs/

# Separate run with retries: only flaky tests
gauge run --tags flaky --retry-failed 3 --env ci specs/

Fail Fast

For smoke test stages where you want to stop on the first failure:

# Stop after first failing scenario
gauge run --fail-safe specs/smoke/

Note: Gauge does not have a native --fail-fast flag for stopping the entire run on first failure. The equivalent is combining small, focused spec runs in CI stages with dependency gates between them.

Complete GitHub Actions Reference

name: Full Test Suite

on:
  schedule:
    - cron: '0 2 * * *'  # Nightly at 2am UTC
  workflow_dispatch:
    inputs:
      env:
        description: 'Test environment'
        required: true
        default: 'staging'
        type: choice
        options: [staging, production]
      tags:
        description: 'Gauge tags filter (empty = all)'
        required: false
        default: ''

jobs:
  gauge:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: 'maven'
      
      - name: Cache Gauge
        uses: actions/cache@v4
        with:
          path: ~/.gauge
          key: gauge-${{ hashFiles('manifest.json') }}
      
      - run: curl -SsL https://downloads.gauge.org/stable | sh
      - run: gauge install java && gauge install html-report && gauge install xml-report
      - uses: browser-actions/setup-chrome@v1
      - run: mvn compile test-compile -q
      
      - name: Run tests
        run: |
          TAGS="${{ github.event.inputs.tags }}"
          TAG_FLAG=""
          if [ -n "$TAGS" ]; then
            TAG_FLAG="--tags $TAGS"
          fi
          gauge run --parallel --n 4 --env ${{ github.event.inputs.env || 'ci' }} $TAG_FLAG specs/
        env:
          BASE_URL: ${{ secrets[format('BASE_URL_{0}', github.event.inputs.env || 'CI')] }}
      
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: gauge-report-${{ github.run_number }}
          path: reports/
          retention-days: 30
      
      - if: always()
        uses: EnricoMi/publish-unit-test-result-action@v2
        with:
          files: reports/xml-report/**/*.xml

Summary

Gauge's CI integration is straightforward once you handle the two main concerns: installing Gauge and its plugins efficiently (use caching), and managing browser dependencies for UI tests. The patterns in this post — GitHub Actions with artifact upload, Jenkins with publishHTML, and Docker Compose for local parity — cover the most common setups. Pick the pattern that matches your stack, layer in environment-specific properties via --env, and you have a production-grade test pipeline.

Read more

Start now free