Robot Framework CI/CD Integration and Test Reporting

09 Sep 2026 — 7 min read

Getting Robot Framework into CI and making the results useful are two different problems. Running RF in CI is straightforward — robot tests/ and done. Making those results actionable, fast, and properly gated takes more work. This post covers both.

What Robot Framework Produces

Every RF execution generates three output files:

output.xml — machine-readable execution data. Everything else is derived from this.
log.html — detailed execution log with keyword-by-keyword traces, screenshots (if captured), timestamps, and pass/fail status for every keyword. This is what you open when debugging a failure.
report.html — high-level summary: test counts, pass rates, elapsed time, tag-based breakdowns.

These files are your primary artifacts. Preserve them in CI — they're the only way to investigate failures without re-running tests.

GitHub Actions Integration

Basic setup:

# .github/workflows/robot-tests.yml
name: Robot Framework Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      
      - name: Cache pip packages
        uses: actions/cache@v3
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Set up Chrome
        uses: browser-actions/setup-chrome@v1
      
      - name: Run Robot Framework tests
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: |
          robot \
            --variable BROWSER:headlesschrome \
            --variable BASE_URL:$BASE_URL \
            --outputdir results/ \
            --xunit xunit-results.xml \
            tests/
        continue-on-error: true  # Don't fail the step — we gate via the next step
      
      - name: Check test results
        run: |
          python -c "
          import xml.etree.ElementTree as ET
          tree = ET.parse('results/output.xml')
          root = tree.getroot()
          stats = root.find('.//statistics/total/stat')
          passed = int(stats.get('pass', 0))
          failed = int(stats.get('fail', 0))
          total = passed + failed
          print(f'Results: {passed}/{total} passed')
          if failed > 0:
              print(f'FAILED: {failed} tests failed')
              exit(1)
          "
      
      - name: Upload test results
        uses: actions/upload-artifact@v3
        if: always()  # Upload even on failure
        with:
          name: robot-test-results
          path: |
            results/log.html
            results/report.html
            results/output.xml
          retention-days: 30
      
      - name: Publish JUnit results
        uses: mikepenz/action-junit-report@v3
        if: always()
        with:
          report_paths: results/xunit-results.xml

The --xunit flag generates JUnit-compatible XML that GitHub's native test reporting can display. Combined with mikepenz/action-junit-report, you get inline test results on the PR.

Multi-environment Strategy

Testing against multiple environments from one workflow:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [staging, production]
        browser: [headlesschrome, headlessfirefox]
      fail-fast: false  # Run all combinations even if one fails
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Run tests
        env:
          BASE_URL: ${{ matrix.environment == 'staging' && secrets.STAGING_URL || secrets.PROD_URL }}
        run: |
          robot \
            --variable BROWSER:${{ matrix.browser }} \
            --variable BASE_URL:$BASE_URL \
            --outputdir results/${{ matrix.environment }}-${{ matrix.browser }}/ \
            --include smoke \
            tests/
      
      - name: Upload results
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: results-${{ matrix.environment }}-${{ matrix.browser }}
          path: results/${{ matrix.environment }}-${{ matrix.browser }}/

Jenkins Integration

For Jenkins, the Robot Framework plugin provides native result visualization:

// Jenkinsfile
pipeline {
    agent any
    
    environment {
        STAGING_URL = credentials('staging-url')
        API_KEY = credentials('api-key')
    }
    
    stages {
        stage('Setup') {
            steps {
                sh 'pip install -r requirements.txt'
            }
        }
        
        stage('Run Tests') {
            steps {
                sh """
                    robot \
                      --variable BROWSER:headlesschrome \
                      --variable BASE_URL:${STAGING_URL} \
                      --outputdir results/ \
                      tests/ || true
                """
            }
        }
        
        stage('Publish Results') {
            steps {
                step([
                    $class: 'RobotPublisher',
                    outputPath: 'results',
                    outputFileName: 'output.xml',
                    reportFileName: 'report.html',
                    logFileName: 'log.html',
                    disableArchiveOutput: false,
                    passThreshold: 90.0,    // Build UNSTABLE below 90%
                    unstableThreshold: 80.0, // Build FAILED below 80%
                    otherFiles: '**/*.png'
                ])
            }
        }
    }
    
    post {
        always {
            archiveArtifacts artifacts: 'results/**', allowEmptyArchive: true
        }
    }
}

The passThreshold and unstableThreshold give you graduated failure modes — don't flip to red on a single flaky test, but do fail the build if the suite is significantly broken.

Understanding log.html vs report.html vs output.xml

log.html is for debugging. It shows:

Every keyword executed, with arguments and return values
Nested keyword call trees (expand to see inner calls)
Timestamps for every step
Embedded screenshots on failure (if configured)
Full error messages and stack traces
PASS/FAIL status at every level

When a test fails in CI, log.html is the first file you open. Download it from the artifacts, open in a browser, find the failed test, expand the keyword tree.

report.html is for status reporting. It shows:

Total pass/fail counts
Tag-based breakdowns (how many smoke tests passed, how many regression tests failed)
Execution time per suite
Trend data if configured

Share report.html with stakeholders. Open log.html yourself.

output.xml is machine-readable source data. It's the input for:

rebot (RF's report regeneration tool)
Allure integration
Custom reporting scripts
Merging results from parallel runs

pabot: Parallel Execution

Robot Framework runs tests serially by default. For large suites, pabot (Parallel Robot Framework Executor) runs test suites in parallel:

pip install robotframework-pabot

Basic parallel execution:

# Run test suites in parallel (one suite per process)
pabot --processes 4 tests/

# Run individual test cases in parallel
pabot --testlevelsplit --processes 8 tests/

# Specify output directory
pabot --processes 4 --outputdir results/ tests/

pabot's key feature: shared resources

When tests run in parallel, they can conflict over shared resources (same user account, same test data). pabot provides a resource allocation system:

# tests/checkout/checkout_test.robot
*** Settings ***
Library    pabot.PabotLib

*** Test Cases ***
Complete Checkout Flow
    Acquire Lock    checkout-user-slot-1
    # Only one parallel process holds this lock at a time
    Login As Test User    checkout1@example.com    password
    # ... test steps ...
    [Teardown]    Release Lock    checkout-user-slot-1

For suite-level setup that should only run once across all parallel processes:

*** Settings ***
Library    pabot.PabotLib

Suite Setup    Run Setup Only Once    Initialize Test Database
Suite Teardown    Run Teardown Only Once    Cleanup Test Database

pabot GitHub Actions:

- name: Run tests in parallel
  run: |
    pabot \
      --processes ${{ runner.cpu_count }} \
      --variable BROWSER:headlesschrome \
      --outputdir results/ \
      tests/

Allure Integration

Allure produces richer reports than RF's built-in output — better for stakeholders, supports history trending, and integrates with Allure TestOps.

pip install allure-robotframework

Generate Allure results alongside RF output:

robot \
  --listener allure_robotframework \
  --outputdir results/ \
  tests/

This creates output/allure directory with Allure-format JSON files. Serve the report:

allure serve output/allure

Or generate static HTML:

allure generate output/allure -o allure-report/ --clean

GitHub Actions with Allure:

- name: Run tests with Allure
  run: |
    robot \
      --listener allure_robotframework \
      --outputdir results/ \
      tests/
  continue-on-error: true

- name: Generate Allure report
  uses: simple-elf/allure-report-action@master
  if: always()
  with:
    allure_results: output/allure
    allure_history: allure-history

- name: Deploy Allure report to GitHub Pages
  uses: peaceiris/actions-gh-pages@v3
  if: always()
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_branch: gh-pages
    publish_dir: allure-history

This publishes Allure reports to GitHub Pages with run-over-run history.

Allure annotations in test files:

*** Settings ***
Library    allure_robotframework

*** Test Cases ***
User Can Complete Purchase
    [Tags]    allure.label:epic:E-commerce    allure.label:feature:Checkout
    [Documentation]    Complete purchase flow from cart to confirmation
    Allure.Step    Add item to cart
    Add Item To Cart    product_id=123
    Allure.Step    Proceed to checkout
    Go To Checkout
    Allure.Step    Complete payment
    Enter Payment Details    ${TEST_CARD}
    Submit Order
    Allure.Step    Verify confirmation
    Order Confirmation Should Be Displayed

Test Result Thresholds and Gating

Don't accept all failures equally. Gate your build on pass rates, not binary pass/fail:

# scripts/check_results.py
import sys
import xml.etree.ElementTree as ET

def check_results(output_xml, pass_threshold=95, unstable_threshold=85):
    tree = ET.parse(output_xml)
    root = tree.getroot()
    
    stats = root.find('.//statistics/total/stat')
    passed = int(stats.get('pass', 0))
    failed = int(stats.get('fail', 0))
    total = passed + failed
    
    if total == 0:
        print("No tests found — failing build")
        return 2
    
    pass_rate = (passed / total) * 100
    print(f"Pass rate: {pass_rate:.1f}% ({passed}/{total})")
    
    if pass_rate >= pass_threshold:
        print("PASS: Above threshold")
        return 0
    elif pass_rate >= unstable_threshold:
        print(f"UNSTABLE: Pass rate below {pass_threshold}%")
        return 1
    else:
        print(f"FAIL: Pass rate below {unstable_threshold}%")
        return 2

if __name__ == '__main__':
    exit(check_results('results/output.xml'))

Tag-specific thresholds — don't let one flaky non-critical test block deployment:

def check_by_tag(output_xml):
    tree = ET.parse(output_xml)
    root = tree.getroot()
    
    results = {}
    for stat in root.findall('.//statistics/tag/stat'):
        tag = stat.get('name')
        passed = int(stat.get('pass', 0))
        failed = int(stat.get('fail', 0))
        results[tag] = {'passed': passed, 'failed': failed}
    
    # Critical tests must all pass
    smoke = results.get('smoke', {})
    if smoke.get('failed', 0) > 0:
        print(f"FAIL: {smoke['failed']} smoke tests failed")
        return False
    
    # Non-critical tests can have some failures
    regression = results.get('regression', {})
    reg_total = regression.get('passed', 0) + regression.get('failed', 0)
    if reg_total > 0:
        reg_pass_rate = regression['passed'] / reg_total
        if reg_pass_rate < 0.90:
            print(f"FAIL: Regression pass rate {reg_pass_rate:.0%} below 90%")
            return False
    
    return True

Merging Parallel Results

pabot produces separate output-N.xml files when running in parallel. Merge them:

# pabot does this automatically, but for manual merges:
rebot --outputdir merged/ output-1.xml output-2.xml output-3.xml

# With custom output names
rebot \
  --outputdir merged/ \
  --output combined.xml \
  --report combined-report.html \
  --log combined-log.html \
  output-*.xml

Slack Notifications

Post results to Slack after CI runs:

- name: Notify Slack on failure
  if: failure()
  uses: 8398a7/action-slack@v3
  with:
    status: ${{ job.status }}
    fields: repo,message,commit,author,action,eventName,ref,workflow
    text: |
      Robot Framework tests failed on ${{ github.ref }}
      Download results: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Or a custom notification with pass/fail counts:

- name: Parse test results
  id: results
  if: always()
  run: |
    python -c "
    import xml.etree.ElementTree as ET
    tree = ET.parse('results/output.xml')
    stats = tree.getroot().find('.//statistics/total/stat')
    print(f'passed={stats.get(\"pass\", 0)}')
    print(f'failed={stats.get(\"fail\", 0)}')
    " >> $GITHUB_OUTPUT

- name: Notify Slack
  if: always()
  run: |
    STATUS="${{ job.status }}"
    PASSED="${{ steps.results.outputs.passed }}"
    FAILED="${{ steps.results.outputs.failed }}"
    curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
      -H 'Content-type: application/json' \
      -d "{
        \"text\": \"Test results: ${PASSED} passed, ${FAILED} failed | Status: ${STATUS}\",
        \"attachments\": [{
          \"color\": \"$([ '$STATUS' = 'success' ] && echo 'good' || echo 'danger')\",
          \"text\": \"<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View full results>\"
        }]
      }"

Rerunning Failed Tests

For flaky test suites, rerun only the failures:

# First run
robot --outputdir results/ tests/

# Rerun failed tests
robot \
  --rerunfailed results/output.xml \
  --outputdir results/rerun/ \
  tests/

# Merge original and rerun results
rebot \
  --outputdir results/final/ \
  --rerunfailedsuites results/output.xml \
  results/output.xml \
  results/rerun/output.xml

The merged output marks a test as passed if it passed in either run. This reduces noise from genuine flakiness while still catching real failures.

In GitHub Actions:

- name: Run tests
  run: robot --outputdir results/ tests/ || true

- name: Rerun failed tests
  run: |
    if [ -f results/output.xml ]; then
      robot \
        --rerunfailed results/output.xml \
        --outputdir results/rerun/ \
        tests/ || true
    fi

- name: Merge results
  run: |
    rebot \
      --outputdir results/final/ \
      results/output.xml \
      results/rerun/output.xml 2>/dev/null || \
    cp -r results/ results/final/

Performance: What Slows Down RF in CI

Browser startup: Each Open Browser call takes 2-4 seconds. Tests that open a new browser for every test case are slow. Use suite-level browser management where tests are stable enough:

*** Settings ***
Suite Setup       Open Test Browser
Suite Teardown    Close Browser
Test Setup        Go To    ${BASE_URL}

Sequential execution: RF's default is single-threaded. A 200-test suite that takes 3 minutes per test = 10 hours. pabot with 8 processes: ~75 minutes. The math is straightforward.

Docker image cold starts: Pre-build a Docker image with all dependencies installed rather than running pip install on every CI run. This alone can save 2-4 minutes per run.

FROM python:3.11-slim
RUN apt-get update && apt-get install -y chromium-driver
COPY requirements.txt .
RUN pip install -r requirements.txt

The combination of pabot + pre-built Docker image + result merging covers the most impactful CI optimizations for Robot Framework suites.