Katalon TestOps: Test Management, Reporting, and CI/CD Integration Guide

Katalon TestOps: Test Management, Reporting, and CI/CD Integration Guide

Katalon TestOps is the cloud-based management layer that sits on top of Katalon Studio. Where Studio is your test authoring environment, TestOps handles everything that happens after you write tests: scheduling, execution history, cross-team reporting, release readiness tracking, and integration with your CI/CD pipeline.

If you're using Katalon Studio and wondering whether TestOps adds enough value to justify the subscription cost — this guide breaks down what it actually does.

What Is Katalon TestOps?

TestOps is a SaaS platform (cloud-hosted at testops.katalon.io) that connects to your Katalon Studio workspace. It provides:

  • Centralized test execution history across projects and teams
  • Scheduled test runs (cron-based, without CI/CD configuration)
  • Test analytics: pass/fail trends, flakiness detection, coverage tracking
  • Release management: map test results to releases/builds
  • Integration with Jira for requirement traceability
  • Integration with CI/CD systems (Jenkins, GitHub Actions, Azure DevOps, CircleCI)
  • Agent management for distributed execution

It's not a replacement for your CI pipeline — it's a reporting and orchestration layer on top of it.

Setting Up TestOps

Creating a Project

  1. Sign in at testops.katalon.io
  2. Create an organization and a project
  3. Get your API key from Settings → API Keys

Linking Katalon Studio to TestOps

In Katalon Studio:

  1. Katalon Studio → Preferences → Katalon TestOps
  2. Enter your API key and select your project
  3. Test results will now sync automatically after each local execution

Installing a TestOps Agent for Remote Execution

For scheduled runs and CI/CD execution, you need a TestOps Agent — a lightweight process running on a machine that can execute tests.

Download the agent from TestOps → Configuration → Agent Setup:

# Linux/macOS
java -jar katalon-agent.jar start \
  --api-key YOUR_API_KEY \
  --server-url https://testops.katalon.io \
  --agent-name "my-linux-agent"

The agent registers with TestOps and waits for job assignments. You can run multiple agents for parallel execution across different machines.

Test Scheduling

TestOps lets you schedule test suites to run automatically without configuring a CI/CD job.

Creating a Schedule

  1. Navigate to your project → Test Executions → Schedules
  2. Click Create Schedule
  3. Select:
    • Test Suite or Test Suite Collection to run
    • Agent to run it on
    • Cron expression (e.g., 0 9 * * 1-5 for 9am weekdays)
    • Katalon version to use
    • Test environment variables

The scheduled job runs on your registered agent, results sync back to TestOps automatically.

When Scheduling Is Useful

Scheduled execution in TestOps is most useful for:

  • Overnight regression runs without CI/CD setup overhead
  • Smoke tests on a fixed cadence independent of deployments
  • Teams that haven't set up a full CI/CD pipeline yet

If you already have Jenkins or GitHub Actions, you'll likely prefer running Katalon from there and pushing results to TestOps for reporting.

Execution Reporting

The reporting layer is where TestOps earns its keep. Raw test execution output is hard to act on — TestOps processes it into something more useful.

Execution Dashboard

The main dashboard shows:

  • Pass/Fail/Error/Skipped counts per execution
  • Duration trends — are tests getting slower?
  • Failure patterns — same tests failing repeatedly (flakiness signals)
  • Environment breakdown — results by browser, OS, device

Test Case History

Drill into any test case to see its history:

  • Each execution result with timestamps
  • Duration per run (performance regression detection)
  • Screenshots and logs for failed runs
  • Flakiness score — percentage of inconsistent results over recent runs

The flakiness detection is genuinely useful. A test that passes 70% of the time isn't testing anything reliably — TestOps flags it so you can investigate and fix it.

Failed Test Analysis

TestOps groups failures by:

  • Error message — clusters tests that fail with the same exception
  • Test object — clusters tests failing on the same UI element (locator issue?)
  • Duration spike — tests that suddenly take much longer (timeout signal)

This grouping saves significant time during failure triage. Instead of reading 50 individual error logs, you see "23 tests failed with NoSuchElementException on btn_checkout" and know exactly where to look.

CI/CD Integration

Jenkins

Install the Katalon Studio Plugin for Jenkins from the Jenkins plugin marketplace.

Pipeline configuration:

pipeline {
    agent any
    stages {
        stage('Test') {
            steps {
                katalon(
                    version: '9.x.x',
                    usePreInstalledKatalon: false,
                    args: '-runMode=console -projectPath="${WORKSPACE}/MyProject.prj" -testSuiteCollectionPath="Test Suites/Regression" -browserType="Chrome (headless)" -apiKey="${KATALON_API_KEY}"'
                )
            }
        }
    }
    post {
        always {
            junit 'Reports/**/*.xml'
        }
    }
}

GitHub Actions

name: Katalon Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Katalon Tests
        uses: katalon-studio/katalon-studio-github-action@v3
        with:
          version: '9.x.x'
          project-path: ${{ github.workspace }}
          args: '-testSuiteCollectionPath="Test Suites/Regression" -browserType="Chrome (headless)" -apiKey=${{ secrets.KATALON_API_KEY }}'

Pushing Results to TestOps from CI

Add --report-folder and the TestOps integration flags to ensure CI results appear in TestOps:

katalonc \
  -noSplash \
  -runMode=console \
  -projectPath="/path/to/project.prj" \
  -testSuiteCollectionPath="Test Suites/Regression" \
  -browserType="Chrome (headless)" \
  -apiKey="YOUR_API_KEY" \
  -orgID="YOUR_ORG_ID" \
  -serverUrl="https://testops.katalon.io"

Results sync to TestOps automatically — execution appears in the dashboard alongside scheduled and manual runs.

Release Management

TestOps includes a Release concept for tracking test coverage against software releases.

Creating a Release

  1. Planning → Releases → New Release
  2. Name it (e.g., "v2.3.0") and set a target date
  3. Link test cases to this release as "required for release"

Tracking Release Readiness

Once test cases are linked to a release, TestOps shows:

  • How many linked tests have been executed
  • Pass/fail status of the last execution for each
  • Percentage of release requirements covered

This is useful for engineering managers and QA leads who need to answer "are we ready to ship?" without manually reviewing hundreds of test results.

Jira Integration

Link TestOps to Jira to close the loop between requirements and test coverage:

  1. TestOps → Configurations → Integrations → Jira
  2. Enter your Jira URL, username, and API token
  3. Map Jira issue types to TestOps test requirements

Once connected, TestOps can import Jira issues as test requirements and track which test cases cover each requirement. Requirements with no linked tests show as uncovered — coverage gaps surface automatically.

TestOps vs. TestRail / Zephyr

If you're evaluating TestOps against dedicated test management tools:

Feature Katalon TestOps TestRail Zephyr Scale
Test execution scheduling ✅ ❌ ❌
Katalon Studio integration Native Plugin Plugin
Flakiness detection ✅ ❌ Limited
CI/CD integration ✅ ✅ ✅
Requirement traceability Jira-dependent Built-in Built-in
Manual test management Limited Strong Strong
Cost Included in Katalon Platform Separate cost Separate cost

TestOps makes the most sense if you're already using Katalon Studio — the native integration is seamless. For teams using other test frameworks or needing strong manual test management, TestRail or Zephyr Scale may be more appropriate.

Practical Tips

Don't create a separate TestOps agent for every machine. Use one agent per execution environment type (e.g., one for Linux headless, one for Windows Chrome). More agents = more scheduling options; more maintenance too.

Use tags to filter results. Tag test cases with @smoke, @regression, @api etc. — TestOps lets you filter results and reports by tag, making it easy to see "how did the smoke suite do?" vs. the full regression.

Set up Slack notifications. TestOps → Configuration → Integrations → Slack. Point it at a #qa-alerts channel. Overnight regressions that fail silently are worse than failures you know about immediately.

Clean up old agents. Agents that go offline but aren't removed show as available for scheduling. This causes jobs to queue indefinitely waiting for an unreachable agent.


Want test execution and monitoring without managing an execution infrastructure? HelpMeTest runs tests on schedule with built-in monitoring, alerting, and execution history — no agent setup required.

Read more

Start now free