Test Management

Xray for Jira: Test Execution & Coverage Reporting

HelpMeTest

HelpMeTest

— 5 min read
Xray for Jira: Test Execution & Coverage Reporting

Xray is the most feature-complete test management app for Jira. Where Zephyr Scale covers the basics well, Xray goes deeper on BDD integration, test coverage reporting, and CI/CD automation. It's the choice for QA-mature teams that need enterprise-grade reporting alongside their Jira workflow.

Xray Issue Types

Xray adds five issue types to your Jira project:

Test: A test case. Steps, preconditions, type (Manual, Generic/Automated, Cucumber/BDD).

Test Set: An unordered collection of tests for reuse. A "Regression Suite" test set contains tests added from across your project.

Test Execution: A scheduled or completed test run. Selects tests from test sets or directly, assigns testers, tracks results per test.

Test Plan: Groups test executions for a version or milestone. Shows aggregate coverage and execution status.

Precondition: Shared setup steps referenced by multiple tests. Define "user is authenticated" once, reference it everywhere.

This hierarchy is more granular than TestRail's (suites → cases → runs) and enables more precise coverage reporting.

Creating Tests

Xray tests are regular Jira issues with extra fields. Create via Create Issue and select issue type Test:

Manual test steps:

Test Type: Manual

Steps:
1. Navigate to /signup
   Data: -
   Expected Result: Registration form loads with name, email, password fields

2. Fill all fields with valid data
   Data: Name: Test User, Email: test+{timestamp}@example.com, Password: Secure!Pass123
   Expected Result: Form accepts input, no validation errors shown

3. Click "Create Account" button
   Data: -
   Expected Result: Confirmation email sent message appears; redirect to /onboarding

4. Open test email inbox (Mailhog or similar)
   Data: -
   Expected Result: Confirmation email received with "Verify your email" link

5. Click verification link
   Data: -
   Expected Result: Browser shows "Email verified" success page; account is active

Cucumber/BDD test:

Feature: User Registration

  Scenario: New user can register with valid credentials
    Given the registration page is open
    When the user fills in all required fields with valid data
    And the user clicks "Create Account"
    Then a confirmation message is displayed
    And a verification email is sent to the provided address

Xray stores the Gherkin scenario in the Jira issue and can export it to .feature files for your test automation framework.

Linking to Requirements

Connect tests to Jira user stories for traceability:

  1. Open the test issue
  2. Add link: Tests (pointing to the story)
  3. Or from the story: Is Tested By (pointing to the test)

Xray's Coverage panel on the story shows:

  • Tests linked to this story
  • Latest execution status per test
  • Overall coverage: how many tests pass, fail, or are unexecuted

This makes sprint reviews concrete — every story shows its test coverage status without pulling a separate report.

Test Executions

Create a test execution when you're ready to run tests:

  1. Create issue → Test Execution
  2. Set: Fix Version (release), Environment (staging, production), Assignee
  3. Add tests: from test sets, manually, or by JQL query
  4. Assign individual tests to testers

During execution, each test transitions through statuses:

  • TODO: Not started
  • EXECUTING: In progress
  • PASS: Verified
  • FAIL: Defect found
  • ABORTED: Cancelled
  • BLOCKED: Dependency issue

For failed tests, link the defect directly from the execution view. Xray tracks the full history: which tester ran it, when, what the result was, which defects were found.

Test Plans and Coverage Reporting

A test plan groups test executions for a version:

v3.0 Test Plan
├── Regression Execution (20/120 passed)
├── Feature A Execution (15/15 passed)
├── Feature B Execution (8/20 passed, 12 blocked)
├── Performance Execution (not started)
└── Mobile Execution (10/30 passed)

The test plan dashboard shows:

  • Execution progress across all child executions
  • Cumulative pass rate
  • Open defects linked from failed tests
  • Coverage by component

Coverage Reports: Xray's coverage analysis shows which Jira issues (stories, epics) have test coverage and which don't. Filter by fix version, component, or label.

The Requirement Coverage report shows a matrix: requirements on one axis, test executions on the other. Each cell shows the test result for that requirement in that execution. This is invaluable for compliance — showing auditors exactly which requirements were tested and what the results were.

CI/CD Integration

Xray's CI integration works through its REST API and official CLI tool. After automated tests run in CI:

# Install Xray CLI
curl -L https://github.com/Xray-App/xray-automation-cli/releases/latest/download/xray-cli-linux -o xray
<span class="hljs-built_in">chmod +x xray

<span class="hljs-comment"># Export Cucumber feature files before test run
./xray export-cucumber \
  --server https://yourcompany.atlassian.net \
  --project PROJ \
  --jql <span class="hljs-string">"issuetype=Test AND labels=cucumber AND status=Ready" \
  --output-dir features/

<span class="hljs-comment"># Run tests (Cypress, Playwright, Cucumber, etc.)
npx cypress run --reporter junit --reporter-options mochaFile=results.xml

<span class="hljs-comment"># Import results back to Xray
./xray import-results \
  --server https://yourcompany.atlassian.net \
  --project PROJ \
  --environment staging \
  --format junit \
  --results results.xml

For Cucumber specifically:

# GitHub Actions
- name: Export feature files from Xray
  run: |
    ./xray export-cucumber \
      --server ${{ vars.JIRA_URL }} \
      --project PROJ \
      --token ${{ secrets.XRAY_TOKEN }} \
      --jql "issuetype=Test AND labels=automation" \
      --output-dir features/

- name: Run Cucumber tests
  run: npx cucumber-js --format json:cucumber-report.json

- name: Import results to Xray
  if: always()
  run: |
    ./xray import-results \
      --server ${{ vars.JIRA_URL }} \
      --project PROJ \
      --token ${{ secrets.XRAY_TOKEN }} \
      --format cucumber \
      --results cucumber-report.json

This round-trip (export feature files → run tests → import results) keeps Xray as the source of truth while automation handles execution.

REST API Direct Integration

For custom integrations without the CLI:

import requests

JIRA_URL = 'https://yourcompany.atlassian.net'
XRAY_TOKEN = 'your-xray-api-key'

def create_test_execution(project_key, fix_version, test_issue_keys):
    response = requests.post(
        f'{JIRA_URL}/rest/raven/1.0/testrun',
        headers={'Authorization': f'Bearer {XRAY_TOKEN}', 'Content-Type': 'application/json'},
        json={
            'fields': {
                'project': {'key': project_key},
                'summary': f'CI Execution - {fix_version}',
                'issuetype': {'name': 'Test Execution'},
                'fixVersions': [{'name': fix_version}]
            },
            'testIssues': test_issue_keys
        }
    )
    return response.json()['key']

def update_test_result(execution_key, test_key, status, defect_key=None):
    payload = {'status': status}  # PASS, FAIL, ABORTED
    if defect_key:
        payload['defects'] = [defect_key]
    
    requests.put(
        f'{JIRA_URL}/rest/raven/1.0/testrun/{execution_key}/test/{test_key}/status',
        headers={'Authorization': f'Bearer {XRAY_TOKEN}'},
        json=payload
    )

Global vs. Project Test Sets

Test sets can be project-scoped or global. Global test sets (Xray Cloud) span multiple projects — useful when a QA team manages tests across a portfolio of Jira projects.

A global "Core Regression" test set includes critical tests from all projects. Create one execution from it before any major release, covering the entire product, regardless of which Jira project each test belongs to.

BDD Workflow in Practice

For teams practicing BDD, Xray's workflow is:

  1. Story refinement: Product writes Acceptance Criteria in Gherkin as Cucumber scenarios
  2. Test creation: Each scenario becomes an Xray Test issue
  3. Automation: Developer implements step definitions
  4. Execution: CI runs the steps, imports results to Xray
  5. Reporting: Product sees pass/fail status on the story in Jira

This closes the loop between product requirements and test results without leaving Jira.

Xray Cloud vs. Xray Data Center

Xray Cloud (for Jira Cloud): Managed by Xray (SmartBear). Different API (GraphQL-based), faster releases, subscription pricing.

Xray Data Center (for Jira Data Center/Server): Self-hosted with Jira. REST API, compatible with Jira Data Center plugins.

The APIs differ significantly. Check which version you're using before implementing integrations — the Cloud GraphQL API is more powerful but requires a different client.

Testing Live Application Behavior

Xray manages test case metadata and tracks historical execution results. For continuous monitoring of your deployed application — verifying it works correctly between test cycles, detecting regressions from infrastructure changes — HelpMeTest runs automated tests against your live app on a schedule. Tests run in plain English without a Jira dependency.

Summary

Xray's strengths over other Jira test management tools:

  • Native BDD: Full Cucumber lifecycle in Jira (export → run → import)
  • Coverage reporting: Requirement coverage matrix, traceability out of the box
  • Test hierarchy: Test → Test Set → Test Execution → Test Plan gives precise control
  • CLI tool: First-class CI integration without custom API code
  • Global test sets: Cross-project regression management

Use Xray if you practice BDD, need compliance-grade traceability reports, or manage testing across multiple Jira projects. For simpler test tracking needs, Zephyr Scale has a lower learning curve.

Read more