Allure Report Categories and Trends: Classify Failures and Track Progress

What Are Categories?

When an Allure test run shows failures, the default view groups them as "Test defects" or "Product defects" based on whether the test threw an assertion error. That's not enough.

A real failure might be:

• A genuine product bug (assertion failed: expected 200, got 500)
• A flaky test (timeout waiting for element)
• Infrastructure failure (database connection refused)
• Test data problem (user account doesn't exist in this environment)
• A known issue you're tracking

Categories let you map failure patterns to these buckets. The report then shows a pie chart: how many failures are product bugs vs. environment issues vs. test problems.

Configuring categories.json

Create allure-results/categories.json:

[
  {
    "name": "Ignored tests",
    "messageRegex": ".*",
    "matchedStatuses": ["skipped"]
  },
  {
    "name": "Infrastructure problems",
    "messageRegex": ".*Connection refused.*|.*ECONNREFUSED.*|.*database.*not available.*",
    "traceRegex": ".*ConnectionException.*|.*SocketException.*",
    "matchedStatuses": ["broken"]
  },
  {
    "name": "Flaky tests",
    "messageRegex": ".*timeout.*|.*TimeoutException.*|.*StaleElementException.*|.*element not interactable.*",
    "matchedStatuses": ["failed", "broken"]
  },
  {
    "name": "Known product bugs",
    "messageRegex": ".*JIRA-[0-9]+.*",
    "matchedStatuses": ["failed"]
  },
  {
    "name": "Test data issues",
    "messageRegex": ".*404.*|.*not found.*|.*user does not exist.*",
    "matchedStatuses": ["broken"]
  },
  {
    "name": "Product defects",
    "messageRegex": ".*AssertionError.*|.*expected.*but was.*",
    "matchedStatuses": ["failed"],
    "flaky": false
  }
]

Each category has:

• name: What it's called in the report
• messageRegex: Matches against the failure message
• traceRegex: Matches against the stack trace
• matchedStatuses: Which test statuses to match (failed, broken, skipped)
• flaky: Mark this category as flaky tests

Matching Logic

Categories are evaluated in order. The first match wins. A test that matches no category goes to "Product defects" (the default).

Test status:

• failed: Assertion error — test logic detected a problem
• broken: Unexpected exception — the test itself broke (NPE, timeout, connection refused)
• passed: Test passed
• skipped: Test was skipped

Most infrastructure problems land in broken. Most assertion errors land in failed.

Using Categories in Practice

In your test, include the Jira issue key in the assertion message for known bugs:

# Python
assert response.status_code == 200, f"JIRA-4521: Payment gateway returns 503 on Fridays"

// Java
Assertions.assertEquals(200, response.getStatus(), 
    "JIRA-4521: Payment gateway returns 503 on Fridays");

// JavaScript/Cypress
expect(response.status, 'JIRA-4521: Payment gateway returns 503 on Fridays').to.equal(200);

The categories config matches .*JIRA-[0-9]+.* and files it under "Known product bugs" — it won't pollute your "Product defects" count while you wait for the fix.

Setting Up History for Trends

Trend graphs show how pass rates change over time. They require history from previous runs.

Manual History Management

# Step 1: Run tests (generates allure-results/)
pytest tests/ --alluredir=allure-results

# Step 2: Before generating report, copy history from last report
cp -r allure-report/history/ allure-results/history/ 2>/dev/null || true

# Step 3: Generate report (includes history)
allure generate allure-results --clean -o allure-report

# Step 4: Open report — Trends tab now shows historical data
allure open allure-report

Automated History in CI

# GitHub Actions
jobs:
  test:
    steps:
      - uses: actions/checkout@v4

      # Download previous report's history
      - name: Download previous report history
        uses: actions/download-artifact@v4
        continue-on-error: true
        with:
          name: allure-history
          path: allure-results/history

      - name: Run tests
        run: pytest tests/ --alluredir=allure-results

      - name: Generate Allure report
        if: always()
        run: allure generate allure-results --clean -o allure-report

      # Save history for next run
      - name: Upload history for next run
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: allure-history
          path: allure-report/history/

      # Upload full report
      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: allure-report-${{ github.run_number }}
          path: allure-report/

With this setup, each CI run downloads the previous run's history, generates the report with trends, then saves the new history for the next run.

Reading the Trends Tab

The Trends tab shows five charts:

1. Launch statistics trend: Passed/failed/broken/skipped counts per run
2. Test duration trend: How long tests take over time (spots regressions)
3. Categories trend: How failure category distribution changes
4. Retries trend: How many tests needed retries
5. Pass rate: Percentage passing per run

Common patterns:

• Increasing "Infrastructure problems" → your test environment is degrading
• Stable "Product defects" count → bugs aren't getting fixed
• Rising test duration → tests are getting slower, likely due to added waits

Categories vs. Labels

Categories classify why tests fail. Labels classify what tests cover:

• @allure.epic("Payments") — what domain
• @allure.severity(CRITICAL) — how important
• Category: "Infrastructure problem" — why this run failed

They're complementary. A critical payment test that failed due to an infrastructure problem is different from a critical payment test that failed due to a product bug.

Use the Behaviors tab (organized by epic/feature/story) to see what's covered. Use the Categories tab to see why things failed.