Test Reporting

Allure Report Advanced Features: Beyond the Basics

HelpMeTest

HelpMeTest

— 6 min read
Allure Report Advanced Features: Beyond the Basics

Most teams get Allure working in an afternoon: install the plugin, run tests, generate the report. But the basic setup only scratches the surface of what Allure can do.

This guide covers the advanced features that turn Allure from a pretty HTML report into a genuine quality intelligence tool: failure categories, environment properties, history trends, flaky test detection, custom labels, and the Allure TestOps integration for team-scale reporting.

Prerequisites

This guide assumes you have Allure working with your framework (JUnit 5, TestNG, Pytest, or similar). If you haven't set up the basics yet, start with the Allure getting started guide first.

Advanced features require:

  • Allure 2.18+
  • allure-results/ directory from a previous run (for history)
  • Allure CLI or CI plugin for report generation

Failure Categories

Without categories, all failures land in "Failed" or "Broken" buckets. Categories let you classify failures by type — product bugs, test infrastructure issues, flaky network calls — so you can route them to the right owner.

Defining Categories

Create allure-results/categories.json:

[
  {
    "name": "Product Defects",
    "matchedStatuses": ["failed"],
    "messageRegex": ".*AssertionError.*"
  },
  {
    "name": "Test Infrastructure Issues",
    "matchedStatuses": ["broken"],
    "messageRegex": ".*ConnectionRefused.*|.*TimeoutException.*|.*NoSuchElementException.*"
  },
  {
    "name": "API Failures",
    "matchedStatuses": ["failed", "broken"],
    "messageRegex": ".*HTTP 5\\d{2}.*|.*HTTP 4\\d{2}.*"
  },
  {
    "name": "Unknown Failures",
    "matchedStatuses": ["failed", "broken"]
  }
]

Fields:

  • name — category label shown in the report
  • matchedStatuses — failed, broken, passed, skipped
  • messageRegex — match against the failure message
  • traceRegex — match against the stack trace

The last category with no regex acts as a catch-all. Allure matches categories top-to-bottom and assigns the first match.

Viewing Categories

In the generated report, the "Categories" section replaces the default failure list. Each category shows count, affected tests, and a drill-down to individual failures. This immediately answers "how many failures are our fault vs. environment noise?"

Environment Properties

The Environments widget shows metadata about the test run — browser version, base URL, OS, build number. Without this, reports from different environments are indistinguishable.

Create allure-results/environment.properties before generating the report:

Browser=Chrome 122
Browser.Version=122.0.6261.69
OS=Ubuntu 22.04
Base.URL=https://staging.example.com
Build=2024-02-15.42
Java.Version=17.0.9
Environment=Staging

Or in XML format (environment.xml):

<environment>
  <parameter>
    <key>Browser</key>
    <value>Chrome 122</value>
  </parameter>
  <parameter>
    <key>Environment</key>
    <value>Staging</value>
  </parameter>
</environment>

In CI, generate this file dynamically:

cat > allure-results/environment.properties << EOF
Browser=Chrome
Build=${BUILD_NUMBER}
Environment=${DEPLOY_ENV}
Base.URL=${BASE_URL}
EOF

Allure's most powerful feature is historical trend tracking — seeing test stability over time, not just the current run.

How History Works

Allure stores history in allure-report/history/. When you generate a new report, you copy the previous report's history/ directory into the new allure-results/history/ before generating. This chains runs together.

# CI pipeline structure:
# 1. Run tests → produces allure-results/
# 2. Copy history from previous report:
cp -r allure-report/history allure-results/history

# 3. Generate new report:
allure generate allure-results -o allure-report --clean

# 4. Save allure-report/ as artifact for next run

With history enabled, Allure shows:

  • Trend graph — pass/fail/broken/skipped counts over last N runs
  • Duration trend — how long the suite takes run-over-run
  • Retries trend — how many tests are being retried each run
  • Categories trend — how failure categories change over time

These are visible in the "Trend" section of the report sidebar.

Jenkins History Integration

The Allure Jenkins plugin handles history automatically:

post {
    always {
        allure([
            includeProperties: false,
            jdk: '',
            properties: [],
            reportBuildPolicy: 'ALWAYS',
            results: [[path: 'allure-results']]
        ])
    }
}

Jenkins stores the history between builds. No manual file copying needed.

Flaky Test Detection

Allure detects flaky tests when you run tests with retries. A test that fails on the first attempt but passes on retry is marked "Flaky" in the report.

Enabling Retries in JUnit 5

@ExtendWith(AllureJunit5.class)
public class FlakyDetectionTest {
    
    @RepeatedTest(3)
    @DisplayName("Login test with retry")
    void loginTest() {
        // test code
    }
}

For TestNG:

@Test(retryAnalyzer = RetryAnalyzer.class)
public void loginTest() {
    // test code
}

Allure collects all execution attempts. If any attempt passes and any fails, the test is marked flaky. The "Retries" section shows each attempt's status and duration.

The Flaky Tests View

In the report, filter by "Flaky" status to see tests that are inconsistent. Common causes:

  • Timing dependencies (missing explicit waits)
  • Shared test data modified by parallel tests
  • Environment variability (network timeouts)
  • Race conditions in the application under test

Flaky test detection is valuable because flaky tests erode trust in your test suite. When developers see intermittent failures they stop treating red builds as blockers.

Epic, Feature, Story Hierarchy

Allure's behavior hierarchy (Epic → Feature → Story → Test) helps organize large test suites:

@Epic("User Management")
@Feature("Authentication")
@Story("Login")
@Test
public void loginWithValidCredentials() {
    // test
}

@Epic("User Management")
@Feature("Authentication")
@Story("Logout")
@Test
public void logoutClearsSession() {
    // test
}

The "Behaviors" section in the report groups tests by this hierarchy, giving product managers a view organized by feature rather than test class.

Link tests to issue trackers or test management tools:

@Issue("BUG-1234")
@TmsLink("TC-567")
@Test
public void checkoutValidation() {
    // test
}

Configure URL templates in allure.properties:

allure.link.issue.pattern=https://jira.example.com/browse/{}
allure.link.tms.pattern=https://testmo.yourteam.net/run/{}/test/{}

@Issue("BUG-1234") becomes a clickable link to https://jira.example.com/browse/BUG-1234 in the report.

Custom Labels

Add arbitrary metadata to tests:

@LabelAnnotation(name = "component")
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface Component {
    String value();
}

// Usage:
@Component("Checkout")
@Test
public void cartTotal() {
    // test
}

Dynamic Step Logging

Static annotations cover structure; dynamic step logging covers execution detail.

import io.qameta.allure.Allure;
import io.qameta.allure.Step;

public class LoginPage {
    
    @Step("Enter username: {username}")
    public void enterUsername(String username) {
        driver.findElement(By.id("username")).sendKeys(username);
    }
    
    @Step("Click login button")
    public void clickLogin() {
        driver.findElement(By.id("login-btn")).click();
    }
}

The @Step annotation creates a named step in the report with method parameters interpolated. This is cleaner than manual log calls.

For dynamic content (values known only at runtime):

Allure.step("Verify cart total: " + expectedTotal, () -> {
    String actual = cartPage.getTotal();
    assertEquals(expectedTotal, actual);
});

Attaching Files

Attach screenshots, HTML, or JSON to any step:

// Screenshot
Allure.addAttachment("Screenshot", "image/png", 
    new ByteArrayInputStream(screenshotBytes), "png");

// Page HTML
Allure.addAttachment("Page Source", "text/html", 
    driver.getPageSource());

// API Response
Allure.addAttachment("Response Body", "application/json", responseBody);

Attachments appear in the test step timeline and can be downloaded from the report.

Parametrized Tests

Allure handles parametrized tests well. Each parameter combination appears as a separate entry in the report:

@ParameterizedTest
@CsvSource({
    "admin, admin123, true",
    "user, wrong_pass, false",
    "locked_user, pass, false"
})
@Step("Login test: {username} / {password} → expect success: {expectSuccess}")
void loginParametrized(String username, String password, boolean expectSuccess) {
    boolean result = loginPage.login(username, password);
    assertEquals(expectSuccess, result);
}

Each row gets its own report entry with the parameter values shown, making it easy to identify which combination failed.

Allure TestOps Integration

For teams running hundreds of tests across multiple pipelines, the standalone HTML report is a limitation — there's no central place to see trends across all pipelines simultaneously.

Allure TestOps is the SaaS/self-hosted platform that solves this. Key additions over the open-source report:

  • Central results collection — all CI pipelines push results to one place
  • Test case management — auto-discover test cases from code annotations
  • Launch comparison — compare two runs side-by-side
  • Analytics dashboards — team-level quality metrics
  • Flaky test tracking — centralized, not per-report

Integration sends results directly from CI without generating HTML locally:

- name: Send results to Allure TestOps
  run: |
    allurectl upload \
      --endpoint ${{ secrets.ALLURE_ENDPOINT }} \
      --token ${{ secrets.ALLURE_TOKEN }} \
      --project-id ${{ vars.ALLURE_PROJECT_ID }} \
      --launch-name "PR #${{ github.event.number }}" \
      allure-results

For teams with 10+ engineers running CI continuously, TestOps is worth evaluating. For smaller teams, the open-source report with Jenkins/GitHub history is sufficient.

Performance Considerations

Large test suites (5000+ tests) can produce slow Allure reports. Optimization options:

  • Split by module — generate separate reports per module and link them
  • Use --single-file flag — generates one self-contained HTML (slower to open but easier to distribute)
  • Limit history depth — don't chain more than 10-15 runs in history; older data becomes noise
  • Archive old results — store raw allure-results/ in object storage, not in the report itself

Summary

Allure's value grows significantly once you move past the defaults. Categories make failure triage faster. Environment properties make runs reproducible and comparable. History trends transform one-time reports into a quality timeline. Flaky test detection surfaces reliability issues before they become blockers.

These features require a bit of configuration investment — writing a categories.json, wiring up history in your CI pipeline, adding @Epic/@Feature annotations to your tests. Each one is worth the time: they reduce the cognitive overhead of understanding test results and make quality visible to everyone on the team, not just the engineer who wrote the tests.

Read more

Start now free