BDD

Living Documentation with Serenity BDD: From Tests to Stakeholder Reports

HelpMeTest

HelpMeTest

— 4 min read
Living Documentation with Serenity BDD: From Tests to Stakeholder Reports

Living documentation solves a specific problem: the gap between what tests prove and what stakeholders can read. Test results are for engineers — pass/fail counts, stack traces, assertion errors. Living documentation is for everyone — narrative descriptions of system behavior, organized by feature, backed by evidence from actual test runs.

Serenity BDD generates living documentation automatically. This guide explains how to configure it, organize your requirements hierarchy, and produce reports that product managers and executives can actually use.

What Living Documentation Means in Serenity

When Serenity runs your tests, it records:

  • The narrative title of each requirement, feature, and scenario
  • Every step executed, in sequence
  • Screenshots at configurable intervals
  • Pass/fail status with full error details
  • Timing for each step and scenario
  • Coverage: which requirements have tests, which don't

It then aggregates this data into an HTML report with a coverage dashboard, requirement-level drill-down, and scenario-level step narratives. The report is self-contained — you can email it, publish it to S3, or post it as a CI artifact.

Requirements Hierarchy

Serenity understands requirements at multiple levels. The default hierarchy:

capabilities
  └── features
        └── stories
              └── scenarios

Map this to your feature file directory structure:

src/test/resources/features/
├── user_management/          ← feature
│   ├── registration.feature  ← story
│   ├── login.feature
│   └── password_reset.feature
├── shopping_cart/
│   ├── add_items.feature
│   ├── checkout.feature
│   └── payments.feature
└── order_management/
    ├── order_tracking.feature
    └── returns.feature

Serenity reads this structure and groups your reports accordingly. The top-level directories become features; the .feature files become stories.

Narrative Annotations

Add narrative context to your feature files with the @Narrative annotation in your runner, or directly in the feature file description:

Feature: User Registration
  In order to access the platform
  As a new visitor
  I want to create an account

  Scenario: Register with valid email
    Given the registration form is open
    When I submit valid registration details
    Then my account should be created
    And I should receive a confirmation email

The "As a / In order to / I want to" block becomes the narrative in Serenity's report — displayed next to coverage statistics for that story.

For JUnit-based tests, use the @Narrative annotation:

@RunWith(SerenityRunner.class)
@Story(UserRegistration.class)
@Narrative(text = {
    "In order to access the platform",
    "As a new visitor",
    "I want to create an account"
})
public class UserRegistrationTest {
    // ...
}

Step-Level Documentation

The @Step annotation on step library methods defines the narrative that appears in reports:

public class RegistrationSteps {

    RegistrationPage registrationPage;

    @Step("Open the registration form at {0}")
    public void openRegistrationForm(String url) {
        registrationPage.open();
    }

    @Step("Fill in the registration form with email '{0}'")
    public void fillRegistrationForm(String email) {
        registrationPage.enterEmail(email);
        registrationPage.enterName("Test User");
        registrationPage.enterPassword("TestPass123!");
    }

    @Step("Submit the registration form")
    public void submitForm() {
        registrationPage.clickSubmit();
    }

    @Step("Verify account confirmation message")
    public void verifyConfirmation() {
        registrationPage.confirmationMessage()
            .shouldContainText("Account created successfully");
    }
}

Each @Step method appears as a labeled row in the report. The {0}, {1} placeholders inject method arguments into the narrative, so reports read like: "Fill in the registration form with email 'alice@example.com'."

Screenshot Configuration

Control when Serenity captures screenshots in serenity.conf:

serenity {
    take.screenshots = AFTER_EACH_STEP  # Maximum documentation
    # Options:
    # FOR_FAILURES        — screenshots only on failure (fastest)
    # BEFORE_AND_AFTER_EACH_STEP — both sides of every step
    # AFTER_EACH_STEP     — after every step (default for documentation)
    # FOR_EACH_ACTION     — on every WebDriver action (very verbose)
}

For living documentation, AFTER_EACH_STEP is the right balance — every step has visual evidence without doubling report size.

Report Customization

Configure report branding and detail in serenity.conf:

serenity {
    project.name = "Acme Platform Test Suite"
    project.version = "2.4.1"

    # Show full step details in report
    reports.show.step.details = true

    # Include manual test results
    test.results.dir = "manual-test-results"

    # Output directory
    output.directory = "target/site/serenity"
}

# Report theme
report {
    customfields {
        environment = "Staging"
        release = "2.4.1"
        team = "QA Platform"
    }
}

Custom fields appear in the report header — useful for stakeholders who need to know which environment and release the report covers.

Generating the Report

# Run tests and generate report in one step
mvn verify

# Generate report from existing test data (re-run without tests)
mvn serenity:aggregate

The report lands in target/site/serenity/index.html. It requires no server — open it in any browser.

Publishing Reports

For continuous documentation, publish after every CI run.

GitHub Actions:

- name: Run tests
  run: mvn verify -Dheadless=true

- name: Upload Serenity reports
  uses: actions/upload-artifact@v3
  if: always()
  with:
    name: serenity-report-${{ github.run_number }}
    path: target/site/serenity/

- name: Deploy to GitHub Pages
  if: github.ref == 'refs/heads/main'
  uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: target/site/serenity

S3 publishing:

aws s3 sync target/site/serenity/ s3://your-docs-bucket/test-reports/latest/ \
  --acl public-read \
  --content-type "text/html"

Share the S3 URL with your product team. No VPN, no login — just the report.

Coverage Reporting

Serenity's coverage report answers: "What percentage of our requirements have automated tests?"

In the index.html overview:

  • Passing — scenarios that passed in the last run
  • Failing — scenarios with failures
  • Pending — scenarios with @Pending tag or undefined steps
  • Ignored — scenarios with @Ignore

For requirements without any tests, Serenity marks them as untested — gaps visible at the feature level. This makes coverage discussions concrete: "We have no tests for the returns workflow" instead of vague estimates.

To mark manual tests as passing (without automation), use result files:

// manual-test-results/returns_happy_path.json
{
  "title": "Manual: Returns — Happy Path",
  "result": "SUCCESS",
  "tags": [{"name": "returns", "type": "feature"}]
}

Serenity merges manual and automated results in the coverage dashboard.

Tagging for Report Organization

Tags create another dimension of organization in reports:

@authentication @regression @sprint-42
Scenario: Login with SSO

In Serenity's report, the Tags view groups scenarios by tag. Useful patterns:

  • @regression / @smoke — test category
  • @sprint-42 — sprint tracking
  • @issue-1234 — link to issue tracker (Serenity auto-links if configured)
  • @manual — marks manual tests

Configure issue tracker links:

serenity {
    issue.tracker.url = "https://jira.example.com/browse/{0}"
}

Any tag formatted like @PROJ-123 becomes a clickable link to your Jira ticket.

Serenity + Continuous Monitoring

Living documentation proves your system worked correctly at the time tests ran. For confidence that it keeps working between test runs and deployments, add continuous monitoring with HelpMeTest. HelpMeTest runs your critical flows every few minutes in production — no Java, no WebDriver, no build pipeline. When something breaks, you know immediately rather than at the next test run.

The combination: Serenity generates living documentation during development and release; HelpMeTest monitors production continuously.

Summary

Serenity BDD's living documentation is the difference between test results that engineers interpret and reports that your entire team can read. The keys: organize feature files into a requirements hierarchy, annotate steps with narrative, configure screenshot capture, and publish HTML reports after every CI run. Done correctly, your Serenity report becomes the authoritative answer to "what does this system do, and how do we know it works?"

Read more

Start now free