TestNG Listeners: Custom Reporting, Retry Logic, and Test Hooks

TestNG Listeners: Custom Reporting, Retry Logic, and Test Hooks

TestNG listeners are the extension point for everything that happens around tests — logging, reporting, screenshots, retries, Slack notifications, custom metrics. The framework fires events at every step of the test lifecycle and your listeners intercept them. This is how you build professional-grade test infrastructure on top of TestNG.

The Listener Interfaces

TestNG provides several listener interfaces. The ones you'll actually use:

Interface Purpose
ITestListener Per-test events: start, pass, fail, skip
ISuiteListener Suite start/finish
IReporter Generate custom reports after the suite
IInvokedMethodListener Before/after every invoked method (including @Before*/@After*)
IRetryAnalyzer Control test retry behavior
IAnnotationTransformer Modify annotations at runtime
IAlterSuiteListener Modify suite XML programmatically before execution

ITestListener

The most commonly implemented interface. Intercepts test-level events:

public class TestEventListener implements ITestListener {

    @Override
    public void onStart(ITestContext context) {
        System.out.println("=== Test run starting: " + context.getName() + " ===");
        System.out.println("Start time: " + new Date(context.getStartDate().getTime()));
    }

    @Override
    public void onFinish(ITestContext context) {
        int passed  = context.getPassedTests().size();
        int failed  = context.getFailedTests().size();
        int skipped = context.getSkippedTests().size();

        System.out.println("\n=== Test run complete ===");
        System.out.printf("Passed: %d | Failed: %d | Skipped: %d%n",
                          passed, failed, skipped);
    }

    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("[START] " + getTestName(result));
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        long duration = result.getEndMillis() - result.getStartMillis();
        System.out.printf("[PASS]  %s (%dms)%n", getTestName(result), duration);
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("[FAIL]  " + getTestName(result));
        System.out.println("        " + result.getThrowable().getMessage());
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        System.out.println("[SKIP]  " + getTestName(result));
        if (result.getThrowable() != null) {
            System.out.println("        Reason: " + result.getThrowable().getMessage());
        }
    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
        // Called when test fails but invocationCount success % is still acceptable
        System.out.println("[PARTIAL] " + getTestName(result));
    }

    private String getTestName(ITestResult result) {
        return result.getTestClass().getName() + "." + result.getName();
    }
}

Accessing Test Parameters in Listeners

@Override
public void onTestStart(ITestResult result) {
    Object[] params = result.getParameters();
    if (params != null && params.length > 0) {
        System.out.println("[START] " + result.getName() +
                           " with params: " + Arrays.toString(params));
    }
}

Accessing the ITestContext

ITestContext gives you suite-level data from inside test-level callbacks:

@Override
public void onTestFailure(ITestResult result) {
    ITestContext context = result.getTestContext();
    String suiteName  = context.getSuite().getName();
    String testName   = context.getName();
    String browser    = context.getCurrentXmlTest().getParameter("browser");

    System.out.printf("[FAIL] Suite=%s Test=%s Browser=%s Method=%s%n",
                      suiteName, testName, browser, result.getName());
}

Screenshot Listener for WebDriver

The most common real-world listener: capture a screenshot on failure.

public class ScreenshotListener implements ITestListener {

    private static final String SCREENSHOT_DIR = "target/screenshots/";

    @Override
    public void onTestFailure(ITestResult result) {
        WebDriver driver = getDriverFromTest(result);
        if (driver == null) return;

        String fileName = result.getName() + "_" +
                          new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date()) +
                          ".png";
        String filePath = SCREENSHOT_DIR + fileName;

        try {
            Files.createDirectories(Path.of(SCREENSHOT_DIR));
            File screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
            Files.copy(screenshot.toPath(), Path.of(filePath));
            System.out.println("Screenshot saved: " + filePath);
        } catch (IOException e) {
            System.err.println("Failed to save screenshot: " + e.getMessage());
        }
    }

    private WebDriver getDriverFromTest(ITestResult result) {
        // Get driver from your DriverManager or test instance
        Object testInstance = result.getInstance();
        if (testInstance instanceof BaseTest) {
            return DriverManager.getDriver();
        }
        return null;
    }
}

ISuiteListener

Fires at suite start and finish. Use it for environment setup, teardown, or notifications:

public class SuiteEventListener implements ISuiteListener {

    @Override
    public void onStart(ISuite suite) {
        System.out.println("Suite starting: " + suite.getName());

        // Read suite-level parameters
        String env = suite.getParameter("environment");
        System.out.println("Target environment: " + env);

        // Example: notify team that test run started
        notifySlack("Test suite '" + suite.getName() + "' started on " + env);
    }

    @Override
    public void onFinish(ISuite suite) {
        SuiteResultCounts counts = getSuiteCounts(suite);
        String message = String.format(
            "Suite '%s' complete. Passed: %d, Failed: %d, Skipped: %d",
            suite.getName(),
            counts.passed,
            counts.failed,
            counts.skipped
        );
        System.out.println(message);
        notifySlack(message);
    }

    private SuiteResultCounts getSuiteCounts(ISuite suite) {
        int passed = 0, failed = 0, skipped = 0;
        for (ISuiteResult result : suite.getResults().values()) {
            ITestContext context = result.getTestContext();
            passed  += context.getPassedTests().getAllResults().size();
            failed  += context.getFailedTests().getAllResults().size();
            skipped += context.getSkippedTests().getAllResults().size();
        }
        return new SuiteResultCounts(passed, failed, skipped);
    }

    private void notifySlack(String message) {
        // Your Slack notification code here
    }
}

IReporter — Custom HTML/JSON Reports

IReporter fires once after the entire suite finishes. You get full access to all results and can generate any output format:

public class JsonReporter implements IReporter {

    @Override
    public void generateReport(List<XmlSuite> xmlSuites,
                               List<ISuite> suites,
                               String outputDirectory) {
        List<Map<String, Object>> results = new ArrayList<>();

        for (ISuite suite : suites) {
            for (ISuiteResult suiteResult : suite.getResults().values()) {
                ITestContext context = suiteResult.getTestContext();
                collectResults(results, context.getPassedTests(),  "PASSED");
                collectResults(results, context.getFailedTests(),  "FAILED");
                collectResults(results, context.getSkippedTests(), "SKIPPED");
            }
        }

        writeJson(results, outputDirectory + "/test-results.json");
    }

    private void collectResults(List<Map<String, Object>> results,
                                IResultMap resultMap,
                                String status) {
        for (ITestResult result : resultMap.getAllResults()) {
            Map<String, Object> entry = new LinkedHashMap<>();
            entry.put("name",       result.getName());
            entry.put("class",      result.getTestClass().getName());
            entry.put("status",     status);
            entry.put("duration",   result.getEndMillis() - result.getStartMillis());
            entry.put("parameters", Arrays.toString(result.getParameters()));
            if (result.getThrowable() != null) {
                entry.put("error", result.getThrowable().getMessage());
            }
            results.add(entry);
        }
    }

    private void writeJson(List<Map<String, Object>> results, String path) {
        try {
            ObjectMapper mapper = new ObjectMapper();
            mapper.writerWithDefaultPrettyPrinter()
                  .writeValue(new File(path), results);
            System.out.println("JSON report written to: " + path);
        } catch (IOException e) {
            System.err.println("Failed to write JSON report: " + e.getMessage());
        }
    }
}

IInvokedMethodListener

Fires before and after every invoked method — including @BeforeMethod, @AfterMethod, and @Test. More granular than ITestListener:

public class MethodTimingListener implements IInvokedMethodListener {

    private final Map<String, Long> startTimes = new ConcurrentHashMap<>();

    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult result) {
        String key = getKey(method, result);
        startTimes.put(key, System.currentTimeMillis());

        if (method.isTestMethod()) {
            System.out.println("Invoking test: " + result.getName());
        }
    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult result) {
        String key = getKey(method, result);
        Long start = startTimes.remove(key);
        if (start != null) {
            long duration = System.currentTimeMillis() - start;
            if (method.isTestMethod() && duration > 5000) {
                System.out.printf("SLOW TEST: %s took %dms%n",
                                  result.getName(), duration);
            }
        }
    }

    private String getKey(IInvokedMethod method, ITestResult result) {
        return result.getName() + "_" + method.getTestMethod().getMethodName() +
               "_" + Thread.currentThread().getId();
    }
}

IRetryAnalyzer — Retry Flaky Tests

The retry analyzer lets you retry failed tests automatically:

public class RetryAnalyzer implements IRetryAnalyzer {
    private static final int MAX_RETRY_COUNT = 2;
    private int retryCount = 0;

    @Override
    public boolean retry(ITestResult result) {
        if (retryCount < MAX_RETRY_COUNT) {
            retryCount++;
            System.out.printf("Retrying test '%s' — attempt %d of %d%n",
                              result.getName(), retryCount, MAX_RETRY_COUNT);
            return true;
        }
        return false;
    }
}

Apply to a single test:

@Test(retryAnalyzer = RetryAnalyzer.class)
public void flakyTest() {
    // Will retry up to 2 times on failure
}

Apply globally using IAnnotationTransformer — so you don't have to add it to every test:

public class GlobalRetryTransformer implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation annotation,
                          Class testClass,
                          Constructor testConstructor,
                          Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

Register it in testng.xml:

<suite name="Suite">
    <listeners>
        <listener class-name="com.example.listeners.GlobalRetryTransformer"/>
    </listeners>
</suite>

Now every test automatically gets retry behavior without any annotation change.

Excluding Retried Results from Reports

By default, the retry failure shows up in your test results even when the retry passes. Use a listener to filter them:

public class RetryCleanupListener implements ITestListener {

    @Override
    public void onFinish(ITestContext context) {
        // Remove failed results that eventually passed on retry
        Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
        Set<ITestResult> passedTests = context.getPassedTests().getAllResults();

        Set<String> passedMethodNames = passedTests.stream()
            .map(ITestResult::getName)
            .collect(Collectors.toSet());

        failedTests.removeIf(result -> passedMethodNames.contains(result.getName()));
    }
}

IAnnotationTransformer for Runtime Configuration

Disable tests at runtime based on environment:

public class EnvironmentFilter implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation annotation,
                          Class testClass,
                          Constructor testConstructor,
                          Method testMethod) {
        if (testMethod == null) return;

        // Disable tests marked with @ProductionOnly when not on production
        ProductionOnly prodAnnotation = testMethod.getAnnotation(ProductionOnly.class);
        if (prodAnnotation != null && !isProduction()) {
            annotation.setEnabled(false);
        }
    }

    private boolean isProduction() {
        return "production".equals(System.getProperty("env"));
    }
}

Custom annotation:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface ProductionOnly {}

Usage:

@Test
@ProductionOnly
public void paymentGatewayLive() {
    // Only runs when -Denv=production
}

Registering Listeners

Three ways to register listeners, in order of scope:

1. On a test class (narrowest scope)

@Listeners({ScreenshotListener.class, TestEventListener.class})
public class LoginTests {
    // Listeners apply only to this class
}

2. In testng.xml (suite scope)

<suite name="Suite">
    <listeners>
        <listener class-name="com.example.listeners.ScreenshotListener"/>
        <listener class-name="com.example.listeners.TestEventListener"/>
        <listener class-name="com.example.listeners.JsonReporter"/>
        <listener class-name="com.example.listeners.GlobalRetryTransformer"/>
    </listeners>
</suite>

3. Via ServiceLoader (automatic discovery)

Create the file src/test/resources/META-INF/services/org.testng.ITestNGListener:

com.example.listeners.ScreenshotListener
com.example.listeners.TestEventListener

TestNG picks these up automatically without any XML or annotation configuration. Best for shared listener libraries.

Building a Complete Listener Stack

A production test suite typically needs:

GlobalRetryTransformer    — retries flaky tests
EnvironmentFilter         — enables/disables tests per environment
TestEventListener         — structured logging
ScreenshotListener        — screenshots on failure
SuiteEventListener        — Slack/webhook notifications
JsonReporter              — machine-readable results for CI

Each listener does one thing. Stack them in testng.xml and they compose cleanly. Avoid putting multiple concerns in a single listener class — it becomes unmaintainable fast.

The ITestResult object passed to most listener methods contains everything: the test instance, parameters, start/end times, exception, and thread ID. When you need to extract data from the running test (like the WebDriver instance), access it through result.getInstance() cast to your base class, then call your accessor method.

Read more

Start now free