TestNG Listeners: Custom Reporting, Retry Logic, and Test Hooks
TestNG listeners are the extension point for everything that happens around tests — logging, reporting, screenshots, retries, Slack notifications, custom metrics. The framework fires events at every step of the test lifecycle and your listeners intercept them. This is how you build professional-grade test infrastructure on top of TestNG.
The Listener Interfaces
TestNG provides several listener interfaces. The ones you'll actually use:
| Interface | Purpose |
|---|---|
ITestListener |
Per-test events: start, pass, fail, skip |
ISuiteListener |
Suite start/finish |
IReporter |
Generate custom reports after the suite |
IInvokedMethodListener |
Before/after every invoked method (including @Before*/@After*) |
IRetryAnalyzer |
Control test retry behavior |
IAnnotationTransformer |
Modify annotations at runtime |
IAlterSuiteListener |
Modify suite XML programmatically before execution |
ITestListener
The most commonly implemented interface. Intercepts test-level events:
public class TestEventListener implements ITestListener {
@Override
public void onStart(ITestContext context) {
System.out.println("=== Test run starting: " + context.getName() + " ===");
System.out.println("Start time: " + new Date(context.getStartDate().getTime()));
}
@Override
public void onFinish(ITestContext context) {
int passed = context.getPassedTests().size();
int failed = context.getFailedTests().size();
int skipped = context.getSkippedTests().size();
System.out.println("\n=== Test run complete ===");
System.out.printf("Passed: %d | Failed: %d | Skipped: %d%n",
passed, failed, skipped);
}
@Override
public void onTestStart(ITestResult result) {
System.out.println("[START] " + getTestName(result));
}
@Override
public void onTestSuccess(ITestResult result) {
long duration = result.getEndMillis() - result.getStartMillis();
System.out.printf("[PASS] %s (%dms)%n", getTestName(result), duration);
}
@Override
public void onTestFailure(ITestResult result) {
System.out.println("[FAIL] " + getTestName(result));
System.out.println(" " + result.getThrowable().getMessage());
}
@Override
public void onTestSkipped(ITestResult result) {
System.out.println("[SKIP] " + getTestName(result));
if (result.getThrowable() != null) {
System.out.println(" Reason: " + result.getThrowable().getMessage());
}
}
@Override
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
// Called when test fails but invocationCount success % is still acceptable
System.out.println("[PARTIAL] " + getTestName(result));
}
private String getTestName(ITestResult result) {
return result.getTestClass().getName() + "." + result.getName();
}
}Accessing Test Parameters in Listeners
@Override
public void onTestStart(ITestResult result) {
Object[] params = result.getParameters();
if (params != null && params.length > 0) {
System.out.println("[START] " + result.getName() +
" with params: " + Arrays.toString(params));
}
}Accessing the ITestContext
ITestContext gives you suite-level data from inside test-level callbacks:
@Override
public void onTestFailure(ITestResult result) {
ITestContext context = result.getTestContext();
String suiteName = context.getSuite().getName();
String testName = context.getName();
String browser = context.getCurrentXmlTest().getParameter("browser");
System.out.printf("[FAIL] Suite=%s Test=%s Browser=%s Method=%s%n",
suiteName, testName, browser, result.getName());
}Screenshot Listener for WebDriver
The most common real-world listener: capture a screenshot on failure.
public class ScreenshotListener implements ITestListener {
private static final String SCREENSHOT_DIR = "target/screenshots/";
@Override
public void onTestFailure(ITestResult result) {
WebDriver driver = getDriverFromTest(result);
if (driver == null) return;
String fileName = result.getName() + "_" +
new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date()) +
".png";
String filePath = SCREENSHOT_DIR + fileName;
try {
Files.createDirectories(Path.of(SCREENSHOT_DIR));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
Files.copy(screenshot.toPath(), Path.of(filePath));
System.out.println("Screenshot saved: " + filePath);
} catch (IOException e) {
System.err.println("Failed to save screenshot: " + e.getMessage());
}
}
private WebDriver getDriverFromTest(ITestResult result) {
// Get driver from your DriverManager or test instance
Object testInstance = result.getInstance();
if (testInstance instanceof BaseTest) {
return DriverManager.getDriver();
}
return null;
}
}ISuiteListener
Fires at suite start and finish. Use it for environment setup, teardown, or notifications:
public class SuiteEventListener implements ISuiteListener {
@Override
public void onStart(ISuite suite) {
System.out.println("Suite starting: " + suite.getName());
// Read suite-level parameters
String env = suite.getParameter("environment");
System.out.println("Target environment: " + env);
// Example: notify team that test run started
notifySlack("Test suite '" + suite.getName() + "' started on " + env);
}
@Override
public void onFinish(ISuite suite) {
SuiteResultCounts counts = getSuiteCounts(suite);
String message = String.format(
"Suite '%s' complete. Passed: %d, Failed: %d, Skipped: %d",
suite.getName(),
counts.passed,
counts.failed,
counts.skipped
);
System.out.println(message);
notifySlack(message);
}
private SuiteResultCounts getSuiteCounts(ISuite suite) {
int passed = 0, failed = 0, skipped = 0;
for (ISuiteResult result : suite.getResults().values()) {
ITestContext context = result.getTestContext();
passed += context.getPassedTests().getAllResults().size();
failed += context.getFailedTests().getAllResults().size();
skipped += context.getSkippedTests().getAllResults().size();
}
return new SuiteResultCounts(passed, failed, skipped);
}
private void notifySlack(String message) {
// Your Slack notification code here
}
}IReporter — Custom HTML/JSON Reports
IReporter fires once after the entire suite finishes. You get full access to all results and can generate any output format:
public class JsonReporter implements IReporter {
@Override
public void generateReport(List<XmlSuite> xmlSuites,
List<ISuite> suites,
String outputDirectory) {
List<Map<String, Object>> results = new ArrayList<>();
for (ISuite suite : suites) {
for (ISuiteResult suiteResult : suite.getResults().values()) {
ITestContext context = suiteResult.getTestContext();
collectResults(results, context.getPassedTests(), "PASSED");
collectResults(results, context.getFailedTests(), "FAILED");
collectResults(results, context.getSkippedTests(), "SKIPPED");
}
}
writeJson(results, outputDirectory + "/test-results.json");
}
private void collectResults(List<Map<String, Object>> results,
IResultMap resultMap,
String status) {
for (ITestResult result : resultMap.getAllResults()) {
Map<String, Object> entry = new LinkedHashMap<>();
entry.put("name", result.getName());
entry.put("class", result.getTestClass().getName());
entry.put("status", status);
entry.put("duration", result.getEndMillis() - result.getStartMillis());
entry.put("parameters", Arrays.toString(result.getParameters()));
if (result.getThrowable() != null) {
entry.put("error", result.getThrowable().getMessage());
}
results.add(entry);
}
}
private void writeJson(List<Map<String, Object>> results, String path) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.writerWithDefaultPrettyPrinter()
.writeValue(new File(path), results);
System.out.println("JSON report written to: " + path);
} catch (IOException e) {
System.err.println("Failed to write JSON report: " + e.getMessage());
}
}
}IInvokedMethodListener
Fires before and after every invoked method — including @BeforeMethod, @AfterMethod, and @Test. More granular than ITestListener:
public class MethodTimingListener implements IInvokedMethodListener {
private final Map<String, Long> startTimes = new ConcurrentHashMap<>();
@Override
public void beforeInvocation(IInvokedMethod method, ITestResult result) {
String key = getKey(method, result);
startTimes.put(key, System.currentTimeMillis());
if (method.isTestMethod()) {
System.out.println("Invoking test: " + result.getName());
}
}
@Override
public void afterInvocation(IInvokedMethod method, ITestResult result) {
String key = getKey(method, result);
Long start = startTimes.remove(key);
if (start != null) {
long duration = System.currentTimeMillis() - start;
if (method.isTestMethod() && duration > 5000) {
System.out.printf("SLOW TEST: %s took %dms%n",
result.getName(), duration);
}
}
}
private String getKey(IInvokedMethod method, ITestResult result) {
return result.getName() + "_" + method.getTestMethod().getMethodName() +
"_" + Thread.currentThread().getId();
}
}IRetryAnalyzer — Retry Flaky Tests
The retry analyzer lets you retry failed tests automatically:
public class RetryAnalyzer implements IRetryAnalyzer {
private static final int MAX_RETRY_COUNT = 2;
private int retryCount = 0;
@Override
public boolean retry(ITestResult result) {
if (retryCount < MAX_RETRY_COUNT) {
retryCount++;
System.out.printf("Retrying test '%s' — attempt %d of %d%n",
result.getName(), retryCount, MAX_RETRY_COUNT);
return true;
}
return false;
}
}Apply to a single test:
@Test(retryAnalyzer = RetryAnalyzer.class)
public void flakyTest() {
// Will retry up to 2 times on failure
}Apply globally using IAnnotationTransformer — so you don't have to add it to every test:
public class GlobalRetryTransformer implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation,
Class testClass,
Constructor testConstructor,
Method testMethod) {
annotation.setRetryAnalyzer(RetryAnalyzer.class);
}
}Register it in testng.xml:
<suite name="Suite">
<listeners>
<listener class-name="com.example.listeners.GlobalRetryTransformer"/>
</listeners>
</suite>Now every test automatically gets retry behavior without any annotation change.
Excluding Retried Results from Reports
By default, the retry failure shows up in your test results even when the retry passes. Use a listener to filter them:
public class RetryCleanupListener implements ITestListener {
@Override
public void onFinish(ITestContext context) {
// Remove failed results that eventually passed on retry
Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
Set<ITestResult> passedTests = context.getPassedTests().getAllResults();
Set<String> passedMethodNames = passedTests.stream()
.map(ITestResult::getName)
.collect(Collectors.toSet());
failedTests.removeIf(result -> passedMethodNames.contains(result.getName()));
}
}IAnnotationTransformer for Runtime Configuration
Disable tests at runtime based on environment:
public class EnvironmentFilter implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation,
Class testClass,
Constructor testConstructor,
Method testMethod) {
if (testMethod == null) return;
// Disable tests marked with @ProductionOnly when not on production
ProductionOnly prodAnnotation = testMethod.getAnnotation(ProductionOnly.class);
if (prodAnnotation != null && !isProduction()) {
annotation.setEnabled(false);
}
}
private boolean isProduction() {
return "production".equals(System.getProperty("env"));
}
}Custom annotation:
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface ProductionOnly {}Usage:
@Test
@ProductionOnly
public void paymentGatewayLive() {
// Only runs when -Denv=production
}Registering Listeners
Three ways to register listeners, in order of scope:
1. On a test class (narrowest scope)
@Listeners({ScreenshotListener.class, TestEventListener.class})
public class LoginTests {
// Listeners apply only to this class
}2. In testng.xml (suite scope)
<suite name="Suite">
<listeners>
<listener class-name="com.example.listeners.ScreenshotListener"/>
<listener class-name="com.example.listeners.TestEventListener"/>
<listener class-name="com.example.listeners.JsonReporter"/>
<listener class-name="com.example.listeners.GlobalRetryTransformer"/>
</listeners>
</suite>3. Via ServiceLoader (automatic discovery)
Create the file src/test/resources/META-INF/services/org.testng.ITestNGListener:
com.example.listeners.ScreenshotListener
com.example.listeners.TestEventListenerTestNG picks these up automatically without any XML or annotation configuration. Best for shared listener libraries.
Building a Complete Listener Stack
A production test suite typically needs:
GlobalRetryTransformer — retries flaky tests
EnvironmentFilter — enables/disables tests per environment
TestEventListener — structured logging
ScreenshotListener — screenshots on failure
SuiteEventListener — Slack/webhook notifications
JsonReporter — machine-readable results for CIEach listener does one thing. Stack them in testng.xml and they compose cleanly. Avoid putting multiple concerns in a single listener class — it becomes unmaintainable fast.
The ITestResult object passed to most listener methods contains everything: the test instance, parameters, start/end times, exception, and thread ID. When you need to extract data from the running test (like the WebDriver instance), access it through result.getInstance() cast to your base class, then call your accessor method.