TestNG Data Providers, Groups, and Listeners: Enterprise Testing Patterns
TestNG's enterprise features go well beyond JUnit's capabilities in specific areas: @DataProvider feeds complex object graphs into tests, group dependencies express "integration tests only after unit tests pass", ITestListener hooks let you build custom dashboards, @Factory creates test class instances dynamically, and XML suites orchestrate multi-module parallel runs. This post covers all of them with production-ready examples.
TestNG was created by Cédric Beust to address limitations in JUnit 3 and 4 — specifically around test configuration, data-driven testing, and parallel execution. While JUnit 5 has closed much of that gap, TestNG still has a stronger story for certain enterprise patterns: explicit test grouping with dependencies, rich listener infrastructure, and XML-based suite orchestration. Teams running Selenium grids, data-intensive API tests, or multi-module enterprise builds often find TestNG a better fit.
This post covers the five capabilities where TestNG distinctly shines: data providers, group management, listener-based reporting, parallel execution, and factory-based test generation.
@DataProvider: Feeding Complex Test Data
TestNG's @DataProvider annotation marks a method as a source of data for test methods. The fundamental difference from JUnit's @ParameterizedTest is that @DataProvider returns Object[][] — a matrix where each row is one test invocation and each column is one parameter. This means you can pass fully constructed objects, not just primitives or strings.
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class PaymentProcessorTest {
@DataProvider(name = "paymentScenarios")
public Object[][] paymentScenarios() {
return new Object[][] {
// { PaymentRequest, expectedStatus, expectedMessage }
{
PaymentRequest.builder()
.amount(new BigDecimal("99.99"))
.currency("USD")
.cardNumber("4111111111111111")
.build(),
PaymentStatus.APPROVED,
"Payment approved"
},
{
PaymentRequest.builder()
.amount(new BigDecimal("0.01"))
.currency("EUR")
.cardNumber("5555555555554444")
.build(),
PaymentStatus.APPROVED,
"Payment approved"
},
{
PaymentRequest.builder()
.amount(new BigDecimal("99999.99"))
.currency("USD")
.cardNumber("4000000000000002") // declined card number
.build(),
PaymentStatus.DECLINED,
"Card declined"
},
{
PaymentRequest.builder()
.amount(new BigDecimal("-1.00"))
.currency("USD")
.cardNumber("4111111111111111")
.build(),
PaymentStatus.ERROR,
"Invalid amount"
}
};
}
@Test(dataProvider = "paymentScenarios")
public void testPaymentProcessing(PaymentRequest request,
PaymentStatus expectedStatus,
String expectedMessage) {
PaymentResult result = processor.process(request);
assertEquals(result.getStatus(), expectedStatus);
assertThat(result.getMessage()).contains(expectedMessage);
}
}Data Providers from External Sources
For large datasets or externally managed test data, data providers can load from files or databases:
@DataProvider(name = "csvPayments", parallel = true)
public Iterator<Object[]> csvPayments() throws Exception {
// Return Iterator for lazy loading — important for large datasets
List<Object[]> rows = new ArrayList<>();
try (CSVReader reader = new CSVReader(new FileReader("test-data/payments.csv"))) {
String[] line;
reader.readNext(); // skip header
while ((line = reader.readNext()) != null) {
rows.add(new Object[] {
new BigDecimal(line[0]), // amount
line[1], // currency
line[2], // card
PaymentStatus.valueOf(line[3]) // expected status
});
}
}
return rows.iterator();
}Note parallel = true on the @DataProvider — this runs each row in a separate thread. Combined with @Test(dataProvider = "csvPayments", threadPoolSize = 4), you get parallel data-driven execution.
Cross-Class Data Providers
Data providers don't have to live in the same class. Reference them with dataProviderClass:
// In TestDataRepository.java
public class TestDataRepository {
@DataProvider(name = "validUsers")
public static Object[][] validUsers() {
return new Object[][] {
{ "admin@example.com", "admin", Role.ADMIN },
{ "user@example.com", "user123", Role.USER },
{ "readonly@example.com", "readonly", Role.VIEWER }
};
}
}
// In UserServiceTest.java
@Test(dataProvider = "validUsers", dataProviderClass = TestDataRepository.class)
public void testUserAuthentication(String email, String password, Role expectedRole) {
AuthResult result = authService.authenticate(email, password);
assertTrue(result.isSuccess());
assertEquals(result.getUser().getRole(), expectedRole);
}Iterator for Large Datasets
When your dataset is too large to hold in memory, return an Iterator:
@DataProvider(name = "millionRecords")
public Iterator<Object[]> largeDataset() {
return new Iterator<Object[]>() {
private final Connection conn = DatabasePool.getConnection();
private final ResultSet rs = conn.prepareStatement(
"SELECT input, expected_output FROM test_vectors"
).executeQuery();
@Override
public boolean hasNext() {
try { return !rs.isLast(); }
catch (SQLException e) { throw new RuntimeException(e); }
}
@Override
public Object[] next() {
try {
rs.next();
return new Object[] {
rs.getString("input"),
rs.getString("expected_output")
};
} catch (SQLException e) { throw new RuntimeException(e); }
}
};
}Test Groups and Group Dependencies
Groups are one of TestNG's most distinctive features. You tag tests with group memberships, then run subsets from XML or command line. More powerfully, you can declare that one group depends on another — meaning if any test in group A fails, all tests in group B are skipped.
public class UserApiTest {
@Test(groups = {"unit", "user"})
public void testUserValidation() {
User user = new User("", "password");
ValidationResult result = validator.validate(user);
assertFalse(result.isValid());
assertTrue(result.getErrors().contains("Email is required"));
}
@Test(groups = {"integration", "user", "api"},
dependsOnGroups = {"unit"})
public void testUserCreationApi() {
// Only runs if ALL unit tests pass
Response response = given()
.contentType("application/json")
.body("""
{"email": "test@example.com", "password": "secure123"}
""")
.when()
.post("/api/users")
.then()
.statusCode(201)
.extract().response();
assertNotNull(response.jsonPath().getString("id"));
}
@Test(groups = {"integration", "user", "api"},
dependsOnGroups = {"unit"},
dependsOnMethods = {"testUserCreationApi"})
public void testUserRetrievalApi() {
// Depends on both the group and the specific creation test
String userId = createdUserId; // stored from previous test
given()
.when()
.get("/api/users/" + userId)
.then()
.statusCode(200)
.body("email", equalTo("test@example.com"));
}
}Method-Level Dependencies
Use dependsOnMethods for sequential workflows where test order matters:
public class CheckoutFlowTest {
private String orderId;
private String paymentId;
@Test
public void addItemToCart() {
CartResponse response = cartService.addItem("product-123", 2);
assertNotNull(response.getCartId());
// store cartId for next test
}
@Test(dependsOnMethods = {"addItemToCart"})
public void proceedToCheckout() {
CheckoutResponse response = checkoutService.initiate(cartId);
orderId = response.getOrderId();
assertNotNull(orderId);
}
@Test(dependsOnMethods = {"proceedToCheckout"})
public void processPayment() {
PaymentResponse response = paymentService.pay(orderId, testCard);
paymentId = response.getPaymentId();
assertEquals(response.getStatus(), "APPROVED");
}
@Test(dependsOnMethods = {"processPayment"})
public void verifyOrderConfirmation() {
Order order = orderService.findById(orderId);
assertEquals(order.getStatus(), OrderStatus.CONFIRMED);
assertEquals(order.getPaymentId(), paymentId);
}
}ITestListener: Custom Reporting Infrastructure
TestNG's listener system is where it truly shines for enterprise use. ITestListener gives you hooks at every significant point in test execution. Build custom reporting, post results to dashboards, trigger alerts, or integrate with your test management system.
import org.testng.*;
public class SlackNotificationListener implements ITestListener, ISuiteListener {
private final SlackClient slack = new SlackClient(System.getenv("SLACK_WEBHOOK_URL"));
private final List<ITestResult> failures = new ArrayList<>();
private long suiteStartTime;
@Override
public void onStart(ISuite suite) {
suiteStartTime = System.currentTimeMillis();
slack.sendMessage(":test_tube: Test suite *" + suite.getName() + "* started");
}
@Override
public void onTestFailure(ITestResult result) {
failures.add(result);
String testName = result.getMethod().getMethodName();
String className = result.getTestClass().getName();
Throwable cause = result.getThrowable();
slack.sendMessage(String.format(
":red_circle: FAILED: `%s#%s`\n```%s```",
className, testName,
cause != null ? cause.getMessage() : "no message"
));
}
@Override
public void onTestSkipped(ITestResult result) {
String reason = "";
if (result.getThrowable() instanceof SkipException se) {
reason = se.getMessage();
}
slack.sendMessage(":yellow_circle: SKIPPED: `" +
result.getMethod().getMethodName() + "` — " + reason);
}
@Override
public void onFinish(ISuite suite) {
long duration = System.currentTimeMillis() - suiteStartTime;
ISuiteResult results = suite.getResults().values().iterator().next();
ITestContext context = results.getTestContext();
int passed = context.getPassedTests().size();
int failed = context.getFailedTests().size();
int skipped = context.getSkippedTests().size();
String status = failed == 0 ? ":white_check_mark:" : ":x:";
slack.sendMessage(String.format(
"%s Suite complete in %.1fs — %d passed, %d failed, %d skipped",
status, duration / 1000.0, passed, failed, skipped
));
}
// ITestListener requires these — default methods in newer TestNG, override if needed
@Override public void onTestStart(ITestResult result) {}
@Override public void onTestSuccess(ITestResult result) {}
@Override public void onTestFailedButWithinSuccessPercentage(ITestResult result) {}
@Override public void onStart(ITestContext context) {}
@Override public void onFinish(ITestContext context) {}
}Writing Results to a Database
For long-running projects, storing test results enables trend analysis:
public class DatabaseResultListener implements ITestListener {
private final TestResultRepository repo = new TestResultRepository();
@Override
public void onTestSuccess(ITestResult result) {
saveResult(result, "PASSED", null);
}
@Override
public void onTestFailure(ITestResult result) {
saveResult(result, "FAILED", result.getThrowable());
}
private void saveResult(ITestResult result, String status, Throwable error) {
TestResultRecord record = TestResultRecord.builder()
.testClass(result.getTestClass().getName())
.testMethod(result.getMethod().getMethodName())
.status(status)
.durationMs(result.getEndMillis() - result.getStartMillis())
.errorMessage(error != null ? error.getMessage() : null)
.stackTrace(error != null ? ExceptionUtils.getStackTrace(error) : null)
.parameters(Arrays.toString(result.getParameters()))
.runId(System.getenv("BUILD_ID"))
.timestamp(Instant.now())
.build();
repo.save(record);
}
}Registering Listeners
Listeners can be registered in three ways:
- In testng.xml (global):
<listeners>
<listener class-name="com.example.SlackNotificationListener"/>
<listener class-name="com.example.DatabaseResultListener"/>
</listeners>- Via @Listeners annotation (class-level):
@Listeners({SlackNotificationListener.class, DatabaseResultListener.class})
public class CriticalPathTest {
// ...
}- Via ServiceLoader — put the class in
META-INF/services/org.testng.ITestNGListener.
Parallel Test Execution with Thread Count
TestNG offers four parallelism modes: methods, classes, tests, and instances. Each controls what unit of work runs in parallel:
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Parallel Suite" parallel="methods" thread-count="8" verbose="1">
<test name="API Tests">
<classes>
<class name="com.example.UserApiTest"/>
<class name="com.example.ProductApiTest"/>
<class name="com.example.OrderApiTest"/>
</classes>
</test>
<!-- Run this test group with different parallelism -->
<test name="Integration Tests" parallel="classes" thread-count="4">
<groups>
<run>
<include name="integration"/>
</run>
</groups>
<classes>
<class name="com.example.DatabaseIntegrationTest"/>
<class name="com.example.CacheIntegrationTest"/>
</classes>
</test>
</suite>With parallel="methods", every @Test method runs in its own thread from the pool. With parallel="classes", each class runs sequentially internally but different classes run in parallel. parallel="instances" is for @Factory-created instances.
Thread-Safe Test Design
When running parallel tests, instance variables are a trap. Use ThreadLocal or make tests stateless:
public class StatefulParallelTest {
// WRONG — shared mutable state between parallel test methods
private String currentUserId;
// RIGHT — ThreadLocal isolates per thread
private static final ThreadLocal<String> currentUserId = new ThreadLocal<>();
@Test
public void createUser() {
String id = userService.create(randomUser());
currentUserId.set(id); // thread-safe
}
@Test(dependsOnMethods = "createUser")
public void verifyUser() {
User user = userService.findById(currentUserId.get());
assertNotNull(user);
}
}@Factory: Dynamic Test Class Instantiation
@Factory lets a method return an array of test object instances. TestNG runs all @Test methods on each instance. This is powerful when you want to run the same test class against multiple configurations:
public class BrowserCompatibilityFactory {
@Factory
public Object[] createTests() {
return new Object[] {
new LoginTest("chrome"),
new LoginTest("firefox"),
new LoginTest("safari"),
new LoginTest("edge")
};
}
}
public class LoginTest {
private final String browser;
private WebDriver driver;
public LoginTest(String browser) {
this.browser = browser;
}
@BeforeClass
public void setupDriver() {
driver = WebDriverFactory.create(browser);
}
@Test
public void loginWithValidCredentials() {
driver.get(BASE_URL + "/login");
driver.findElement(By.id("email")).sendKeys("admin@example.com");
driver.findElement(By.id("password")).sendKeys("password");
driver.findElement(By.id("submit")).click();
assertEquals(driver.getTitle(), "Dashboard");
}
@AfterClass
public void tearDown() {
if (driver != null) driver.quit();
}
}Combined with parallel="instances" in your suite XML, each browser instance runs in its own thread:
<suite name="Browser Suite" parallel="instances" thread-count="4">
<test name="Cross Browser Tests">
<classes>
<class name="com.example.BrowserCompatibilityFactory"/>
</classes>
</test>
</suite>Environment-Based Factory
A more realistic pattern creates instances based on environment configuration:
@Factory
public Object[] createEnvironmentTests() {
String[] environments = System.getProperty("test.environments", "dev").split(",");
return Arrays.stream(environments)
.map(env -> new ApiContractTest(
EnvironmentConfig.load(env)
))
.toArray();
}Run with -Dtest.environments=dev,staging,prod to run contract tests against all three.
TestNG XML Suite Configuration
The full power of TestNG suite configuration comes together in testng.xml. Here's a production-grade example:
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Full Regression" parallel="tests" thread-count="8"
preserve-order="true" verbose="1">
<!-- Global listeners -->
<listeners>
<listener class-name="com.example.DatabaseResultListener"/>
<listener class-name="com.example.SlackNotificationListener"/>
<listener class-name="com.example.RetryAnalyzer"/>
</listeners>
<!-- Parameter available to all tests -->
<parameter name="base-url" value="${BASE_URL}"/>
<parameter name="api-version" value="v2"/>
<!-- Unit tests first, no parallelism needed, fast -->
<test name="Unit Tests" preserve-order="true">
<groups>
<run>
<include name="unit"/>
<exclude name="flaky"/>
</run>
</groups>
<packages>
<package name="com.example.unit"/>
</packages>
</test>
<!-- Integration tests in parallel after unit tests -->
<test name="Integration Tests" parallel="methods" thread-count="4">
<groups>
<dependencies>
<group name="api" depends-on="unit"/>
<group name="database" depends-on="unit"/>
</dependencies>
<run>
<include name="integration"/>
</run>
</groups>
<packages>
<package name="com.example.integration"/>
</packages>
</test>
<!-- E2E tests last, sequential -->
<test name="E2E Tests">
<groups>
<run>
<include name="e2e"/>
</run>
</groups>
<classes>
<class name="com.example.e2e.CheckoutFlowTest"/>
<class name="com.example.e2e.UserRegistrationFlowTest"/>
</classes>
</test>
</suite>Retry Logic via IRetryAnalyzer
For flaky tests (network calls, browser tests), implement retry:
public class RetryAnalyzer implements IRetryAnalyzer {
private static final int MAX_RETRIES = 3;
private int retryCount = 0;
@Override
public boolean retry(ITestResult result) {
if (retryCount < MAX_RETRIES) {
retryCount++;
return true; // retry the test
}
return false; // give up
}
}
// Apply to a test:
@Test(retryAnalyzer = RetryAnalyzer.class)
public void flakyNetworkTest() {
// ...
}For applying retry to all tests without annotating each one, use a listener that sets the analyzer:
public class GlobalRetryListener implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation, Class testClass,
Constructor testConstructor, Method testMethod) {
annotation.setRetryAnalyzer(RetryAnalyzer.class);
}
}Summary
TestNG's enterprise testing patterns solve problems that simpler frameworks leave to you:
- @DataProvider feeds real object graphs into tests, supports external sources via Iterator, and runs in parallel with
parallel = true - Groups and dependencies let you model test contracts: integration tests only pass after unit tests
- ITestListener and ISuiteListener hooks provide production-grade reporting infrastructure without third-party plugins
- Parallel execution modes (
methods,classes,tests,instances) give granular control over concurrency - @Factory generates test class instances dynamically, perfect for cross-browser and cross-environment matrices
- XML suite configuration is the glue that orchestrates all of the above into a single, configurable test run
For teams running Selenium grids, large data-driven test suites, or multi-environment regression runs, these features make TestNG the more expressive choice over out-of-the-box JUnit 5.