Data-Driven Testing

Data-Driven Testing Frameworks: Run One Test Against Hundreds of Inputs

HelpMeTest

HelpMeTest

— 6 min read
Data-Driven Testing Frameworks: Run One Test Against Hundreds of Inputs

Data-driven testing solves a specific problem: when you have one test scenario but many input variations, writing a separate test for each variation is unsustainable. Data-driven testing lets you write the test logic once and run it against a dataset.

This guide covers how data-driven testing works, how the major frameworks implement it, and how to manage test data effectively.

The Core Concept

Data-driven testing separates test logic from test data. The test script defines what to do; a data source defines what values to use.

Without data-driven testing:

def test_login_valid_admin():
    login('admin@example.com', 'adminpass')
    assert_dashboard_visible()

def test_login_valid_user():
    login('user@example.com', 'userpass')
    assert_dashboard_visible()

def test_login_valid_manager():
    login('manager@example.com', 'managerpass')
    assert_dashboard_visible()

With data-driven testing:

@pytest.mark.parametrize('email,password', [
    ('admin@example.com', 'adminpass'),
    ('user@example.com', 'userpass'),
    ('manager@example.com', 'managerpass'),
])
def test_login_valid(email, password):
    login(email, password)
    assert_dashboard_visible()

Same behavior, one test definition. Add more test cases by adding rows to the data source.

When to Use Data-Driven Testing

Data-driven testing is appropriate when:

  • The same test logic needs to run against many input combinations
  • Test data varies but the validation logic is identical
  • You want non-technical stakeholders to add test cases by editing spreadsheets or CSV files
  • You're testing input validation with valid and invalid data boundaries

It's not appropriate when:

  • Each "variant" requires different assertions or different test steps
  • You're testing different features, not the same feature with different inputs
  • The dataset is small (2-3 cases) — just write separate tests

Pytest Parametrize: The Python Standard

Pytest's parametrize decorator is the most common data-driven approach in Python:

# Basic parametrize
@pytest.mark.parametrize('username,password,expected_role', [
    ('admin@test.com', 'pass', 'admin'),
    ('user@test.com', 'pass', 'user'),
    ('manager@test.com', 'pass', 'manager'),
])
def test_login_returns_correct_role(username, password, expected_role):
    response = api_client.post('/auth/login', json={
        'username': username,
        'password': password
    })
    assert response.status_code == 200
    assert response.json()['role'] == expected_role

Loading Data From External Files

For large datasets, load from CSV or JSON:

import csv
import json
import pytest

def load_csv_data(filepath):
    with open(filepath) as f:
        reader = csv.DictReader(f)
        return [row for row in reader]

def load_json_data(filepath):
    with open(filepath) as f:
        return json.load(f)

# tests/data/login_cases.csv:
# email,password,expected_status,expected_role
# valid_admin@test.com,adminpass,200,admin
# invalid@test.com,wrongpass,401,
# locked@test.com,pass,403,

@pytest.mark.parametrize(
    'email,password,expected_status,expected_role',
    [(r['email'], r['password'], int(r['expected_status']), r['expected_role']) 
     for r in load_csv_data('tests/data/login_cases.csv')]
)
def test_login_scenarios(email, password, expected_status, expected_role):
    response = api_client.post('/auth/login', json={
        'email': email, 
        'password': password
    })
    assert response.status_code == expected_status
    if expected_role:
        assert response.json().get('role') == expected_role

Parametrize With IDs

Test IDs in pytest default to numeric indices. Named IDs make failures readable:

@pytest.mark.parametrize('amount,currency,expected', [
    pytest.param(100, 'USD', '$100.00', id='usd-basic'),
    pytest.param(1000, 'EUR', '€1,000.00', id='eur-thousands'),
    pytest.param(0.01, 'USD', '$0.01', id='usd-minimum'),
    pytest.param(-100, 'USD', None, marks=pytest.mark.xfail, id='negative-invalid'),
], )
def test_format_currency(amount, currency, expected):
    result = format_currency(amount, currency)
    assert result == expected

Output: FAILED tests/test_currency.py::test_format_currency[usd-basic] instead of FAILED tests/test_currency.py::test_format_currency[0].

TestNG Data Providers: The Java Standard

TestNG's @DataProvider is the standard for Java data-driven testing:

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class LoginTest {
    
    @DataProvider(name = "loginCredentials")
    public Object[][] loginCredentials() {
        return new Object[][] {
            {"admin@test.com", "adminpass", "admin"},
            {"user@test.com", "userpass", "user"},
            {"manager@test.com", "managerpass", "manager"},
        };
    }
    
    @Test(dataProvider = "loginCredentials")
    public void testLogin(String email, String password, String expectedRole) {
        LoginResponse response = apiClient.login(email, password);
        
        Assert.assertEquals(response.getStatusCode(), 200);
        Assert.assertEquals(response.getRole(), expectedRole);
    }
}

Loading TestNG Data From CSV

import java.io.*;
import java.util.*;

@DataProvider(name = "csvLoginData")
public Object[][] csvLoginData() throws IOException {
    List<Object[]> data = new ArrayList<>();
    
    try (BufferedReader reader = new BufferedReader(
            new FileReader("src/test/resources/login_cases.csv"))) {
        String line;
        boolean firstLine = true;
        
        while ((line = reader.readLine()) != null) {
            if (firstLine) { firstLine = false; continue; } // Skip header
            
            String[] parts = line.split(",");
            data.add(new Object[]{
                parts[0], // email
                parts[1], // password
                Integer.parseInt(parts[2]), // expected status
                parts[3]  // expected role
            });
        }
    }
    
    return data.toArray(new Object[0][]);
}

@Test(dataProvider = "csvLoginData")
public void testLoginScenarios(String email, String password, 
                                int expectedStatus, String expectedRole) {
    LoginResponse response = apiClient.login(email, password);
    Assert.assertEquals(response.getStatusCode(), expectedStatus);
    if (!expectedRole.isEmpty()) {
        Assert.assertEquals(response.getRole(), expectedRole);
    }
}

Parallel Execution in TestNG

TestNG can run data-driven tests in parallel:

@DataProvider(name = "loginCredentials", parallel = true)
public Object[][] loginCredentials() {
    return new Object[][] {
        {"user1@test.com", "pass1"},
        {"user2@test.com", "pass2"},
        {"user3@test.com", "pass3"},
        {"user4@test.com", "pass4"},
    };
}

Combined with @Test(dataProvider = "loginCredentials"), TestNG runs each data row in a separate thread. Ensure your test code is thread-safe.

Playwright Data-Driven Tests

For end-to-end browser tests with Playwright:

import { test, expect } from '@playwright/test';

const loginTestCases = [
  { email: 'admin@test.com', password: 'adminpass', expectedPath: '/admin' },
  { email: 'user@test.com', password: 'userpass', expectedPath: '/dashboard' },
  { email: 'viewer@test.com', password: 'viewerpass', expectedPath: '/dashboard' },
];

for (const { email, password, expectedPath } of loginTestCases) {
  test(`login redirects ${email} to ${expectedPath}`, async ({ page }) => {
    await page.goto('/login');
    await page.fill('[data-testid="email"]', email);
    await page.fill('[data-testid="password"]', password);
    await page.click('[data-testid="submit"]');
    await expect(page).toHaveURL(expectedPath);
  });
}

Loading External Data in Playwright

import { test, expect } from '@playwright/test';
import fs from 'fs';
import path from 'path';

interface FormTestCase {
  input: Record<string, string>;
  expectedErrors: string[];
  shouldSubmit: boolean;
}

const formCases: FormTestCase[] = JSON.parse(
  fs.readFileSync(path.join(__dirname, 'data/form-cases.json'), 'utf-8')
);

for (const testCase of formCases) {
  test(`form validation: ${JSON.stringify(testCase.input)}`, async ({ page }) => {
    await page.goto('/registration');
    
    for (const [field, value] of Object.entries(testCase.input)) {
      await page.fill(`[data-testid="${field}"]`, value);
    }
    
    await page.click('[data-testid="submit"]');
    
    if (testCase.shouldSubmit) {
      await expect(page).toHaveURL('/success');
    } else {
      for (const error of testCase.expectedErrors) {
        await expect(page.locator(`text=${error}`)).toBeVisible();
      }
    }
  });
}

Managing Test Data Files

File Format Selection

CSV: Best for tabular data, easy to edit in Excel/Sheets, readable in diffs. Poor for nested structures.

JSON: Best for complex or nested data structures. Harder to edit manually, good for programmatic generation.

Excel: Non-technical stakeholders can edit. Requires library dependencies to parse, binary format is not diff-friendly.

Inline in code: Best for small datasets (under 10 rows). Easy to maintain alongside the test. No file management overhead.

Rule of thumb: use inline for 1-10 cases, CSV/JSON for 10-100 cases, and consider generated test data for 100+ cases.

Test Data Versioning

Test data files are part of your test suite. Treat them accordingly:

  • Commit data files to version control alongside tests
  • Review changes to data files in code review
  • Don't store sensitive data (real emails, real passwords) in test data files — use fake data

Generating Test Data Programmatically

For large datasets or complex boundary conditions, generate data programmatically:

# conftest.py
import pytest
from faker import Faker

fake = Faker()

def generate_valid_users(count: int):
    return [
        (fake.email(), fake.password(length=12), fake.name())
        for _ in range(count)
    ]

def generate_invalid_emails():
    return [
        ('notanemail', 'Invalid format'),
        ('missing@domain', 'Invalid domain'),
        ('', 'Email required'),
        ('a' * 256 + '@test.com', 'Email too long'),
    ]

@pytest.mark.parametrize('email,expected_error', generate_invalid_emails())
def test_email_validation(email, expected_error):
    response = api_client.post('/users', json={'email': email})
    assert response.status_code == 400
    assert expected_error in response.json()['errors']

Boundary Value Analysis as Data-Driven Tests

Data-driven testing pairs naturally with boundary value analysis:

# Test all significant boundaries for an age field
@pytest.mark.parametrize('age,expected_valid', [
    (-1, False),   # Below minimum
    (0, False),    # Minimum invalid
    (1, True),     # Minimum valid
    (17, True),    # Just below adult threshold
    (18, True),    # Adult threshold
    (120, True),   # Maximum valid
    (121, False),  # Above maximum
    (None, False), # Missing value
])
def test_age_validation(age, expected_valid):
    response = api_client.post('/users', json={'age': age})
    if expected_valid:
        assert response.status_code == 201
    else:
        assert response.status_code == 400

This documents the validation rules as test cases, makes boundary conditions explicit, and ensures regressions in edge cases are caught.

Pitfalls to Avoid

Test data pollution: If test cases modify shared state (database records, user settings), they interfere with each other. Each parametrized case should have independent state.

Opaque failure messages: AssertionError: assert False doesn't tell you which data row failed. Use assertion messages that include the test data.

Too many cases: More test cases means more maintenance. If you have 500 parametrized cases, are you testing 500 meaningful scenarios or just exhaustively covering arbitrary inputs? Quality over quantity.

Data and assertions coupled: When the expected values in your data file embed too much business logic, updating the application requires updating the data file. Keep data files focused on inputs; compute expected outputs in test code when possible.

Summary

Data-driven testing is the right tool when test logic is constant and input variation is high. Pytest parametrize, TestNG DataProvider, and Playwright's for loop pattern are the standard implementations. Manage test data in version control, use appropriate file formats for the data complexity, and generate data programmatically for large boundary value suites. The productivity gain is real — but only if the test data is managed as carefully as the test code.

Read more

Start now free