Pairwise Testing Tools: PICT, AllPairs, and Combinatorial Test Design

Knowing the theory of pairwise testing is one thing. Actually generating test suites is another. The good news: the tooling is mature, free, and takes under an hour to set up. The less-good news: each tool has quirks, and the documentation ranges from sparse to nonexistent in places.

This is a hands-on guide to the main combinatorial testing tools — PICT, AllPairs, and ACTS — with real examples, constraint syntax, and practical notes on integrating them into a CI/CD pipeline.

The Tool Landscape

Before diving in, a quick map of the options:

For most teams, PICT is the right choice. It's fast, supports N-wise coverage up to any t-value, handles constraints elegantly, runs everywhere, and outputs formats that slot directly into test frameworks. AllPairs is a useful fallback if you need quick Python integration or a simpler setup. ACTS is worth knowing about for research contexts or if you need detailed coverage metrics.

PICT: The Workhorse

PICT (Pairwise Independent Combinatorial Testing) is a command-line tool from Microsoft Research, now open source on GitHub. It implements a variant of the IPOG (In-Parameter-Order-General) algorithm, which generates near-optimal test suites quickly even for large parameter spaces.

Installation

macOS:

brew install pict

Windows: Download the binary from https://github.com/microsoft/pict/releases

Linux (build from source):

git clone https://github.com/microsoft/pict.git
cd pict
make
sudo make install

Basic Usage

PICT reads a model file that defines parameters and their values. Create a file called model.txt:

Browser: Chrome, Firefox, Safari, Edge
OS: Windows, macOS, Linux
Network: Fast, Slow, Offline
AuthMethod: Password, SSO, MagicLink
UserType: Free, Pro, Admin

Run PICT:

pict model.txt

Output (tab-separated):

Browser	OS	Network	AuthMethod	UserType
Chrome	Windows	Fast	Password	Free
Chrome	macOS	Slow	SSO	Pro
Chrome	Linux	Offline	MagicLink	Admin
Firefox	Windows	Slow	MagicLink	Pro
Firefox	macOS	Offline	Password	Admin
Firefox	Linux	Fast	SSO	Free
Safari	Windows	Offline	SSO	Admin
Safari	macOS	Fast	MagicLink	Free
Safari	Linux	Slow	Password	Pro
Edge	Windows	Fast	MagicLink	Admin
Edge	macOS	Offline	Password	Free
Edge	Linux	Slow	SSO	Pro
...

For 5 parameters with values (4, 3, 3, 3, 3), full coverage would be 324 tests. PICT generates roughly 20 tests for 2-way coverage.

Controlling Coverage Level

By default, PICT generates 2-way (all-pairs) coverage. Use the /o flag to specify a different t-value:

pict model.txt /o:3    # 3-way coverage
pict model.txt /o:4    # 4-way coverage
pict model.txt /o:1    # 1-way (sanity check)

For 3-way coverage on the same 5-parameter model, you'll get approximately 50–60 tests instead of 20.

Output Formats

PICT outputs tab-separated values by default. Common transformations:

CSV:

pict model.txt | tr '\t' ',' > test-cases.csv

JSON (using jq + a small script):

pict model.txt > /tmp/pict_out.txt
python3 -c "
import csv, json, sys
with open('/tmp/pict_out.txt') as f:
    reader = csv.DictReader(f, delimiter='\t')
    print(json.dumps(list(reader), indent=2))
"

Constraints: The Killer Feature

Real parameter spaces have invalid combinations. PICT handles this with a constraint syntax appended to the model file, separated by IF/THEN/ELSE and NOT/AND/OR logic.

Browser: Chrome, Firefox, Safari, Edge
OS: Windows, macOS, Linux
AuthMethod: Password, SSO, MagicLink

# Constraints
IF [Browser] = "Safari" THEN [OS] <> "Windows";
IF [AuthMethod] = "SSO" THEN [Browser] <> "MagicLink";

Wait — that second constraint has a type error. Let's be precise:

# Safari doesn't run natively on Windows
IF [Browser] = "Safari" THEN [OS] <> "Windows";

# Magic Link requires JS, so exclude Offline + MagicLink
IF [Network] = "Offline" THEN [AuthMethod] <> "MagicLink";

PICT will never generate a test case that violates a constraint. This is critical for avoiding meaningless test cases that no QA engineer would run.

Seeding: Pinning Critical Combinations

You can force PICT to include specific test cases using a seed file:

# seed.txt
Browser	OS	AuthMethod
Safari	macOS	SSO
Edge	Windows	Password

pict model.txt /e:seed.txt

PICT guarantees these combinations appear in the output, then fills in the remaining test cases to achieve the specified coverage level. This is how you combine pairwise-generated test cases with manually specified critical scenarios — the best of both worlds.

Statistics

To see coverage statistics without the full test output:

pict model.txt /s

Output shows the number of test cases, the number of parameter combinations covered, and the percentage of all pairs covered (should always be 100% for 2-way). Useful for validating that your model is set up correctly.

AllPairs: Python-Friendly All-Pairs

AllPairs is a Python implementation of the all-pairs algorithm, originally by James Bach and now maintained in various forms. It's strictly 2-way only, but it integrates cleanly into Python test frameworks.

Installation

pip install allpairs

Or use the standalone script from: https://github.com/thombashi/allpairspy

pip install allpairspy

Basic Usage

from allpairspy import AllPairs

parameters = [
    ["Chrome", "Firefox", "Safari", "Edge"],
    ["Windows", "macOS", "Linux"],
    ["Fast", "Slow", "Offline"],
    ["Password", "SSO", "MagicLink"],
    ["Free", "Pro", "Admin"],
]

for i, test_case in enumerate(AllPairs(parameters)):
    print(i, test_case.pairs)

Output is a list of test case objects. Each test case has an index and a list of value indices (or values, depending on the library version).

Integration with pytest

The real power of Python-based all-pairs generation is parametrized testing. Here's a complete pattern:

import pytest
from allpairspy import AllPairs

# Define your parameter space
browsers = ["Chrome", "Firefox", "Safari", "Edge"]
os_list = ["Windows", "macOS", "Linux"]
networks = ["Fast", "Slow", "Offline"]
auth_methods = ["Password", "SSO", "MagicLink"]

parameters = [browsers, os_list, networks, auth_methods]

# Generate all-pairs test cases
test_cases = [
    (browsers[tc[0]], os_list[tc[1]], networks[tc[2]], auth_methods[tc[3]])
    for tc in AllPairs(parameters)
]

@pytest.mark.parametrize("browser,os,network,auth", test_cases)
def test_login(browser, os, network, auth):
    # Your test logic here
    result = login_with_config(browser, os, network, auth)
    assert result.success, f"Login failed: {browser}/{os}/{network}/{auth}"

Running pytest will execute exactly the all-pairs-generated test cases — no more, no less. This integrates cleanly with CI, test reporting, and pytest plugins.

ACTS: Academic-Grade Tool with N-Wise Support

The Automated Combinatorial Testing for Software (ACTS) tool from NIST is the reference implementation used in most academic research on combinatorial testing. It supports t-way coverage for any t, handles constraints, and provides detailed coverage reports.

Getting ACTS

Download from: https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software

It requires Java 8+.

CLI Usage

java -jar acts_cmd.jar -t 3 -i model.txt -o output.txt

Model file format for ACTS differs from PICT:

[System]
Name: LoginSystem

[Parameter]
Browser (enum): Chrome, Firefox, Safari, Edge
OS (enum): Windows, macOS, Linux
Network (enum): Fast, Slow, Offline
AuthMethod (enum): Password, SSO, MagicLink

[Constraint]
Browser != "Safari" || OS != "Windows"

ACTS outputs a test suite with coverage statistics, which is useful for audit trails and documentation.

When to Choose ACTS Over PICT

• You need detailed coverage reports for documentation or compliance
• You're doing research comparing coverage algorithms
• Your team is more comfortable with Java tooling and GUI interfaces
• You need the IPOG-F algorithm specifically (ACTS implements several variants)

For production test automation, PICT is faster and easier to script.

Integrating Pairwise Testing into CI/CD

The goal is to make pairwise test generation a standard step in your pipeline, not a manual one-off activity.

Approach 1: Generate Once, Commit the Output

Generate the test cases locally, commit the CSV/JSON to the repo, and have tests read from the file. Regenerate when parameters change.

# In your dev workflow
pict tests/login-model.txt > tests/login-test-cases.tsv
git add tests/login-test-cases.tsv
git commit -m "Regenerate login test cases after adding Edge browser"

Pros: stable, reproducible, no tool dependency in CI. Cons: test cases can drift from the model file if someone updates the model but forgets to regenerate.

Approach 2: Generate at Test Time

Run PICT as part of the test execution. Works well with PICT as a binary in the repo or installed via a package manager in CI.

# .github/workflows/test.yml
- name: Install PICT
  run: brew install pict

- name: Generate test cases
  run: pict tests/login-model.txt > tests/login-test-cases.tsv

- name: Run tests
  run: pytest tests/

Pros: test cases always reflect the current model. Cons: adds PICT as a CI dependency.

Approach 3: Python Generation Inline

For Python-based test suites, generate inline at pytest collection time:

# conftest.py
from allpairspy import AllPairs

def pytest_configure(config):
    """Generate test cases at collection time."""
    pass  # AllPairs is called directly in parametrize decorators

This requires no external tools and works anywhere Python runs.

Makefile Integration

A practical Makefile target:

.PHONY: generate-test-cases
generate-test-cases:
    pict tests/login-model.txt /s > tests/login-test-cases.tsv
    pict tests/checkout-model.txt /s > tests/checkout-test-cases.tsv
    echo "Test case files regenerated"

test: generate-test-cases
    pytest tests/

Practical Workflow for a New Parameter Space

Here's the workflow from "I have a new feature with multiple parameters" to "I have a test suite":

Step 1: List all parameters and their valid values

Write them down explicitly. Don't assume. Talk to developers about what the code actually branches on. Identify every parameter that might affect behavior differently.

Step 2: Identify constraints

Which combinations are invalid? Which are equivalent (and can be merged)? Document constraints before modeling — it's easy to forget them and end up with PICT generating test cases nobody would run.

Step 3: Create the model file

Write the PICT model file. Start simple, add constraints iteratively. Run pict model.txt /s to see statistics without generating the full output.

Step 4: Choose t-value

Default to 2. Step up to 3 if the feature is high-risk or if you have history of three-way interaction bugs.

Step 5: Generate and review

Run PICT, review the output. Do the test cases make sense? Are there obvious missing cases that fall between pairs? If yes, add them as seeds.

Step 6: Map to test scripts

Map each PICT-generated row to a test case in your framework. Use the tab-separated output as the data source for parameterized tests.

Step 7: Maintain the model file

When parameters change (new browser version, new auth method), update the model file and regenerate. This is the model file's job — capture the parameter space so it can be updated and regenerated.

Common Mistakes

Treating PICT output as sacred. PICT generates a mathematically minimal test suite. It doesn't know about your domain. Always review the output and add critical known combinations manually via seeds.

Forgetting constraints. Unconstrained invalid combinations generate useless test cases. A test case for Safari/Windows is usually a waste of time. Model constraints from the start.

Using wrong t-value for the context. Don't use 4-way for a low-risk feature just because it sounds more thorough. And don't use 2-way for a safety-critical subsystem just because it's smaller.

Regenerating test cases without updating the test scripts. If you add a parameter value and regenerate, new test cases will appear. Make sure your test script handles the new values.

Conflating parameter count with parameter complexity. A parameter with 10 values is more combinatorially significant than 10 parameters with 2 values each. Size up the t-value based on the highest-value-count parameters, not just the total parameter count.

The Test Data Angle

For teams using HelpMeTest's cloud-hosted automation, PICT-generated test data maps directly to parameterized test scenarios. Generate your test cases in CSV, feed them as test data to your automated scenarios, and let the platform execute them across environments. HelpMeTest's usage-based pricing ($0.003/run, no base fee) supports parameterized test runs without requiring you to write test code — you define the scenarios, and the platform handles execution.

Summary

For most teams, the tool choice is simple: PICT for anything serious, AllPairs/allpairspy for Python-native integration. Start with a basic model file, run with default 2-way coverage, add constraints for invalid combinations, and seed critical scenarios. Map the output to parameterized tests and integrate into CI.

The tooling handles the math. Your job is to model the parameter space accurately — because a wrong model produces a test suite that covers the wrong combinations efficiently.