HPC and MPI Parallel Code Testing: Strategies for Distributed Scientific Software
Testing parallel scientific code is one of the hardest problems in software engineering. The bugs are non-deterministic, the failures are often silent (wrong answers instead of crashes), and the test infrastructure itself requires a parallel runtime. This guide covers the full range of testing strategies for MPI-based HPC code, from unit tests that run on a single rank to scaling validation that exercises hundreds of processes.
The Core Challenge: What Can Go Wrong
MPI programs fail in ways that serial code never does:
- Wrong answers without crashes — a domain decomposition bug produces plausible-looking results that are subtly wrong at partition boundaries
- Deadlocks — blocking send/receive pairs that never complete because the matching operation is on a different code path
- Race conditions — non-collective operations that depend on message ordering
- Seed correlation — parallel RNG streams that are correlated because they share the same seed
- Halo exchange errors — ghost cell regions are not updated correctly, causing boundary artifacts
- Rank-dependent failures — a bug that only manifests on rank 0, or only on the last rank
The implication for testing: you cannot test MPI code purely in serial. You need a parallel test runner.
Setting Up: pytest with mpi4py
The mpi4py library provides Python bindings for MPI. The pytest-mpi plugin (or a custom conftest) lets you run pytest under mpirun.
pip install mpi4py pytest pytest-mpiThe minimal test execution pattern:
mpirun -n 4 python -m pytest tests/test_parallel.py -vFor SLURM environments:
srun --ntasks=4 --mpi=pmi2 python -m pytest tests/test_parallel.pyA critical conftest.py pattern — all ranks must participate in collective operations, including test collection:
# conftest.py
from mpi4py import MPI
import pytest
def pytest_configure(config):
"""Ensure all ranks initialize MPI before any test runs."""
comm = MPI.COMM_WORLD
config._mpi_rank = comm.Get_rank()
config._mpi_size = comm.Get_size()
@pytest.fixture
def comm():
return MPI.COMM_WORLD
@pytest.fixture
def rank(comm):
return comm.Get_rank()
@pytest.fixture
def size(comm):
return comm.Get_size()
def pytest_runtest_call(item):
"""Synchronize all ranks before each test."""
MPI.COMM_WORLD.Barrier()
def pytest_runtest_teardown(item, nextitem):
"""Synchronize after each test to prevent interference."""
MPI.COMM_WORLD.Barrier()Writing MPI Unit Tests
The pattern: every rank runs the test function, but assertions can be rank-specific or global via reduction.
# tests/test_parallel.py
import numpy as np
import pytest
from mpi4py import MPI
def test_scatter_gather_roundtrip(comm, rank, size):
"""Data scattered and gathered should match the original."""
root = 0
n_per_rank = 10
if rank == root:
original = np.arange(size * n_per_rank, dtype=np.float64)
else:
original = None
# Scatter: root sends chunks to each rank
local = np.zeros(n_per_rank, dtype=np.float64)
comm.Scatter(original, local, root=root)
# Each rank squares its chunk
local_result = local ** 2
# Gather back to root
if rank == root:
gathered = np.zeros(size * n_per_rank, dtype=np.float64)
else:
gathered = None
comm.Gather(local_result, gathered, root=root)
if rank == root:
expected = original ** 2
np.testing.assert_array_equal(gathered, expected)
def test_allreduce_sum(comm, rank, size):
"""Allreduce sum should equal the sum of all rank contributions."""
local_value = np.array([float(rank + 1)], dtype=np.float64)
global_sum = np.zeros(1, dtype=np.float64)
comm.Allreduce(local_value, global_sum, op=MPI.SUM)
expected = sum(range(1, size + 1))
np.testing.assert_allclose(global_sum[0], expected)
def test_halo_exchange_1d(comm, rank, size):
"""
1D halo exchange: each rank shares boundary cells with neighbors.
After exchange, each rank's ghost cells should contain neighbor data.
"""
n_local = 20
# Each rank fills its domain with its rank value
domain = np.full(n_local, float(rank), dtype=np.float64)
# Left and right ghost cells
left_ghost = np.zeros(1, dtype=np.float64)
right_ghost = np.zeros(1, dtype=np.float64)
left_rank = (rank - 1) % size
right_rank = (rank + 1) % size
# Send right boundary, receive from left
comm.Sendrecv(
sendbuf=domain[-1:], dest=right_rank, sendtag=0,
recvbuf=left_ghost, source=left_rank, recvtag=0,
)
# Send left boundary, receive from right
comm.Sendrecv(
sendbuf=domain[:1], dest=left_rank, sendtag=1,
recvbuf=right_ghost, source=right_rank, recvtag=1,
)
# Verify ghost cells contain the correct neighbor values
expected_left = float(left_rank)
expected_right = float(right_rank)
assert left_ghost[0] == expected_left, (
f"Rank {rank}: left ghost={left_ghost[0]}, expected={expected_left}"
)
assert right_ghost[0] == expected_right, (
f"Rank {rank}: right ghost={right_ghost[0]}, expected={expected_right}"
)Domain Decomposition Correctness Testing
Domain decomposition bugs are the most common source of wrong-but-not-crashing results in HPC codes. The key test strategy: compare the parallel result against a serial reference on the full domain.
def parallel_laplacian(local_domain: np.ndarray, dx: float, comm) -> np.ndarray:
"""Finite-difference Laplacian with halo exchange."""
rank = comm.Get_rank()
size = comm.Get_size()
n = len(local_domain)
# Halo exchange
left_ghost = np.zeros(1)
right_ghost = np.zeros(1)
left_rank = (rank - 1) % size
right_rank = (rank + 1) % size
comm.Sendrecv(local_domain[-1:], right_rank, 0, left_ghost, left_rank, 0)
comm.Sendrecv(local_domain[:1], left_rank, 1, right_ghost, right_rank, 1)
# Extend with ghost cells
extended = np.concatenate([left_ghost, local_domain, right_ghost])
# Interior Laplacian
result = np.zeros(n)
for i in range(n):
result[i] = (extended[i] - 2 * extended[i+1] + extended[i+2]) / dx**2
return result
def test_parallel_laplacian_matches_serial(comm, rank, size):
"""
Parallel Laplacian on decomposed domain must match serial Laplacian
on the full domain.
"""
n_total = size * 100
dx = 1.0 / n_total
# Each rank owns a slice
n_local = n_total // size
start = rank * n_local
# Smooth test function: sin(pi * x)
x_global = np.linspace(0, 1, n_total, endpoint=False)
x_local = x_global[start:start + n_local]
local_domain = np.sin(np.pi * x_local)
# Parallel result
local_laplacian = parallel_laplacian(local_domain, dx, comm)
# Gather to rank 0 for comparison
all_laplacian = comm.gather(local_laplacian, root=0)
if rank == 0:
parallel_result = np.concatenate(all_laplacian)
# Serial reference
full_domain = np.sin(np.pi * x_global)
serial_result = np.zeros(n_total)
for i in range(1, n_total - 1):
serial_result[i] = (full_domain[i-1] - 2*full_domain[i] + full_domain[i+1]) / dx**2
# Compare interior points only (boundary handling differs)
np.testing.assert_allclose(
parallel_result[1:-1],
serial_result[1:-1],
rtol=1e-10,
err_msg="Parallel Laplacian does not match serial reference"
)Strong and Weak Scaling Validation
Scaling tests verify that your parallelization is actually working. They belong in your CI as nightly tests, not in every PR build.
import time
import subprocess
import json
def run_timed_benchmark(n_procs: int, problem_size: int) -> float:
"""Run the solver benchmark as a subprocess and return wall time."""
result = subprocess.run(
["mpirun", "-n", str(n_procs), "python", "benchmark_solver.py",
"--size", str(problem_size), "--output-json"],
capture_output=True, text=True, timeout=300
)
data = json.loads(result.stdout)
return data["wall_time"]
def test_strong_scaling_efficiency():
"""
Strong scaling: fixed problem size, increasing ranks.
Efficiency = T(1) / (N * T(N)) should be > 0.7 (70% efficiency).
"""
problem_size = 10000
t1 = run_timed_benchmark(n_procs=1, problem_size=problem_size)
t4 = run_timed_benchmark(n_procs=4, problem_size=problem_size)
t8 = run_timed_benchmark(n_procs=8, problem_size=problem_size)
efficiency_4 = t1 / (4 * t4)
efficiency_8 = t1 / (8 * t8)
assert efficiency_4 > 0.7, f"4-rank efficiency too low: {efficiency_4:.2%}"
assert efficiency_8 > 0.6, f"8-rank efficiency too low: {efficiency_8:.2%}"
def test_weak_scaling_efficiency():
"""
Weak scaling: problem size grows with rank count.
T(N) / T(1) should be < 1.3 (30% overhead tolerance).
"""
base_size = 1000
t1 = run_timed_benchmark(n_procs=1, problem_size=base_size)
t4 = run_timed_benchmark(n_procs=4, problem_size=base_size * 4)
t8 = run_timed_benchmark(n_procs=8, problem_size=base_size * 8)
overhead_4 = t4 / t1
overhead_8 = t8 / t1
assert overhead_4 < 1.3, f"4-rank weak scaling overhead too high: {overhead_4:.2f}x"
assert overhead_8 < 1.5, f"8-rank weak scaling overhead too high: {overhead_8:.2f}x"Race Condition Detection with Valgrind Helgrind
Valgrind's Helgrind tool detects data races in multithreaded code. For OpenMP regions inside MPI ranks, it is invaluable.
# Run with Helgrind on a single MPI rank
valgrind --tool=helgrind \
--error-exitcode=1 \
python my_omp_test.py 2>&1 | tee helgrind_report.txt
# Full MPI run — each rank gets its own Valgrind instance
mpirun -n 4 valgrind \
--tool=helgrind \
--log-file=helgrind_rank%q{OMPI_COMM_WORLD_RANK}.txt \
--error-exitcode=1 \
python my_solver.pyIn CI, parse the Helgrind output for Possible data race:
#!/bin/bash
# check_helgrind.sh
for logfile in helgrind_rank*.txt; do
if grep -q "ERROR SUMMARY: 0 errors" "$logfile"; then
echo "PASS: $logfile"
else
echo "FAIL: $logfile — race conditions detected"
grep -A5 "Possible data race" "$logfile"
exit 1
fi
doneMUST: MPI Correctness Checking
MUST (Marmot Umpire Scalable Tool) is a dynamic MPI correctness checker. It detects:
- Deadlocks and potential deadlocks
- Type mismatches between send and receive
- Invalid MPI object usage (using a freed communicator)
- Buffer aliasing violations
- Mismatched collective operations
# Install MUST (via Spack)
spack install must
# Wrap your mpirun invocation
mustrun -n 4 python my_solver.py
# MUST generates a detailed HTML report in MUST_Output/MUST works by instrumenting MPI calls at runtime and building a happens-before graph to detect violations. Unlike Valgrind, it understands MPI semantics natively.
For CI integration, check the MUST output for [MUST-REPORT] error:
mustrun -n 4 python my_solver.py 2>&1 | grep -c "\[MUST-REPORT\] error" && exit 1 || exit 0Testing RNG Parallelization: Seed Correlation
A common bug: each MPI rank seeds the RNG with the same seed, producing correlated streams. This invalidates Monte Carlo results silently.
def test_parallel_rng_streams_are_independent(comm, rank, size):
"""
Each rank must have an independent RNG stream.
Cross-rank correlation should be near zero.
"""
# Correct pattern: unique seed per rank
rng = np.random.default_rng(seed=42 + rank)
local_samples = rng.standard_normal(10000)
# Gather samples from all ranks to rank 0
all_samples = comm.gather(local_samples, root=0)
if rank == 0 and size > 1:
for i in range(size):
for j in range(i + 1, size):
correlation = np.corrcoef(all_samples[i], all_samples[j])[0, 1]
assert abs(correlation) < 0.05, (
f"Ranks {i} and {j} have correlated RNG streams: r={correlation:.4f}. "
f"Check that each rank uses a unique seed."
)
def test_parallel_rng_wrong_pattern_is_correlated(comm, rank, size):
"""
Document the failure mode: same seed on all ranks produces correlation.
This test is expected to detect the anti-pattern.
"""
# WRONG: all ranks use the same seed
rng = np.random.default_rng(seed=42)
local_samples = rng.standard_normal(10000)
all_samples = comm.gather(local_samples, root=0)
if rank == 0 and size > 1:
correlation = np.corrcoef(all_samples[0], all_samples[1])[0, 1]
# When all ranks use the same seed, correlation is exactly 1.0
assert abs(correlation) > 0.99, (
"Expected full correlation with same seed on all ranks"
)CI Integration
Structure your CI to run serial tests on every commit and parallel tests nightly or on HPC hardware:
# .github/workflows/test.yml
jobs:
serial-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements-test.txt
- run: pytest tests/test_unit.py tests/test_serial.py -v
parallel-tests:
runs-on: [self-hosted, hpc]
if: github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'run-parallel')
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements-test.txt
- run: mpirun -n 4 python -m pytest tests/test_parallel.py -v
- run: mpirun -n 8 python -m pytest tests/test_parallel.py -vThe label run-parallel lets you trigger parallel tests on a PR when you've changed the communication layer.
Summary
Testing MPI code requires accepting that parallelism introduces qualitatively new failure modes. The strategy:
- pytest + mpi4py: run tests under
mpirun, synchronize ranks withBarrier()between tests - Serial reference comparison: the canonical way to verify domain decomposition correctness
- Strong/weak scaling tests: nightly benchmarks that catch parallelization regressions
- Valgrind Helgrind: detect data races in OpenMP regions
- MUST: detect MPI protocol violations, deadlocks, type mismatches
- RNG independence tests: verify that parallel streams are not correlated
The biggest investment is the infrastructure: a conftest.py that handles MPI lifecycle correctly, and CI runners with the MPI runtime installed. Once that exists, writing the individual tests follows the same patterns as serial code.