Property-Based Testing vs Example-Based Testing: When to Use Each

When developers first encounter property-based testing, the most common question is: "Does this replace my existing tests?" The answer is no — but understanding why requires a clear picture of what each approach does well and where each one falls short.

Example-based testing and property-based testing are complementary tools. Used together, they cover a much wider space than either can alone. This guide explains both approaches, compares them directly, shows what example-based tests miss with concrete examples, and gives you a framework for deciding when to use each.

Example-Based Testing: The Baseline

Example-based testing is what most developers mean when they say "unit testing." You write specific inputs and assert specific outputs.

def test_add():
    assert add(2, 3) == 5
    assert add(0, 0) == 0
    assert add(-1, 1) == 0
    assert add(100, -200) == -100

This approach has real strengths:

• Readable as documentation. A test named test_add_two_positive_integers tells you exactly what behavior it specifies.
• Fast to write. You pick a representative case, write the assertion, move on.
• Easy to debug. When a test fails, you have a specific input/output pair to inspect.
• Great for known edge cases. If you know zero is tricky, you write a test for zero.

The weakness: you only test the cases you think of. Your imagination is the limit of your coverage.

Property-Based Testing: Specifying Invariants

Property-based testing shifts the question from "what does my function do for input X?" to "what is always true about my function's output?"

from hypothesis import given
from hypothesis import strategies as st

@given(st.integers(), st.integers())
def test_add_is_commutative(a, b):
    assert add(a, b) == add(b, a)

@given(st.integers(), st.integers(), st.integers())
def test_add_is_associative(a, b, c):
    assert add(add(a, b), c) == add(a, add(b, c))

@given(st.integers())
def test_add_zero_is_identity(n):
    assert add(n, 0) == n

These three properties will be run with hundreds of randomly generated integer pairs. They catch entire classes of bugs — any implementation of add that isn't commutative or associative is wrong, for any inputs.

The Core Difference in Thinking

The shift from example-based to property-based thinking is a shift from specific cases to universal statements.

This changes not just how you write tests, but how you think about your code. Writing properties forces you to articulate what your function is supposed to do in general terms, which often reveals ambiguities in requirements you hadn't noticed.

What Example-Based Tests Miss: Concrete Cases

Let's look at real scenarios where example-based tests pass but property-based tests find bugs.

Case 1: Integer Overflow

def multiply(a: int, b: int) -> int:
    return a * b

# Example-based tests — all pass
def test_multiply():
    assert multiply(3, 4) == 12
    assert multiply(-2, 5) == -10
    assert multiply(0, 100) == 0
    assert multiply(1000, 1000) == 1_000_000

These tests pass easily. But what if multiply is implemented in a language or library with fixed-size integers? A property test immediately exposes this:

@given(st.integers(), st.integers())
def test_multiply_is_commutative(a, b):
    assert multiply(a, b) == multiply(b, a)

@given(st.integers())
def test_multiply_by_one_is_identity(n):
    assert multiply(n, 1) == n

If the underlying implementation has an overflow bug at 2^31, fast-check or Hypothesis will find inputs that trigger it within the first few hundred runs.

Case 2: String Encoding Edge Cases

def safe_encode(s: str) -> bytes:
    return s.encode('ascii', errors='ignore')

def safe_decode(b: bytes) -> str:
    return b.decode('ascii', errors='ignore')

# Example-based tests — all pass
def test_encoding():
    assert safe_encode('hello') == b'hello'
    assert safe_encode('world') == b'world'
    assert safe_decode(b'hello') == 'hello'

These tests give false confidence. The errors='ignore' means that non-ASCII characters are silently dropped. A property test exposes the lossy behavior:

@given(st.text())
def test_encode_decode_roundtrip(s):
    assert safe_decode(safe_encode(s)) == s  # FAILS for non-ASCII input

The property test immediately fails on any string containing non-ASCII characters — accented letters, emoji, Chinese characters. The example-based tests never tried those.

Case 3: Sorting with a Custom Comparator

def sort_by_last_name(people):
    return sorted(people, key=lambda p: p['last_name'])

# Example-based tests
def test_sort_by_last_name():
    people = [
        {'first': 'Charlie', 'last_name': 'Wilson'},
        {'first': 'Alice', 'last_name': 'Smith'},
        {'first': 'Bob', 'last_name': 'Jones'},
    ]
    result = sort_by_last_name(people)
    assert result[0]['last_name'] == 'Jones'
    assert result[1]['last_name'] == 'Smith'
    assert result[2]['last_name'] == 'Wilson'

This test passes. But it doesn't check:

• What happens with people who have the same last name?
• Is the sort stable (original order preserved for equal keys)?
• What happens with empty last names?
• What happens with None values?

A property test covers all of these:

from hypothesis import given
from hypothesis import strategies as st

person = st.fixed_dictionaries({
    'first': st.text(min_size=0, max_size=50),
    'last_name': st.text(min_size=0, max_size=50),
})

@given(st.lists(person))
def test_sort_produces_non_decreasing_last_names(people):
    result = sort_by_last_name(people)
    for i in range(len(result) - 1):
        assert result[i]['last_name'] <= result[i+1]['last_name']

@given(st.lists(person))
def test_sort_preserves_length(people):
    assert len(sort_by_last_name(people)) == len(people)

@given(st.lists(person))
def test_sort_preserves_elements(people):
    result = sort_by_last_name(people)
    assert sorted(result, key=lambda p: p['first']) == sorted(people, key=lambda p: p['first'])

Case 4: JSON Serialization

import json

def serialize_config(config: dict) -> str:
    return json.dumps(config)

def deserialize_config(s: str) -> dict:
    return json.loads(s)

# Example-based tests
def test_config_roundtrip():
    config = {'timeout': 30, 'retries': 3, 'debug': False}
    assert deserialize_config(serialize_config(config)) == config

This test passes. But it doesn't test configurations with unusual values:

@given(st.dictionaries(
    keys=st.text(max_size=20),
    values=st.one_of(st.integers(), st.floats(allow_nan=False, allow_infinity=False),
                     st.text(), st.booleans(), st.none())
))
def test_serialize_deserialize_roundtrip(config):
    serialized = serialize_config(config)
    deserialized = deserialize_config(serialized)
    assert deserialized == config

This property test will quickly find that integers are preserved but floats like 1.0000000000000002 may not roundtrip exactly, and that dictionary keys that look like integers may be converted to integers in some JSON implementations.

When to Use Example-Based Testing

Use example-based testing when:

You're documenting known requirements. If the spec says "a user with age less than 18 must be rejected," write an example test for a 17-year-old and an 18-year-old. This makes the business rule explicit and readable.

You're testing specific business logic. Pricing rules, discount calculations, permission checks — these often depend on specific threshold values that are best expressed as examples.

You're testing error messages and UX. "When the user submits an empty form, show 'Name is required'" — this is a specific case that should be documented as an example.

You're in TDD mode. Red-green-refactor cycles work best with concrete examples that drive one small behavior at a time.

You have known tricky edge cases. If you've been bitten by a leap year bug or a daylight-saving-time issue before, write an example test for that specific case.

When to Use Property-Based Testing

Use property-based testing when:

You're testing transformations and algorithms. Sort functions, encoding/decoding, compression, mathematical operations — these have properties that should hold universally.

You're testing serialization/deserialization. Roundtrip tests are the single best use case for property-based testing. They catch encoding bugs that example-based tests almost never find.

You're testing functions with large input spaces. Any function that takes a string, integer, or collection as input has an effectively infinite input space. Property tests explore this space automatically.

You want to find bugs you haven't thought of. The whole point of property-based testing is that the framework generates inputs you wouldn't think to try.

You're refactoring. If you're replacing an implementation with a faster one, a property test comparing the two implementations will catch any divergence.

You're testing state machines. Stateful property testing tools (Hypothesis's RuleBasedStateMachine, fast-check's model-based testing) are unmatched for finding bugs in systems with complex state transitions.

Combining Both Approaches

The most effective test suite uses both approaches together. Here's a practical pattern:

# Example-based tests: document known requirements and edge cases
def test_divide_by_zero_raises():
    with pytest.raises(ZeroDivisionError):
        divide(10, 0)

def test_divide_negative_numbers():
    assert divide(-10, 2) == -5

def test_divide_result_is_float():
    assert isinstance(divide(1, 3), float)

# Property-based tests: verify universal invariants
@given(st.integers(), st.integers().filter(lambda n: n != 0))
def test_divide_multiplication_inverse(a, b):
    """Dividing and multiplying by the same number should give back the original."""
    assert divide(a, b) * b == pytest.approx(a)

@given(st.integers().filter(lambda n: n != 0))
def test_divide_by_self_is_one(n):
    assert divide(n, n) == pytest.approx(1.0)

@given(st.integers(), st.integers().filter(lambda n: n != 0))
def test_divide_result_sign(a, b):
    """Sign of result matches sign of quotient."""
    result = divide(a, b)
    if a > 0 and b > 0:
        assert result > 0
    elif a < 0 and b < 0:
        assert result > 0
    elif a == 0:
        assert result == 0

The example-based tests document what the function does for specific inputs. The property-based tests verify that the mathematical relationships hold universally. Together, they give you both documentation and coverage.

The Coverage Question

Example-based tests excel at line and branch coverage — you can craft examples to hit every code path. But coverage of lines is not coverage of inputs. Property-based tests cover inputs, not lines.

A function with 100% line coverage from example-based tests can still fail on inputs the examples never tried. Property-based testing addresses this by exploring the input space, not the code paths.

The ideal is both: high line/branch coverage from example-based tests, plus property-based tests that verify invariants across the input space.

How HelpMeTest Supports Both Testing Approaches

HelpMeTest is built to work with both testing styles. Its AI-powered test generation can identify both specific examples worth testing (based on your code's logic and branches) and properties that should hold for your functions. The Robot Framework and Playwright integration supports running property tests as part of your full test suite, and the cloud infrastructure handles the computational cost of running thousands of generated test cases. Teams using HelpMeTest's usage-based pricing get centralized visibility into both their example-based and property-based test results.

Conclusion

Property-based testing and example-based testing are not competing philosophies — they're complementary tools that address different testing challenges. Example-based tests are better for documenting known requirements and specific edge cases. Property-based tests are better for finding unknown bugs and verifying universal invariants.

The bugs that example-based tests miss — encoding edge cases, integer overflow, unexpected input combinations, state machine violations — are exactly the bugs that property-based testing is designed to find. Use both, and your test suite will catch more bugs than either approach can alone.

Start by adding property tests to your most critical utility functions. Roundtrip tests for any serialization code are the easiest place to begin, and they consistently find bugs that years of example-based testing have missed.