Hypothesis (Python): Advanced Shrinking and Stateful Testing Deep Dive

Hypothesis (Python): Advanced Shrinking and Stateful Testing Deep Dive

Hypothesis is the gold standard for property-based testing in Python. Most tutorials cover the basics—@given, st.integers(), simple assertions. This post goes deeper: shrinking internals, custom composite strategies, RuleBasedStateMachine for stateful systems, and integrating with CI so failures are always reproducible.

How Hypothesis Shrinking Actually Works

When Hypothesis finds a failing example, it doesn't report the raw random input. It shrinks it—repeatedly simplifying the example while the test still fails. This gives you the minimal counterexample, which is almost always more debuggable than the original.

Understanding the shrinking process matters when you write custom strategies, because a poorly written strategy can produce inputs that shrink badly (or not at all).

The Shrinking Algorithm

Hypothesis uses a lexicographic shrinking approach on an internal byte stream. Every generated value is derived from this byte stream. Shrinking reduces the byte stream in a principled way, which automatically shrinks the derived values.

This is why Hypothesis shrinking is so powerful: you don't need to write custom shrinkers for composite strategies. The byte-stream approach handles shrinking for free.

from hypothesis import given, settings
from hypothesis import strategies as st

@given(st.lists(st.integers(), min_size=1))
def test_sum_not_negative(xs):
    # This will find the minimal failing case
    assert sum(xs) >= 0

When this fails (it will, for [-1]), Hypothesis gives you [-1], not [-3847, 12, -99, 2048].

When Shrinking Fails

Shrinking can get stuck when your strategy generates values with implicit constraints. Consider:

# Problematic: implicit constraint between two values
@given(st.integers(min_value=0, max_value=100),
       st.integers(min_value=0, max_value=100))
def test_something(a, b):
    assume(a < b)  # This creates an implicit constraint
    ...

The assume() call filters examples but doesn't help shrinking understand that a must remain less than b. A better approach:

@given(st.integers(min_value=0, max_value=99))
def test_something(a):
    b = a + 1  # Express the relationship explicitly
    ...

Or use st.tuples with a flatmap:

@given(st.integers(min_value=0, max_value=99).flatmap(
    lambda a: st.tuples(st.just(a), st.integers(min_value=a+1, max_value=100))
))
def test_something(ab):
    a, b = ab
    ...

Custom Composite Strategies

The @composite decorator lets you build strategies that express domain constraints directly.

Building a Valid HTTP Request Strategy

from hypothesis import given, strategies as st
from hypothesis.strategies import composite
import string

@composite
def http_requests(draw):
    method = draw(st.sampled_from(['GET', 'POST', 'PUT', 'DELETE', 'PATCH']))
    
    # Build a valid path
    segments = draw(st.lists(
        st.text(alphabet=string.ascii_lowercase + string.digits + '-_',
                min_size=1, max_size=20),
        min_size=1,
        max_size=5
    ))
    path = '/' + '/'.join(segments)
    
    # Only POST/PUT/PATCH have bodies
    if method in ('POST', 'PUT', 'PATCH'):
        body = draw(st.one_of(
            st.none(),
            st.dictionaries(
                st.text(alphabet=string.ascii_letters, min_size=1, max_size=20),
                st.one_of(st.integers(), st.text(), st.booleans())
            )
        ))
    else:
        body = None
    
    headers = draw(st.fixed_dictionaries({
        'Content-Type': st.sampled_from([
            'application/json',
            'application/x-www-form-urlencoded',
            'text/plain'
        ]),
        'Accept': st.just('application/json'),
    }))
    
    return {'method': method, 'path': path, 'body': body, 'headers': headers}

@given(http_requests())
def test_router_handles_all_requests(request):
    response = router.handle(request)
    assert response['status'] in range(200, 600)
    assert 'Content-Type' in response['headers']

Recursive Strategies for Tree-Shaped Data

For nested structures like ASTs, JSON, or directory trees:

@composite
def json_value(draw, max_depth=3):
    if max_depth == 0:
        # Base case: only primitives
        return draw(st.one_of(
            st.none(),
            st.booleans(),
            st.integers(),
            st.floats(allow_nan=False, allow_infinity=False),
            st.text()
        ))
    
    return draw(st.one_of(
        st.none(),
        st.booleans(),
        st.integers(),
        st.text(),
        # Recursive cases with reduced depth
        st.lists(json_value(max_depth=max_depth - 1), max_size=5),
        st.dictionaries(
            st.text(min_size=1),
            json_value(max_depth=max_depth - 1),
            max_size=5
        )
    ))

@given(json_value())
def test_json_roundtrip(value):
    import json
    assert json.loads(json.dumps(value)) == value

Hypothesis also has st.recursive() built in:

json_strategy = st.recursive(
    base=st.one_of(st.none(), st.booleans(), st.integers(), st.text()),
    extend=lambda children: st.one_of(
        st.lists(children, max_size=5),
        st.dictionaries(st.text(min_size=1), children, max_size=5)
    ),
    max_leaves=20
)

Stateful Testing with RuleBasedStateMachine

The most powerful Hypothesis feature is RuleBasedStateMachine. It models a system as a state machine and generates sequences of operations, checking invariants after each step.

Testing a Stack Implementation

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant, initialize
from hypothesis import strategies as st

class StackMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.model = []      # Reference implementation (Python list)
        self.impl = Stack()  # Your implementation under test
    
    @rule(value=st.integers())
    def push(self, value):
        self.model.append(value)
        self.impl.push(value)
    
    @rule()
    def pop(self):
        if self.model:
            expected = self.model.pop()
            actual = self.impl.pop()
            assert actual == expected, f"Pop mismatch: expected {expected}, got {actual}"
    
    @rule()
    def peek(self):
        if self.model:
            assert self.impl.peek() == self.model[-1]
    
    @invariant()
    def size_consistent(self):
        assert len(self.impl) == len(self.model)
    
    @invariant()
    def empty_consistent(self):
        assert self.impl.is_empty() == (len(self.model) == 0)

TestStack = StackMachine.TestCase

Run with pytest: pytest test_stack.py::TestStack

Hypothesis generates random sequences of push, pop, and peek operations and checks all invariants after every step. If your stack implementation has a bug in edge cases (empty stack pop, push after pop, etc.), this will find it.

Testing a Cache with Eviction Policy

from hypothesis.stateful import (
    RuleBasedStateMachine, rule, invariant, 
    initialize, Bundle, consumes
)
from hypothesis import strategies as st

class LRUCacheMachine(RuleBasedStateMachine):
    Keys = Bundle('keys')
    
    def __init__(self):
        super().__init__()
        self.capacity = 3
        self.cache = LRUCache(capacity=self.capacity)
        self.model = {}  # Simple dict as reference
        self.access_order = []
    
    @rule(target=Keys, key=st.text(min_size=1, max_size=10), 
          value=st.integers())
    def put(self, key, value):
        # Update model (LRU eviction logic)
        if key not in self.model and len(self.model) >= self.capacity:
            # Evict LRU key
            lru_key = self.access_order[0]
            del self.model[lru_key]
            self.access_order.remove(lru_key)
        
        if key in self.access_order:
            self.access_order.remove(key)
        self.access_order.append(key)
        self.model[key] = value
        self.cache.put(key, value)
        return key
    
    @rule(key=Keys)
    def get(self, key):
        expected = self.model.get(key)
        actual = self.cache.get(key)
        assert actual == expected
        
        # Update access order in model
        if key in self.model:
            self.access_order.remove(key)
            self.access_order.append(key)
    
    @invariant()
    def size_within_capacity(self):
        assert len(self.cache) <= self.capacity
    
    @invariant()
    def cache_size_matches_model(self):
        assert len(self.cache) == len(self.model)

TestLRUCache = LRUCacheMachine.TestCase

The Hypothesis Database: Reproducible CI Failures

Hypothesis stores failing examples in a local database (.hypothesis/ directory). This ensures:

  1. Failures are replayed on subsequent runs—the counterexample is always tested first
  2. CI can reproduce local failures—commit the .hypothesis/ directory

Configuring the Database

# conftest.py
from hypothesis import settings, HealthCheck
from hypothesis.database import DirectoryBasedExampleDatabase

settings.register_profile("ci", 
    max_examples=500,
    suppress_health_check=[HealthCheck.too_slow],
    database=DirectoryBasedExampleDatabase(".hypothesis/ci")
)

settings.register_profile("dev",
    max_examples=100,
    database=DirectoryBasedExampleDatabase(".hypothesis/dev")
)

settings.register_profile("fast",
    max_examples=10,
    database=None  # Don't persist during fast local checks
)

import os
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))

In CI:

# .github/workflows/test.yml
- name: Run property tests
  env:
    HYPOTHESIS_PROFILE: ci
  run: pytest tests/

- name: Cache hypothesis database
  uses: actions/cache@v3
  with:
    path: .hypothesis/
    key: hypothesis-${{ hashFiles('**/requirements*.txt') }}

Explicit Example Replay

Pin specific examples to always be tested:

from hypothesis import given, example
from hypothesis import strategies as st

@given(st.integers())
@example(0)        # Always test zero
@example(-1)       # Always test negative
@example(2**31)    # Always test overflow boundary
def test_process_value(n):
    result = process_value(n)
    assert result is not None

Performance: Avoiding Slow Property Tests

Property tests can be slow if strategies generate expensive-to-construct objects repeatedly.

Profile Before Optimizing

from hypothesis import given, settings
from hypothesis.strategies import integers

@settings(max_examples=1000, stateful_step_count=50)
@given(integers())
def test_slow(n):
    ...

Run with --hypothesis-show-statistics to see where time goes:

hypothesis statistics:
  - Avg test duration: 45ms
  - Fastest: 2ms, Slowest: 892ms
  - Shrinking: 3.2s total

Use st.builds() for Complex Objects

Instead of constructing objects inside tests, use st.builds():

from dataclasses import dataclass
from hypothesis import strategies as st

@dataclass
class Order:
    customer_id: int
    items: list
    total: float

# Good: Hypothesis controls construction, can cache
order_strategy = st.builds(
    Order,
    customer_id=st.integers(min_value=1),
    items=st.lists(st.text(), min_size=1),
    total=st.floats(min_value=0.01, max_value=10000.0)
)

Deadline Configuration

Hypothesis has a 200ms per-example deadline by default. For integration tests touching real databases or network:

from hypothesis import given, settings
from hypothesis.strategies import integers

@settings(deadline=None)  # Disable deadline for slow integration tests
@given(integers())
def test_database_operation(n):
    ...

Or increase it:

@settings(deadline=2000)  # 2 second deadline

Real-World Integration: Testing a REST API

import pytest
import requests
from hypothesis import given, settings, HealthCheck
from hypothesis import strategies as st

BASE_URL = "http://localhost:8000"

user_strategy = st.fixed_dictionaries({
    'username': st.text(
        alphabet=st.characters(whitelist_categories=['Lu', 'Ll', 'Nd']),
        min_size=3,
        max_size=30
    ),
    'email': st.emails(),
    'age': st.integers(min_value=18, max_value=120)
})

@settings(
    max_examples=50,
    suppress_health_check=[HealthCheck.too_slow],
    deadline=5000
)
@given(user_strategy)
def test_create_user_roundtrip(user_data):
    # Create
    resp = requests.post(f"{BASE_URL}/users", json=user_data)
    assert resp.status_code in (200, 201, 422), \
        f"Unexpected status {resp.status_code}: {resp.text}"
    
    if resp.status_code in (200, 201):
        user_id = resp.json()['id']
        
        # Read back
        get_resp = requests.get(f"{BASE_URL}/users/{user_id}")
        assert get_resp.status_code == 200
        returned = get_resp.json()
        
        assert returned['username'] == user_data['username']
        assert returned['email'] == user_data['email']
        
        # Cleanup
        requests.delete(f"{BASE_URL}/users/{user_id}")

Integrating with HelpMeTest for Continuous Monitoring

Property tests are excellent candidates for 24/7 monitoring via HelpMeTest. The combination is powerful: Hypothesis finds edge cases during development, and HelpMeTest verifies your API handles them continuously in production.

Set up a HelpMeTest scenario that:

  1. Generates representative inputs from your known-failure database
  2. Posts them to your production endpoint
  3. Asserts invariants hold

This catches regressions where a property that held during development breaks after a deploy.

Key Takeaways

  • Shrinking is automatic with Hypothesis—write strategies that express constraints explicitly rather than using assume() excessively
  • @composite strategies let you express domain constraints cleanly and shrink correctly
  • RuleBasedStateMachine finds bugs in stateful systems that unit tests miss
  • Commit .hypothesis/ so CI always replays known failures
  • Use @example() to pin boundary cases that must always run

Property-based testing with Hypothesis finds classes of bugs that are nearly impossible to catch with hand-written test cases. The investment in writing good strategies pays off every time Hypothesis finds the minimal counterexample you never thought to write.

Read more

Start now free