Hypothesis (Python): Advanced Shrinking and Stateful Testing Deep Dive
Hypothesis is the gold standard for property-based testing in Python. Most tutorials cover the basics—@given, st.integers(), simple assertions. This post goes deeper: shrinking internals, custom composite strategies, RuleBasedStateMachine for stateful systems, and integrating with CI so failures are always reproducible.
How Hypothesis Shrinking Actually Works
When Hypothesis finds a failing example, it doesn't report the raw random input. It shrinks it—repeatedly simplifying the example while the test still fails. This gives you the minimal counterexample, which is almost always more debuggable than the original.
Understanding the shrinking process matters when you write custom strategies, because a poorly written strategy can produce inputs that shrink badly (or not at all).
The Shrinking Algorithm
Hypothesis uses a lexicographic shrinking approach on an internal byte stream. Every generated value is derived from this byte stream. Shrinking reduces the byte stream in a principled way, which automatically shrinks the derived values.
This is why Hypothesis shrinking is so powerful: you don't need to write custom shrinkers for composite strategies. The byte-stream approach handles shrinking for free.
from hypothesis import given, settings
from hypothesis import strategies as st
@given(st.lists(st.integers(), min_size=1))
def test_sum_not_negative(xs):
# This will find the minimal failing case
assert sum(xs) >= 0When this fails (it will, for [-1]), Hypothesis gives you [-1], not [-3847, 12, -99, 2048].
When Shrinking Fails
Shrinking can get stuck when your strategy generates values with implicit constraints. Consider:
# Problematic: implicit constraint between two values
@given(st.integers(min_value=0, max_value=100),
st.integers(min_value=0, max_value=100))
def test_something(a, b):
assume(a < b) # This creates an implicit constraint
...The assume() call filters examples but doesn't help shrinking understand that a must remain less than b. A better approach:
@given(st.integers(min_value=0, max_value=99))
def test_something(a):
b = a + 1 # Express the relationship explicitly
...Or use st.tuples with a flatmap:
@given(st.integers(min_value=0, max_value=99).flatmap(
lambda a: st.tuples(st.just(a), st.integers(min_value=a+1, max_value=100))
))
def test_something(ab):
a, b = ab
...Custom Composite Strategies
The @composite decorator lets you build strategies that express domain constraints directly.
Building a Valid HTTP Request Strategy
from hypothesis import given, strategies as st
from hypothesis.strategies import composite
import string
@composite
def http_requests(draw):
method = draw(st.sampled_from(['GET', 'POST', 'PUT', 'DELETE', 'PATCH']))
# Build a valid path
segments = draw(st.lists(
st.text(alphabet=string.ascii_lowercase + string.digits + '-_',
min_size=1, max_size=20),
min_size=1,
max_size=5
))
path = '/' + '/'.join(segments)
# Only POST/PUT/PATCH have bodies
if method in ('POST', 'PUT', 'PATCH'):
body = draw(st.one_of(
st.none(),
st.dictionaries(
st.text(alphabet=string.ascii_letters, min_size=1, max_size=20),
st.one_of(st.integers(), st.text(), st.booleans())
)
))
else:
body = None
headers = draw(st.fixed_dictionaries({
'Content-Type': st.sampled_from([
'application/json',
'application/x-www-form-urlencoded',
'text/plain'
]),
'Accept': st.just('application/json'),
}))
return {'method': method, 'path': path, 'body': body, 'headers': headers}
@given(http_requests())
def test_router_handles_all_requests(request):
response = router.handle(request)
assert response['status'] in range(200, 600)
assert 'Content-Type' in response['headers']Recursive Strategies for Tree-Shaped Data
For nested structures like ASTs, JSON, or directory trees:
@composite
def json_value(draw, max_depth=3):
if max_depth == 0:
# Base case: only primitives
return draw(st.one_of(
st.none(),
st.booleans(),
st.integers(),
st.floats(allow_nan=False, allow_infinity=False),
st.text()
))
return draw(st.one_of(
st.none(),
st.booleans(),
st.integers(),
st.text(),
# Recursive cases with reduced depth
st.lists(json_value(max_depth=max_depth - 1), max_size=5),
st.dictionaries(
st.text(min_size=1),
json_value(max_depth=max_depth - 1),
max_size=5
)
))
@given(json_value())
def test_json_roundtrip(value):
import json
assert json.loads(json.dumps(value)) == valueHypothesis also has st.recursive() built in:
json_strategy = st.recursive(
base=st.one_of(st.none(), st.booleans(), st.integers(), st.text()),
extend=lambda children: st.one_of(
st.lists(children, max_size=5),
st.dictionaries(st.text(min_size=1), children, max_size=5)
),
max_leaves=20
)Stateful Testing with RuleBasedStateMachine
The most powerful Hypothesis feature is RuleBasedStateMachine. It models a system as a state machine and generates sequences of operations, checking invariants after each step.
Testing a Stack Implementation
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant, initialize
from hypothesis import strategies as st
class StackMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.model = [] # Reference implementation (Python list)
self.impl = Stack() # Your implementation under test
@rule(value=st.integers())
def push(self, value):
self.model.append(value)
self.impl.push(value)
@rule()
def pop(self):
if self.model:
expected = self.model.pop()
actual = self.impl.pop()
assert actual == expected, f"Pop mismatch: expected {expected}, got {actual}"
@rule()
def peek(self):
if self.model:
assert self.impl.peek() == self.model[-1]
@invariant()
def size_consistent(self):
assert len(self.impl) == len(self.model)
@invariant()
def empty_consistent(self):
assert self.impl.is_empty() == (len(self.model) == 0)
TestStack = StackMachine.TestCaseRun with pytest: pytest test_stack.py::TestStack
Hypothesis generates random sequences of push, pop, and peek operations and checks all invariants after every step. If your stack implementation has a bug in edge cases (empty stack pop, push after pop, etc.), this will find it.
Testing a Cache with Eviction Policy
from hypothesis.stateful import (
RuleBasedStateMachine, rule, invariant,
initialize, Bundle, consumes
)
from hypothesis import strategies as st
class LRUCacheMachine(RuleBasedStateMachine):
Keys = Bundle('keys')
def __init__(self):
super().__init__()
self.capacity = 3
self.cache = LRUCache(capacity=self.capacity)
self.model = {} # Simple dict as reference
self.access_order = []
@rule(target=Keys, key=st.text(min_size=1, max_size=10),
value=st.integers())
def put(self, key, value):
# Update model (LRU eviction logic)
if key not in self.model and len(self.model) >= self.capacity:
# Evict LRU key
lru_key = self.access_order[0]
del self.model[lru_key]
self.access_order.remove(lru_key)
if key in self.access_order:
self.access_order.remove(key)
self.access_order.append(key)
self.model[key] = value
self.cache.put(key, value)
return key
@rule(key=Keys)
def get(self, key):
expected = self.model.get(key)
actual = self.cache.get(key)
assert actual == expected
# Update access order in model
if key in self.model:
self.access_order.remove(key)
self.access_order.append(key)
@invariant()
def size_within_capacity(self):
assert len(self.cache) <= self.capacity
@invariant()
def cache_size_matches_model(self):
assert len(self.cache) == len(self.model)
TestLRUCache = LRUCacheMachine.TestCaseThe Hypothesis Database: Reproducible CI Failures
Hypothesis stores failing examples in a local database (.hypothesis/ directory). This ensures:
- Failures are replayed on subsequent runs—the counterexample is always tested first
- CI can reproduce local failures—commit the
.hypothesis/directory
Configuring the Database
# conftest.py
from hypothesis import settings, HealthCheck
from hypothesis.database import DirectoryBasedExampleDatabase
settings.register_profile("ci",
max_examples=500,
suppress_health_check=[HealthCheck.too_slow],
database=DirectoryBasedExampleDatabase(".hypothesis/ci")
)
settings.register_profile("dev",
max_examples=100,
database=DirectoryBasedExampleDatabase(".hypothesis/dev")
)
settings.register_profile("fast",
max_examples=10,
database=None # Don't persist during fast local checks
)
import os
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))In CI:
# .github/workflows/test.yml
- name: Run property tests
env:
HYPOTHESIS_PROFILE: ci
run: pytest tests/
- name: Cache hypothesis database
uses: actions/cache@v3
with:
path: .hypothesis/
key: hypothesis-${{ hashFiles('**/requirements*.txt') }}Explicit Example Replay
Pin specific examples to always be tested:
from hypothesis import given, example
from hypothesis import strategies as st
@given(st.integers())
@example(0) # Always test zero
@example(-1) # Always test negative
@example(2**31) # Always test overflow boundary
def test_process_value(n):
result = process_value(n)
assert result is not NonePerformance: Avoiding Slow Property Tests
Property tests can be slow if strategies generate expensive-to-construct objects repeatedly.
Profile Before Optimizing
from hypothesis import given, settings
from hypothesis.strategies import integers
@settings(max_examples=1000, stateful_step_count=50)
@given(integers())
def test_slow(n):
...Run with --hypothesis-show-statistics to see where time goes:
hypothesis statistics:
- Avg test duration: 45ms
- Fastest: 2ms, Slowest: 892ms
- Shrinking: 3.2s totalUse st.builds() for Complex Objects
Instead of constructing objects inside tests, use st.builds():
from dataclasses import dataclass
from hypothesis import strategies as st
@dataclass
class Order:
customer_id: int
items: list
total: float
# Good: Hypothesis controls construction, can cache
order_strategy = st.builds(
Order,
customer_id=st.integers(min_value=1),
items=st.lists(st.text(), min_size=1),
total=st.floats(min_value=0.01, max_value=10000.0)
)Deadline Configuration
Hypothesis has a 200ms per-example deadline by default. For integration tests touching real databases or network:
from hypothesis import given, settings
from hypothesis.strategies import integers
@settings(deadline=None) # Disable deadline for slow integration tests
@given(integers())
def test_database_operation(n):
...Or increase it:
@settings(deadline=2000) # 2 second deadlineReal-World Integration: Testing a REST API
import pytest
import requests
from hypothesis import given, settings, HealthCheck
from hypothesis import strategies as st
BASE_URL = "http://localhost:8000"
user_strategy = st.fixed_dictionaries({
'username': st.text(
alphabet=st.characters(whitelist_categories=['Lu', 'Ll', 'Nd']),
min_size=3,
max_size=30
),
'email': st.emails(),
'age': st.integers(min_value=18, max_value=120)
})
@settings(
max_examples=50,
suppress_health_check=[HealthCheck.too_slow],
deadline=5000
)
@given(user_strategy)
def test_create_user_roundtrip(user_data):
# Create
resp = requests.post(f"{BASE_URL}/users", json=user_data)
assert resp.status_code in (200, 201, 422), \
f"Unexpected status {resp.status_code}: {resp.text}"
if resp.status_code in (200, 201):
user_id = resp.json()['id']
# Read back
get_resp = requests.get(f"{BASE_URL}/users/{user_id}")
assert get_resp.status_code == 200
returned = get_resp.json()
assert returned['username'] == user_data['username']
assert returned['email'] == user_data['email']
# Cleanup
requests.delete(f"{BASE_URL}/users/{user_id}")Integrating with HelpMeTest for Continuous Monitoring
Property tests are excellent candidates for 24/7 monitoring via HelpMeTest. The combination is powerful: Hypothesis finds edge cases during development, and HelpMeTest verifies your API handles them continuously in production.
Set up a HelpMeTest scenario that:
- Generates representative inputs from your known-failure database
- Posts them to your production endpoint
- Asserts invariants hold
This catches regressions where a property that held during development breaks after a deploy.
Key Takeaways
- Shrinking is automatic with Hypothesis—write strategies that express constraints explicitly rather than using
assume()excessively @compositestrategies let you express domain constraints cleanly and shrink correctlyRuleBasedStateMachinefinds bugs in stateful systems that unit tests miss- Commit
.hypothesis/so CI always replays known failures - Use
@example()to pin boundary cases that must always run
Property-based testing with Hypothesis finds classes of bugs that are nearly impossible to catch with hand-written test cases. The investment in writing good strategies pays off every time Hypothesis finds the minimal counterexample you never thought to write.