Hypothesis Stateful Testing: RuleBasedStateMachine for Complex Systems
Hypothesis generates random inputs for individual functions. But many bugs only appear when a sequence of operations interact — you add an item to a cache, fetch it, update it, then fetch it again and get a stale value. These bugs are invisible to property-based tests on individual functions.
Hypothesis stateful testing solves this by generating random sequences of operations (not just random inputs), then checking invariants after each step. It's implemented through RuleBasedStateMachine — a state machine where each method is a possible operation and Hypothesis explores random orderings.
Basic RuleBasedStateMachine
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant, initialize
from hypothesis import given, settings
import hypothesis.strategies as st
class CacheStateMachine(RuleBasedStateMachine):
"""
Test an LRU cache by generating random get/set/delete sequences
and verifying consistency invariants after each operation.
"""
def __init__(self):
super().__init__()
self.model = {} # Our reference model (dict)
self.cache = LRUCache(capacity=5) # System under test
@initialize(key=st.text(min_size=1, max_size=10), value=st.integers())
def seed_data(self, key, value):
"""Initialize with some data before rules run."""
self.model[key] = value
self.cache.set(key, value)
@rule(key=st.text(min_size=1, max_size=10), value=st.integers())
def set_value(self, key, value):
self.model[key] = value
self.cache.set(key, value)
@rule(key=st.text(min_size=1, max_size=10))
def get_value(self, key):
model_value = self.model.get(key)
cache_value = self.cache.get(key)
assert model_value == cache_value, (
f"Cache inconsistency: model has {model_value!r}, "
f"cache has {cache_value!r} for key {key!r}"
)
@rule(key=st.text(min_size=1, max_size=10))
def delete_value(self, key):
self.model.pop(key, None)
self.cache.delete(key)
@invariant()
def cache_size_bounded(self):
"""Cache should never exceed its capacity."""
assert len(self.cache) <= self.cache.capacity
@invariant()
def model_cache_keys_consistent(self):
"""
Every key in the model that fits in the cache should be retrievable.
(LRU eviction means not all model keys are in cache when capacity exceeded)
"""
for key in list(self.cache.keys()):
assert key in self.model, f"Cache contains {key!r} not in model"
TestCache = CacheStateMachine.TestCaseThe TestCase attribute is a standard unittest test class that Hypothesis generates. Run it with pytest:
pytest test_cache.py -vKey Concepts
Rules
Rules are the operations Hypothesis can apply. Each @rule method receives randomly generated arguments:
@rule(amount=st.decimals(min_value="0.01", max_value="1000.00", places=2))
def deposit(self, amount):
self.account.deposit(amount)
self.balance += amount
@rule(amount=st.decimals(min_value="0.01", max_value="1000.00", places=2))
def withdraw(self, amount):
if amount <= self.balance:
self.account.withdraw(amount)
self.balance -= amountHypothesis picks a random sequence of rules and arguments. The same rule can be applied multiple times in a sequence.
Preconditions
Use precondition to guard rules that should only run in specific states:
from hypothesis.stateful import precondition
@rule(amount=st.decimals(min_value="0.01", max_value="1000.00", places=2))
@precondition(lambda self: self.balance > 0)
def withdraw(self, amount):
amount = min(amount, self.balance)
self.account.withdraw(amount)
self.balance -= amountWithout @precondition, Hypothesis generates withdrawal amounts even when the balance is 0, which might raise an exception rather than testing the real invariant.
Invariants
Invariants run after every step. If an invariant fails, Hypothesis shows the sequence of operations that led to the failure:
@invariant()
def balance_never_negative(self):
assert self.account.balance >= Decimal("0.00"), \
f"Balance went negative: {self.account.balance}"
@invariant()
def model_matches_system(self):
assert self.account.balance == self.balance, \
f"Model: {self.balance}, System: {self.account.balance}"Initialize
@initialize runs once at the start of each test run, before any rules. Use it to put the system in a valid starting state:
@initialize(
username=st.text(alphabet=st.characters(whitelist_categories=('Lu', 'Ll')), min_size=3),
email=st.emails()
)
def create_account(self, username, email):
self.user = UserService.create(username=username, email=email)
self.expected_username = username
self.expected_email = emailTesting a REST API
Apply stateful testing to HTTP APIs by treating endpoints as rules:
import httpx
from hypothesis.stateful import RuleBasedStateMachine, rule, initialize, invariant
import hypothesis.strategies as st
class TodoApiStateMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.client = httpx.Client(base_url="http://localhost:8000")
self.todos = {} # id -> {title, completed}
self._clear_database()
def _clear_database(self):
response = self.client.delete("/api/todos/all")
assert response.status_code == 204
@initialize(title=st.text(min_size=1, max_size=100))
def create_initial_todo(self, title):
response = self.client.post("/api/todos", json={"title": title})
assert response.status_code == 201
todo = response.json()
self.todos[todo["id"]] = {"title": title, "completed": False}
@rule(title=st.text(min_size=1, max_size=100))
def create_todo(self, title):
response = self.client.post("/api/todos", json={"title": title})
assert response.status_code == 201
todo = response.json()
self.todos[todo["id"]] = {"title": title, "completed": False}
@rule(data=st.data())
def complete_todo(self, data):
if not self.todos:
return
todo_id = data.draw(st.sampled_from(list(self.todos.keys())))
response = self.client.patch(f"/api/todos/{todo_id}", json={"completed": True})
assert response.status_code in (200, 404)
if response.status_code == 200:
self.todos[todo_id]["completed"] = True
@rule(data=st.data())
def delete_todo(self, data):
if not self.todos:
return
todo_id = data.draw(st.sampled_from(list(self.todos.keys())))
response = self.client.delete(f"/api/todos/{todo_id}")
assert response.status_code in (204, 404)
self.todos.pop(todo_id, None)
@invariant()
def list_matches_model(self):
response = self.client.get("/api/todos")
assert response.status_code == 200
api_todos = {t["id"]: t for t in response.json()}
assert set(api_todos.keys()) == set(self.todos.keys()), \
f"API has {set(api_todos.keys())}, model has {set(self.todos.keys())}"
for todo_id, expected in self.todos.items():
assert api_todos[todo_id]["completed"] == expected["completed"], \
f"Todo {todo_id} completion mismatch"
def teardown(self):
self._clear_database()
self.client.close()
TestTodoApi = TodoApiStateMachine.TestCaseUsing st.data() for Dynamic Sampling
When a rule needs to choose from the current state (e.g., pick an existing item), use st.data():
@rule(data=st.data())
def update_existing_item(self, data):
if not self.items:
return
# Dynamically sample from current state
item_id = data.draw(st.sampled_from(list(self.items.keys())))
new_value = data.draw(st.integers(min_value=0, max_value=1000))
result = self.service.update(item_id, new_value)
self.items[item_id] = new_value
assert result.value == new_valuest.data() lets you use dynamic strategies that depend on the current state, rather than pre-generated random values.
Configuring State Machine Tests
from hypothesis import settings, HealthCheck
TestTodoApi = TodoApiStateMachine.TestCase
TestTodoApi.settings = settings(
max_examples=50, # Number of sequences to generate
stateful_step_count=20, # Maximum steps per sequence
suppress_health_check=[HealthCheck.too_slow],
)For longer investigations:
TestTodoApi.settings = settings(
max_examples=200,
stateful_step_count=50,
deadline=None, # No time limit per test
)Database Testing
State machines excel at finding consistency bugs in database-backed systems:
class UserRepositoryStateMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.db = TestDatabase() # In-memory SQLite
self.db.create_tables()
self.repo = UserRepository(self.db)
self.model = {} # Expected state
@initialize(
email=st.emails(),
name=st.text(min_size=1, max_size=50)
)
def create_user(self, email, name):
user = self.repo.create(email=email, name=name)
self.model[user.id] = {"email": email, "name": name, "active": True}
@rule(data=st.data(), name=st.text(min_size=1, max_size=50))
def update_name(self, data, name):
if not self.model:
return
user_id = data.draw(st.sampled_from(list(self.model.keys())))
self.repo.update_name(user_id, name)
self.model[user_id]["name"] = name
@rule(data=st.data())
def deactivate_user(self, data):
if not self.model:
return
active_ids = [id for id, u in self.model.items() if u["active"]]
if not active_ids:
return
user_id = data.draw(st.sampled_from(active_ids))
self.repo.deactivate(user_id)
self.model[user_id]["active"] = False
@invariant()
def active_user_count_matches(self):
expected_active = sum(1 for u in self.model.values() if u["active"])
actual_active = self.repo.count_active()
assert expected_active == actual_active
@invariant()
def all_model_users_exist_in_db(self):
for user_id in self.model:
user = self.repo.find_by_id(user_id)
assert user is not None, f"User {user_id} missing from database"
def teardown(self):
self.db.drop_tables()
TestUserRepository = UserRepositoryStateMachine.TestCaseReading the Output on Failure
When stateful testing finds a failure, Hypothesis shows the sequence:
Falsifying example:
state = UserRepositoryStateMachine()
v1 = state.create_user(email='a@example.com', name='Alice')
v2 = state.update_name(data=data(...), name='')
state.deactivate_user(data=data(...))
v3 = state.invariant_active_user_count_matches() # FAILED: expected 1, got 2
state.teardown()This trace is reproducible — run it again with the same database seed to get the same sequence. The sequence is also shrunk to the minimum steps that reproduce the failure.
When to Use Stateful Testing
Stateful testing is most valuable for:
- Caches and data structures: LRU eviction, priority queues, bloom filters
- REST APIs with CRUD operations: Create/read/update/delete consistency
- Event sourcing / CQRS: Commands generate events, queries read projections
- Session and auth systems: Login/logout/token refresh sequences
- File systems and storage: Upload/download/delete ordering
It's overkill for pure functions with no internal state. Use regular @given properties for those.
The golden rule: if a bug requires multiple operations in a specific order, stateful testing will find it. If bugs are independent of order, regular properties are sufficient and faster.