Schemathesis: Property-Based Testing That Finds Bugs You Didn't Think to Write

19 Sep 2026 — 8 min read

The hardest bugs to find with manual testing are the ones you don't think to test. You write tests for the happy path, a few expected error cases, and maybe some edge cases you've burned by before. But the input space of a real API is enormous — and somewhere in that space is the combination that crashes your server, corrupts your data, or triggers a 500 instead of a 400.

Schemathesis is a property-based testing tool for HTTP APIs. Instead of you writing test cases, Schemathesis generates them automatically from your OpenAPI or GraphQL schema. It uses the spec as a grammar for valid (and intentionally invalid) inputs, sends thousands of requests, and reports every case where your server's behavior violates what the spec promises.

The result: bugs you would never have written a test case for, caught automatically.

What "Property-Based" Means for APIs

Traditional testing: you pick specific inputs and assert specific outputs.

Property-based testing: you define properties that should hold for all inputs, and the framework generates inputs automatically to try to violate them.

For HTTP APIs, the foundational property is simple but powerful: for any request that conforms to the schema, the server should not return a 5xx error. If your spec says an endpoint accepts an integer between 0 and 100, your server should handle any integer in that range — including 0, 100, and anything in between — without crashing.

Schemathesis also checks:

Response bodies match the documented schema
Required headers are present in responses
4xx responses include the documented error format
The server stays responsive after many requests

Installation

# Via pip (Python 3.8+)
pip install schemathesis

# Or with extras for specific features
pip install schemathesis[all]

# Verify
st --version

Schemathesis also ships as a Docker image:

docker pull schemathesis/schemathesis:stable

Basic CLI Usage

The fastest path to results is pointing Schemathesis at a running server:

# Test against a live URL
st run https://api.example.com/openapi.json

# Test a local server
st run http://localhost:8080/openapi.yaml

# Test a file (still needs a running server for the URL)
st run openapi.yaml --url http://localhost:8080

A typical output looks like:

================================ Schemathesis test session starts =================================
Schema location: http://localhost:8080/openapi.json
Base URL:        http://localhost:8080
Specification version: Open API 3.0.3
Workers: 1
Collected API operations: 12

GET /api/users ✓ 100 samples
POST /api/users ✓ 100 samples
GET /api/users/{userId} ✓ 100 samples
PUT /api/users/{userId} ✓ 100 samples
DELETE /api/users/{userId} ✓ 100 samples
GET /api/orders ✓ 100 samples
POST /api/orders F

=================================================================================================
FAILED

POST /api/orders
1. Test Case ID: abcd1234

- Response violates schema
    
    Check           : response_schema_conformance
    Request         : POST /api/orders
                      Body: {"customerId": -1, "productId": "   ", "quantity": 0}
    Response        : 
      Status code   : 201
      Body          : {}
    
    Response is missing required field: 'id'

Schemathesis found that sending quantity: 0 created an order (201) with an empty response body, violating the documented response schema. That's a real bug — a zero-quantity order that shouldn't be created, and when it is, doesn't return the promised data.

Fuzzing Strategies

By default Schemathesis generates random valid inputs. But it also has specific strategies designed to probe for common vulnerability classes:

# Use the default strategy
st run http://localhost:8080/openapi.yaml

# Include negative testing — send deliberately invalid data
st run http://localhost:8080/openapi.yaml --checks all

# Focus on specific checks
st run http://localhost:8080/openapi.yaml \
  --checks not_a_server_error \
  --checks response_schema_conformance \
  --checks content_type_conformance

# Increase sample count
st run http://localhost:8080/openapi.yaml --count 500

# Use multiple workers for parallel testing
st run http://localhost:8080/openapi.yaml --workers 4

Available checks:

not_a_server_error — no 5xx responses (default)
response_schema_conformance — response body matches documented schema
content_type_conformance — Content-Type header matches spec
accept_header_conformance — server handles Accept headers correctly
response_headers_conformance — required response headers are present
use_after_free — stateful: resource operations after deletion
ensure_resource_availability — stateful: created resources can be retrieved
negative_data_rejection — invalid input gets 4xx, not 5xx or 2xx

Authentication

Many endpoints require auth. Pass credentials on the command line:

# Bearer token
st run http://localhost:8080/openapi.yaml \
  --auth-type bearer \
  --auth "your-token-here"

# Basic auth
st run http://localhost:8080/openapi.yaml \
  --auth-type basic \
  --auth "username:password"

# Custom header
st run http://localhost:8080/openapi.yaml \
  --header "Authorization: Bearer your-token" \
  --header "X-API-Key: your-key"

For OAuth flows or session-based auth, you need the Python API.

Python API: Full Control

The Python API gives you complete control over test generation, filtering, and assertions:

import schemathesis
from schemathesis import Case
import requests

# Load schema
schema = schemathesis.from_uri("http://localhost:8080/openapi.yaml")

# Basic test — all operations, default checks
@schema.parametrize()
def test_api(case: Case):
    response = case.call()
    case.validate_response(response)

Run it with pytest:

pytest test_api.py -v

Custom Authentication

import schemathesis
from schemathesis import Case
import pytest
import requests

@pytest.fixture(scope="session")
def auth_token():
    response = requests.post(
        "http://localhost:8080/auth/token",
        json={
            "client_id": "test-client",
            "client_secret": "test-secret",
            "grant_type": "client_credentials"
        }
    )
    return response.json()["access_token"]

schema = schemathesis.from_uri("http://localhost:8080/openapi.yaml")

@schema.parametrize()
def test_api_authenticated(case: Case, auth_token):
    # Add auth to every generated request
    case.headers = case.headers or {}
    case.headers["Authorization"] = f"Bearer {auth_token}"
    
    response = case.call()
    case.validate_response(response)

Filtering Operations

Test only specific endpoints or methods:

# Test only POST endpoints
@schema.parametrize(method="POST")
def test_post_endpoints(case: Case):
    response = case.call()
    case.validate_response(response)

# Test specific paths
@schema.parametrize(endpoint="/api/orders.*")
def test_orders_api(case: Case):
    response = case.call()
    case.validate_response(response)

# Exclude specific operations
@schema.parametrize(
    endpoint="^(?!.*/admin).*$"  # Exclude /admin endpoints
)
def test_non_admin_api(case: Case):
    response = case.call()
    case.validate_response(response)

Custom Assertions

Beyond the built-in checks, add domain-specific assertions:

@schema.parametrize()
def test_api_business_rules(case: Case):
    response = case.call()
    
    # Standard schema validation
    case.validate_response(response)
    
    # Custom assertions
    if response.status_code == 200:
        data = response.json()
        
        # IDs should never be negative
        if "id" in data:
            assert int(data["id"]) > 0, f"ID {data['id']} is not positive"
        
        # Timestamps should be valid ISO 8601
        if "createdAt" in data:
            from datetime import datetime
            try:
                datetime.fromisoformat(data["createdAt"].replace("Z", "+00:00"))
            except ValueError:
                raise AssertionError(f"Invalid timestamp: {data['createdAt']}")
    
    if response.status_code == 422:
        # Validation errors should have an 'errors' field
        error_body = response.json()
        assert "errors" in error_body or "detail" in error_body, \
            f"422 response missing error detail: {error_body}"

Stateful Testing

Standard property-based testing generates each request independently, which misses a whole class of bugs that only appear when operations are sequenced. Stateful testing uses links between operations to build realistic request chains.

Schemathesis uses OpenAPI's links feature to define these chains:

# openapi.yaml
paths:
  /orders:
    post:
      operationId: createOrder
      responses:
        '201':
          description: Order created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Order'
          links:
            getOrder:
              operationId: getOrder
              parameters:
                orderId: '$response.body#/id'
            cancelOrder:
              operationId: cancelOrder
              parameters:
                orderId: '$response.body#/id'
  
  /orders/{orderId}:
    get:
      operationId: getOrder
      parameters:
        - name: orderId
          in: path
          required: true
          schema:
            type: string
    delete:
      operationId: cancelOrder
      parameters:
        - name: orderId
          in: path
          required: true
          schema:
            type: string

Enable stateful testing:

st run http://localhost:8080/openapi.yaml --stateful=links

# Increase the depth of state chains
st run http://localhost:8080/openapi.yaml \
  --stateful=links \
  --stateful-recursion-limit 5

Or in Python:

schema = schemathesis.from_uri(
    "http://localhost:8080/openapi.yaml",
    stateful=schemathesis.Stateful.links
)

@schema.parametrize()
def test_stateful(case: Case):
    response = case.call()
    case.validate_response(response)

Stateful testing is where Schemathesis often finds the most interesting bugs:

Creating a resource returns an ID that, when used to fetch the resource, returns 404
Deleting a resource, then fetching it returns 200 with stale data instead of 404
A resource's state changes prevent subsequent valid operations from succeeding

GraphQL Testing

Schemathesis also supports GraphQL APIs:

st run http://localhost:8080/graphql --app=graphql

Python API:

schema = schemathesis.graphql.from_url("http://localhost:8080/graphql")

@schema.parametrize()
def test_graphql(case: Case):
    response = case.call()
    # GraphQL always returns 200, so check for errors in the body
    if response.status_code == 200:
        data = response.json()
        assert "errors" not in data or case.data.get("expected_error"), \
            f"Unexpected GraphQL errors: {data.get('errors')}"

Schemathesis introspects the schema, generates valid and invalid queries, and tests that the server handles them without 5xx errors.

CI/CD Integration

GitHub Actions

name: Schemathesis API Tests

on:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 2 * * *'  # Nightly full run

jobs:
  schemathesis:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install schemathesis
          pip install -r requirements.txt

      - name: Start API server
        run: |
          python -m uvicorn app.main:app --host 0.0.0.0 --port 8080 &
          python -c "
          import urllib.request, time
          for _ in range(30):
              try:
                  urllib.request.urlopen('http://localhost:8080/health')
                  break
              except:
                  time.sleep(1)
          "

      - name: Run Schemathesis (fast, PR checks)
        if: github.event_name == 'pull_request'
        env:
          API_TOKEN: ${{ secrets.TEST_API_TOKEN }}
        run: |
          st run http://localhost:8080/openapi.json \
            --checks all \
            --count 50 \
            --header "Authorization: Bearer $API_TOKEN" \
            --junit-xml schemathesis-results.xml \
            --workers 2

      - name: Run Schemathesis (deep, nightly)
        if: github.event_name == 'schedule'
        env:
          API_TOKEN: ${{ secrets.TEST_API_TOKEN }}
        run: |
          st run http://localhost:8080/openapi.json \
            --checks all \
            --count 1000 \
            --stateful=links \
            --header "Authorization: Bearer $API_TOKEN" \
            --junit-xml schemathesis-results.xml \
            --workers 4

      - name: Upload test results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: schemathesis-results
          path: schemathesis-results.xml

      - name: Publish results
        uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: schemathesis-results.xml
          check_name: Schemathesis API Tests

Docker-Based CI

- name: Run Schemathesis via Docker
  run: |
    docker run --rm \
      --network host \
      -e SCHEMATHESIS_TOKEN=${{ secrets.TEST_API_TOKEN }} \
      schemathesis/schemathesis:stable \
      run http://localhost:8080/openapi.json \
        --checks all \
        --header "Authorization: Bearer $SCHEMATHESIS_TOKEN"

Reproducing Failures

When Schemathesis finds a bug, it outputs the minimal reproducing case. Use the --seed flag to reproduce the same random sequence:

# Reproduce a specific failing run
st run http://localhost:8080/openapi.yaml --seed 12345

The Python API also supports saving and replaying found issues as pytest fixtures, making it easy to add a non-property-based regression test for each bug found:

# After schemathesis finds: POST /orders with {"quantity": 0}
# Add a regression test:
def test_zero_quantity_order_rejected():
    response = requests.post(
        "http://localhost:8080/orders",
        json={"customerId": "user-1", "productId": "prod-1", "quantity": 0},
        headers={"Authorization": f"Bearer {auth_token}"}
    )
    assert response.status_code == 422, \
        "Zero quantity order should be rejected with 422"

Integrating with Hypothesis

Schemathesis is built on top of Hypothesis, Python's property-based testing framework. You can use Hypothesis directly for more advanced strategies:

from hypothesis import given, settings, HealthCheck
import schemathesis
from schemathesis.extra.pytest_plugin import schema

@given(case=schema.parametrize())
@settings(
    max_examples=200,
    suppress_health_check=[HealthCheck.too_slow],
    deadline=5000  # 5 second deadline per example
)
def test_with_hypothesis(case):
    response = case.call()
    case.validate_response(response)
    
    # Hypothesis shrinks failing examples automatically
    # The simplest failing case is reported, not the first

Hypothesis's shrinking means that when a bug is found, the framework automatically minimizes the input to the smallest case that reproduces it. Instead of {"name": "aGxS3kPqwZ", "quantity": 847, ...}, you get {"name": "a", "quantity": 0}.

What Schemathesis Won't Find

Property-based testing has limits. Schemathesis will tell you when your server violates its own spec. It won't tell you when the spec is wrong — when the documented behavior is itself incorrect.

It also won't verify business logic beyond what the spec encodes. If your spec says POST /transfers returns 201 and {"status": "pending"}, but the actual money movement fails silently, Schemathesis won't catch that. For business logic correctness, you need tests that understand your domain.

Think of Schemathesis as a first line of defense: it ensures your server is robust against the full input space your spec permits. Conventional tests handle intent. Together, they cover substantially more ground than either alone.

Summary

Schemathesis's core proposition is compelling: your API spec already describes what inputs your server should accept and what outputs it should produce. Property-based testing treats that spec as executable, generating thousands of test cases from it automatically.

The bugs it finds — malformed responses, missing required fields, 500s on edge-case inputs, state-dependent failures after deletion — are exactly the kind that slip past human-written test suites. Not because the tests are poorly written, but because no one thought to test those specific inputs.

Running Schemathesis in CI on every pull request, with a deeper nightly run for stateful chains, gives you a continuous automated safety net that scales with your API's complexity without requiring proportional growth in test maintenance.

HelpMeTest can complement this setup by providing continuous monitoring against your production API, catching regressions that only appear under real traffic conditions — but Schemathesis is the right tool for the pre-deploy phase where you want to probe the full input space before anything reaches users.