Schemathesis: Property-Based API Testing from OpenAPI Specs

Manual API tests cover the happy path and a handful of error cases you thought to write down. Property-based testing covers the space you didn't think about — and that's exactly where real bugs live.

Schemathesis is a Python-based tool that reads your OpenAPI (or GraphQL) spec and automatically generates hundreds of test cases per endpoint. It doesn't need you to write a single test case manually. Give it your spec, point it at a running server, and it will hammer every endpoint with valid-but-unexpected inputs until something breaks.

How Schemathesis Works

Unlike fuzzing tools that send random garbage, Schemathesis generates valid inputs according to your schema. If your spec says a field is an integer between 1 and 100, Schemathesis will try 1, 100, 0, -1, 101, and boundary-adjacent values. If a field is a string with a pattern, it generates strings that match the pattern — and strings that almost match.

This is the property-based testing approach, borrowed from tools like Hypothesis (which Schemathesis uses under the hood). The idea: define the properties your API must satisfy (e.g., "always returns JSON", "never returns 500 for valid input") and let the framework find counterexamples.

Installation and Basic Usage

pip install schemathesis

Run against a live server:

st run https://api.example.com/openapi.json

Or against a local spec file:

st run ./openapi.yaml --base-url http://localhost:8080

Schemathesis will test every endpoint it finds, report status codes, response times, and flag anything that looks wrong — 5xx errors, malformed JSON responses, schema violations on the response side.

Writing Schemathesis Tests in Python

For more control, use the Python API:

import schemathesis

schema = schemathesis.from_path("./openapi.yaml", base_url="http://localhost:8080")

@schema.parametrize()
def test_api(case):
    response = case.call()
    case.validate_response(response)

The @schema.parametrize() decorator generates one test case per endpoint per HTTP method. case.validate_response() checks the response against the spec — wrong status codes, missing required fields, wrong types all get caught here.

Run it with pytest:

pytest test_api.py -v

Stateful Testing

Single-endpoint tests miss a class of bugs: those that only appear when endpoints interact. You create a resource, update it, then delete it — and the sequence matters.

Schemathesis supports stateful testing via OpenAPI links. Links define how the output of one operation feeds into the input of another:

paths:
  /users:
    post:
      operationId: createUser
      responses:
        '201':
          content:
            application/json:
              schema:
                properties:
                  id:
                    type: integer
          links:
            GetUserById:
              operationId: getUser
              parameters:
                userId: '$response.body#/id'

With links defined, Schemathesis can chain operations — create a user, then call getUser with the returned ID, then call deleteUser. This catches bugs like:

• Resources that can be created but not retrieved
• Delete endpoints that return 200 but leave the resource in the database
• Update endpoints that don't validate ownership

Enable stateful testing:

st run ./openapi.yaml --base-url http://localhost:8080 --stateful=links

Filtering and Targeting

You don't always want to test everything. Common options:

# Test only specific endpoints
st run ./openapi.yaml --base-url http://localhost:8080 --endpoint /users

# Test only GET requests
st run ./openapi.yaml --base-url http://localhost:8080 --method GET

# Exclude endpoints matching a pattern
st run ./openapi.yaml --base-url http://localhost:8080 --exclude-endpoint /health

For Python tests, use marks:

@schema.parametrize()
@pytest.mark.filterwarnings("ignore")
def test_read_endpoints(case):
    if case.method != "GET":
        pytest.skip()
    response = case.call()
    assert response.status_code < 500

CI Integration

Add Schemathesis to your GitHub Actions pipeline:

name: API Contract Tests

on: [push, pull_request]

jobs:
  schemathesis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Start API server
        run: docker-compose up -d api
        
      - name: Wait for API
        run: npx wait-on http://localhost:8080/health

      - name: Run Schemathesis
        uses: schemathesis/action@v1
        with:
          schema: 'http://localhost:8080/openapi.json'
          base-url: 'http://localhost:8080'
          args: '--checks all --stateful=links'

The --checks all flag enables every built-in check: response schema validation, not_a_server_error, status_code_conformance, content_type_conformance, and response_headers_conformance.

Custom Checks

Built-in checks catch many issues, but you can add domain-specific assertions:

from schemathesis import check

@check
def no_deprecated_fields(response, case):
    if response.headers.get("Content-Type", "").startswith("application/json"):
        data = response.json()
        assert "legacy_id" not in data, "Response contains deprecated 'legacy_id' field"

@schema.parametrize()
def test_api(case):
    response = case.call()
    case.validate_response(response)

This runs your custom check on every generated test case automatically.

Reading the Output

Schemathesis produces structured output. When it finds a failure, it prints the minimal reproducing example:

FAILED POST /users
  Request:
    Body: {"name": "", "email": "a@b.com", "age": -1}
  Response:
    Status: 500
    Body: {"error": "Internal Server Error"}

Falsifying example:
  case = Case(method='POST', path='/users', body={"name": "", "email": "a@b.com", "age": -1})

That empty string for name and negative age are the minimal inputs that triggered a 500. Your handler doesn't validate these — Schemathesis found it in seconds.

When to Use Schemathesis

Schemathesis is not a replacement for integration tests. It's a complement. Use it to:

• Catch validation gaps (inputs your handler doesn't reject that it should)
• Find unexpected 500s from edge case inputs
• Verify your response bodies match your spec
• Run regression checks when specs change

It's particularly valuable when your OpenAPI spec is the contract between frontend and backend teams. If the spec says a field is nullable, Schemathesis will send null — and if your backend crashes, you've found a contract violation before it hits production.

Monitoring Beyond the Build

Schemathesis runs in CI and tells you about regressions before deploy. But after deploy, your API is running 24/7 with real traffic and real edge cases you haven't anticipated.

HelpMeTest provides continuous monitoring for your API endpoints in production — running real test scenarios around the clock and alerting you when behavior changes. Pair property-based testing in CI with continuous monitoring in production to cover both the spec-time and run-time dimensions of API reliability.