API Testing

Schemathesis: Property-Based API Testing From OpenAPI Specs

HelpMeTest

HelpMeTest

— 5 min read
Schemathesis: Property-Based API Testing From OpenAPI Specs

Property-based testing for APIs works like this: instead of writing POST /users with {"name": "Alice"} and checking the response, you define what properties the API should always satisfy, and let the test tool generate hundreds of inputs automatically.

Schemathesis does exactly this for REST APIs using your OpenAPI spec as the source of truth. It generates valid (and intentionally invalid) requests from your schema and checks that the server responds correctly. This approach finds bugs that manually-written test cases miss — because humans test the happy path.

What Schemathesis Actually Tests

Given an OpenAPI spec, Schemathesis:

  1. Generates valid requests that conform to your schema — correct types, required fields present, values within constraints
  2. Generates invalid requests with out-of-range values, wrong types, missing required fields
  3. Checks response invariants: any 5xx response is a bug, response headers should be consistent, response bodies should match the declared schema

By default, Schemathesis checks:

  • No 5xx responses for any valid input (server errors = bugs)
  • Response content-type matches what's declared
  • Response schema matches the declared response schema
  • Server doesn't crash with invalid input

Installing Schemathesis

pip install schemathesis

# Or with all extras (for auth, ASGI testing, etc.)
pip install schemathesis[all]

Running Against a Live API

The simplest approach: point Schemathesis at your running server.

# Run against a live server
schemathesis run http://localhost:8000/openapi.json

# Specify which operations to test
schemathesis run http://localhost:8000/openapi.json \
  --endpoint /users \
  --method POST

# Run with more test cases per operation (default: 100)
schemathesis run http://localhost:8000/openapi.json \
  --hypothesis-max-examples=500

# Save results to a file
schemathesis run http://localhost:8000/openapi.json \
  --report schemathesis-results.json

Running Via Python Tests

For more control, integrate Schemathesis into your pytest suite:

# test_api_schema.py
import schemathesis
from schemathesis import DataGenerationMethod

# Load schema from file or URL
schema = schemathesis.from_path('openapi.json', base_url='http://localhost:8000')

# Test all operations
@schema.parametrize()
def test_api(case):
    response = case.call()
    case.validate_response(response)

# Test with authentication
@schema.parametrize()
def test_authenticated_api(case):
    response = case.call(headers={'Authorization': 'Bearer test-token'})
    case.validate_response(response)

Run with pytest:

pytest test_api_schema.py -v \
  --hypothesis-settings-max-examples=200

Custom Checks

The default checks catch server errors and schema violations. Add custom checks for your business logic:

import schemathesis
from schemathesis import Check
from requests import Response

schema = schemathesis.from_path('openapi.json', base_url='http://localhost:8000')

def check_response_time(response: Response, case) -> None:
    """API must respond within 1 second for any valid request."""
    if response.elapsed.total_seconds() > 1.0:
        raise AssertionError(
            f"Response too slow: {response.elapsed.total_seconds():.2f}s "
            f"for {case.method} {case.path}"
        )

def check_no_stack_traces_in_response(response: Response, case) -> None:
    """Error responses must not include Python stack traces."""
    if response.status_code >= 400:
        body = response.text
        if 'Traceback (most recent call last)' in body:
            raise AssertionError(
                f"Stack trace found in response for {case.method} {case.path}\n"
                f"Response body: {body[:500]}"
            )

def check_content_type_always_json(response: Response, case) -> None:
    """All responses must include application/json content type."""
    content_type = response.headers.get('Content-Type', '')
    if 'application/json' not in content_type:
        raise AssertionError(
            f"Wrong content type: {content_type} for {case.method} {case.path}"
        )

@schema.parametrize()
@schemathesis.check(check_response_time, check_no_stack_traces_in_response, check_content_type_always_json)
def test_api_with_custom_checks(case):
    response = case.call()
    case.validate_response(response)

Stateful Testing: Multi-Step Scenarios

Stateful testing chains multiple API calls — creating a resource, reading it, updating it, deleting it:

# test_stateful.py
import schemathesis
from schemathesis.stateful import StatefulApp

schema = schemathesis.from_path('openapi.json', base_url='http://localhost:8000')

# Define links between operations using OpenAPI $links
# Or use the stateful runner directly

@schema.parametrize(stateful=True)
def test_stateful_api(case):
    """
    Schemathesis will chain operations based on `links` defined in your
    OpenAPI spec, or based on response data that matches path parameters.
    """
    response = case.call()
    case.validate_response(response)

For explicit control over stateful flows, define scenarios:

import schemathesis
import pytest

schema = schemathesis.from_path('openapi.json', base_url='http://localhost:8000')

@pytest.fixture(scope='module')
def api_client():
    return schema._base_url

def test_user_lifecycle():
    """Test the full user lifecycle: create → read → update → delete."""
    import requests
    base = 'http://localhost:8000'
    token = get_test_token()
    headers = {'Authorization': f'Bearer {token}'}

    # Use Schemathesis to generate a valid user payload
    create_op = schema['/users']['POST']
    for case in create_op.explicit_header_parameters_mark.iters():
        create_payload = case.body
        break  # Take the first generated payload

    # Step 1: Create
    response = requests.post(f'{base}/users', json=create_payload, headers=headers)
    assert response.status_code == 201
    user_id = response.json()['id']

    # Step 2: Read
    response = requests.get(f'{base}/users/{user_id}', headers=headers)
    assert response.status_code == 200
    assert response.json()['id'] == user_id

    # Step 3: Update (use Schemathesis to generate a valid update payload)
    response = requests.patch(
        f'{base}/users/{user_id}',
        json={'name': 'Updated Name'},
        headers=headers
    )
    assert response.status_code == 200

    # Step 4: Delete
    response = requests.delete(f'{base}/users/{user_id}', headers=headers)
    assert response.status_code == 204

    # Verify deletion
    response = requests.get(f'{base}/users/{user_id}', headers=headers)
    assert response.status_code == 404

Testing Against FastAPI and ASGI Apps

Schemathesis supports testing ASGI apps directly (no network, faster):

# test_fastapi.py
import schemathesis
from fastapi.testclient import TestClient
from myapp.main import app

# Test directly via ASGI without network
schema = schemathesis.from_asgi('/openapi.json', app)

@schema.parametrize()
def test_api(case):
    response = case.call_asgi()  # No network call
    case.validate_response(response)

This runs at Python function call speed — much faster than HTTP tests.

Filtering Operations

For large APIs, focus Schemathesis on specific areas:

# Test only POST operations
@schema.parametrize(method='POST')
def test_write_operations(case):
    response = case.call()
    case.validate_response(response)

# Test only user-related endpoints
@schema.parametrize(endpoint='/users')
def test_user_endpoints(case):
    response = case.call()
    case.validate_response(response)

# Exclude health check and metrics endpoints
@schema.parametrize(endpoint='~/(health|metrics)')
def test_non_operational_endpoints(case):
    response = case.call()
    case.validate_response(response)

Reading Schemathesis Output

When Schemathesis finds a bug, it shows:

FAILED test_api[GET /users/{user_id}]
  Response violates schema
  Falsifying example: case = Case(
    path_parameters={'user_id': '\x00'},  # Null byte in user_id
    headers={...}
  )
  Response: 500 Internal Server Error
  {
    "error": "invalid input value for type uuid: ...",
    "traceback": "..."
  }

This tells you exactly what input triggered the bug (null byte in user_id) and what went wrong (500 instead of 400). The input is the minimum that reproduces the failure — Schemathesis shrinks the example automatically.

CI Integration

# .github/workflows/schemathesis.yml
name: API Property Tests

on: [push, pull_request]

jobs:
  schemathesis:
    runs-on: ubuntu-latest

    services:
      api:
        image: myapp:${{ github.sha }}
        ports:
          - 8000:8000
        options: --health-cmd "curl -sf http://localhost:8000/health"

    steps:
      - uses: actions/checkout@v4

      - name: Wait for API
        run: |
          /usr/local/bin/await 'curl -sf http://localhost:8000/health'

      - name: Run Schemathesis
        uses: schemathesis/action@v1
        with:
          schema: http://localhost:8000/openapi.json
          args: >
            --checks all
            --hypothesis-max-examples=100
            --report schemathesis-report.json

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: schemathesis-report
          path: schemathesis-report.json

What Schemathesis Finds (and What It Doesn't)

Finds well:

  • Server errors (5xx) triggered by unusual but valid inputs
  • Schema violations where the response doesn't match the OpenAPI spec
  • Missing error handling for null bytes, unicode edge cases, boundary values
  • Type coercion bugs (sending "1" where 1 is expected)

Doesn't find:

  • Business logic correctness (it doesn't know what the API should do)
  • Race conditions (single-threaded test generation)
  • Authorization bugs (unless you configure specific auth scenarios)
  • Performance issues under load

Schemathesis is a complement to hand-written API tests, not a replacement. Use it to find the bugs you didn't think to test for. Use Postman or pytest for the scenarios you know matter.

HelpMeTest adds behavioral end-to-end testing on top of API testing — verifying that correct API responses produce correct behavior in your application. Start free →

Read more

Start now free