Property-Based Testing of REST APIs from OpenAPI Specs with Schemathesis

You can't write a test for every possible input. A string field accepts billions of values. A numeric parameter has edge cases you haven't considered. An API that looks solid against your 20 hand-written test cases might crash on the 21st input — the one a real user will eventually send.

Property-based testing changes the equation. Instead of specifying inputs, you specify properties that must hold for all inputs. The framework generates the inputs. Schemathesis applies this idea directly to OpenAPI specs, turning your API contract into a test generator.

How Schemathesis Works

Schemathesis reads your OpenAPI spec and automatically generates HTTP requests for every endpoint. For each operation, it:

1. Parses the parameter schemas (path params, query params, request body)
2. Generates random valid inputs according to those schemas
3. Sends hundreds or thousands of requests to your running API
4. Checks that responses conform to the spec and that the server doesn't crash

The key insight: your spec already defines what valid inputs look like. Schemathesis uses that definition to explore the input space systematically, including edge cases you wouldn't think to test: empty strings, very long strings, Unicode characters, negative numbers, null values, and schema boundary conditions.

Installation and Basic Usage

pip install schemathesis

Run against a live API:

st run http://localhost:8000/openapi.yaml --url http://localhost:8000

Or against a hosted spec with a separate target:

st run https://api.example.com/openapi.yaml --url https://staging.api.example.com

Schemathesis runs and reports any failures:

FAILURES

POST /api/users
Body: {"email": "", "name": "A" * 10001}
Response: 500 Internal Server Error

Reproduced with:
curl -X POST http://localhost:8000/api/users \
  -H "Content-Type: application/json" \
  -d '{"email": "", "name": "AAAA..."}'

This output tells you exactly what request triggered the bug, with a curl command to reproduce it. You didn't write this test case — Schemathesis found it by exploring the input space.

What Schemathesis Checks Automatically

By default, Schemathesis runs several checks on every response:

• not_a_server_error — the server must not return 5xx responses to valid inputs
• status_code_conformance — response status codes must be documented in the spec
• content_type_conformance — response Content-Type must match the spec
• response_schema_conformance — response bodies must validate against their schema

Any violation fails the test. You get a concrete failing example and a full traceback.

Writing Stateful Tests

Stateless property tests hit each endpoint independently. For APIs with state — create a resource, then retrieve it — you need stateful testing. Schemathesis supports OpenAPI links to chain requests:

paths:
  /users:
    post:
      operationId: createUser
      responses:
        "201":
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/User"
          links:
            GetUserById:
              operationId: getUser
              parameters:
                userId: "$response.body#/id"
  /users/{userId}:
    get:
      operationId: getUser
      parameters:
        - name: userId
          in: path
          schema:
            type: string

Run with stateful testing enabled:

st run openapi.yaml --url http://localhost:8000 --stateful=links

Schemathesis creates a user, extracts the id from the response, and uses it to fetch that user — testing the complete lifecycle automatically.

Authentication

Most APIs require authentication. Pass headers or tokens:

# Bearer token
st run openapi.yaml --url http://localhost:8000 \
  --header <span class="hljs-string">"Authorization: Bearer $TOKEN"

<span class="hljs-comment"># API key
st run openapi.yaml --url http://localhost:8000 \
  --header <span class="hljs-string">"X-API-Key: your-key-here"

For OAuth or session-based auth, use Schemathesis in Python to hook into the auth flow:

import schemathesis

schema = schemathesis.from_uri("http://localhost:8000/openapi.yaml")

@schema.parametrize()
def test_api(case):
    # Inject auth before each request
    response = case.call(
        headers={"Authorization": f"Bearer {get_token()}"}
    )
    case.validate_response(response)

Running in Python Tests

Schemathesis integrates with pytest:

import schemathesis

schema = schemathesis.from_uri("http://localhost:8000/openapi.yaml")

@schema.parametrize()
def test_api_endpoints(case):
    response = case.call_and_validate()

Running pytest now generates and executes property-based tests for every endpoint. The @schema.parametrize() decorator creates a test case for each operation, and call_and_validate() sends the request and checks all default properties.

Add custom assertions:

@schema.parametrize()
def test_api_endpoints(case):
    response = case.call()
    # Standard checks
    case.validate_response(response)
    # Custom: paginated list responses must include metadata
    if response.status_code == 200 and "items" in response.json():
        assert "total" in response.json(), "Paginated response missing 'total'"
        assert "page" in response.json(), "Paginated response missing 'page'"

CI Integration

# .github/workflows/api-property-tests.yml
name: API Property Tests
on: [pull_request]
jobs:
  schemathesis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Start API server
        run: docker-compose up -d api
      - uses: schemathesis/action@v1
        with:
          schema: "http://localhost:8000/openapi.yaml"
          args: "--checks all --hypothesis-max-examples 100"

--hypothesis-max-examples 100 controls how many inputs Schemathesis generates per endpoint. Higher values find more bugs but take longer. 100 is a reasonable CI default; run 500+ overnight for deeper coverage.

Shrinking: Minimal Failing Examples

When Schemathesis finds a failure, it "shrinks" the input — systematically reduces it to the smallest input that still triggers the bug. Instead of a 10,000-character string that crashes your server, you get a 1-character string. This makes bugs dramatically easier to debug.

What Schemathesis Catches

Real bugs found by property-based API testing in practice:

• Unhandled null inputs — fields marked optional in the spec cause NullPointerException when actually omitted
• Integer overflow — numeric parameters with no maximum constraint accept values that overflow database columns
• Unicode path parameters — emojis or RTL characters in path params cause routing failures
• Empty string edge cases — required string fields with no minLength accept empty strings that violate business logic
• Schema drift — response bodies include undocumented fields, or omit documented required fields

These bugs exist in production APIs that have extensive hand-written test suites. Property testing finds them because it explores the input space systematically rather than testing what developers think to test.

For complete API test coverage — combining property-based testing with scenario-based end-to-end tests — tools like HelpMeTest let you write behavior tests in plain English on top of deployed APIs. Schemathesis and end-to-end tests are complementary: Schemathesis finds spec violations automatically; scenario tests verify business logic intentionally.