API Testing

Schemathesis: OpenAPI and GraphQL Fuzzing Guide

HelpMeTest

HelpMeTest

— 5 min read
Schemathesis: OpenAPI and GraphQL Fuzzing Guide

Schemathesis takes your OpenAPI or GraphQL schema and automatically generates hundreds of test cases from it — including edge cases your manual test suite never thought of. It finds server crashes, unhandled exceptions, and schema violations that sit invisible in hand-written tests. This guide covers everything from basic setup to stateful multi-step fuzzing.

What Schemathesis Does Differently

Most API testing tools test what you explicitly tell them to test. Schemathesis tests what your schema says is possible. If your OpenAPI spec says a field accepts a string, Schemathesis sends empty strings, strings with null bytes, 100,000-character strings, and Unicode edge cases — and checks whether your API either returns a valid response or a documented error code.

The critical insight: your schema is a contract with your users. If your spec says the endpoint accepts a particular input, it must handle it gracefully. Schemathesis verifies this automatically.

Installation

pip install schemathesis

# Or with all optional features
pip install schemathesis[all]

Verify installation:

st --version

Basic Fuzzing Run

If your API is running locally with an OpenAPI spec at /api/openapi.json:

st run http://localhost:8080/api/openapi.json

Against a remote API:

st run https://api.example.com/openapi.json \
  --header "Authorization: Bearer $API_TOKEN"

Schemathesis will:

  1. Parse the schema
  2. Generate test cases for every endpoint and method
  3. Check each response for server errors (5xx), schema violations, and malformed responses
  4. Report failures with full request/response details for reproduction

Understanding the Output

A typical Schemathesis run shows:

POST /api/users .......F.......
FAILED POST /api/users
ResponseSchemaConformance: Response violates schema

Response: {"id": 123, "email": null}
Expected: {"id": "integer", "email": "string"}

Request: {"name": "", "email": "test@example.com"}

The failure tells you:

  • Which endpoint failed
  • What kind of failure (schema violation, server error, etc.)
  • The exact request that triggered it
  • The exact response that was wrong

This is immediately actionable — no guessing, no investigation needed.

Filtering Tests

Run only specific endpoints:

# Only GET endpoints
st run http://localhost:8080/openapi.json --method GET

<span class="hljs-comment"># Only specific path
st run http://localhost:8080/openapi.json --endpoint /api/users

<span class="hljs-comment"># Specific path pattern
st run http://localhost:8080/openapi.json --endpoint <span class="hljs-string">"/api/users/{user_id}"

<span class="hljs-comment"># Exclude certain endpoints
st run http://localhost:8080/openapi.json --exclude-endpoint /api/health

Authentication

Pass static auth headers:

st run http://localhost:8080/openapi.json \
  --header "Authorization: Bearer $TOKEN"

For APIs with authentication endpoints, use auth in a YAML configuration:

# schemathesis.yml
schema: http://localhost:8080/openapi.json
checks:
  - not_a_server_error
  - response_schema_conformance
auth:
  type: bearer
  token: "${API_TOKEN}"
st run --config schemathesis.yml

GraphQL Fuzzing

Schemathesis supports GraphQL introspection-based fuzzing:

st run http://localhost:8080/graphql --app graphql

For GraphQL with authentication:

st run http://localhost:8080/graphql \
  --app graphql \
  --header "Authorization: Bearer $TOKEN"

Schemathesis will introspect your schema, discover all queries and mutations, and generate test inputs for each. It catches:

  • Queries that return 500 instead of null for missing data
  • Mutations that accept invalid inputs without error
  • Types that don't match the schema definition

Python API for Custom Scenarios

For complex authentication flows or stateful testing, use the Python API:

import schemathesis
from schemathesis import Case

schema = schemathesis.from_uri("http://localhost:8080/openapi.json")

@schema.parametrize()
def test_api(case: Case):
    # Add auth before each request
    response = case.call(headers={"Authorization": "Bearer test-token"})
    case.validate_response(response)

Run with pytest:

pytest test_api.py -v

This integrates Schemathesis into your standard pytest suite, enabling coverage tracking and parallel execution.

Stateful Testing

Schemathesis can chain requests — creating resources with POST, then accessing them with GET, then deleting with DELETE — to find bugs in stateful flows:

st run http://localhost:8080/openapi.json --stateful=links

The --stateful=links flag uses OpenAPI links definitions to chain related endpoints. When your spec defines that the POST /users response contains a link to GET /users/{id}, Schemathesis will automatically create a user and then fetch it with a generated ID.

For explicit stateful scenarios in Python:

import schemathesis
from schemathesis.stateful import Stateful

schema = schemathesis.from_uri("http://localhost:8080/openapi.json")

@schema.parametrize(stateful=Stateful.links)
def test_stateful(case):
    response = case.call_and_validate()

Custom Checks

Define your own assertions beyond schema conformance:

import schemathesis
from schemathesis import Case, Response

schema = schemathesis.from_uri("http://localhost:8080/openapi.json")

@schemathesis.check
def no_debug_info(response: Response, case: Case) -> None:
    """Ensure no debug information leaks in responses."""
    body = response.text
    assert "stack trace" not in body.lower(), "Stack trace leaked in response"
    assert "sql" not in body.lower(), "SQL query leaked in response"
    assert "password" not in body.lower(), "Password leaked in response"

@schema.parametrize()
def test_api(case: Case):
    response = case.call()
    case.validate_response(response, checks=[no_debug_info])

CI/CD Integration

GitHub Actions:

name: API Fuzzing
on: [push, pull_request]

jobs:
  fuzz:
    runs-on: ubuntu-latest
    services:
      api:
        image: your-api:latest
        ports:
          - 8080:8080

    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install Schemathesis
        run: pip install schemathesis
      - name: Wait for API
        run: |
          until curl -f http://localhost:8080/health; do sleep 2; done
      - name: Run Schemathesis
        run: |
          st run http://localhost:8080/openapi.json \
            --checks all \
            --hypothesis-max-examples 100 \
            --junit-xml=results.xml
      - name: Upload results
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: schemathesis-results
          path: results.xml

Controlling Test Generation

Schemathesis uses Hypothesis internally for property-based testing. You can control the test generation:

# More thorough testing (more examples)
st run http://localhost:8080/openapi.json \
  --hypothesis-max-examples 500

<span class="hljs-comment"># Faster CI runs (fewer examples)
st run http://localhost:8080/openapi.json \
  --hypothesis-max-examples 50

<span class="hljs-comment"># Set timeout
st run http://localhost:8080/openapi.json \
  --hypothesis-deadline 5000

Common Bugs Schemathesis Finds

Server errors on edge case inputs: Your API returns 200 for normal inputs but 500 when the string contains a null byte. Manual tests never send null bytes. Schemathesis does.

Schema mismatches: Your spec says a field is required, but your code sometimes omits it in responses. Schemathesis catches this.

Type coercion bugs: Your API accepts a string where you expected a number and crashes instead of returning 400. Schemathesis sends strings, integers, booleans, and nulls for every field.

Missing error handling: An endpoint crashes on empty arrays when your spec says arrays are valid. Schemathesis sends empty arrays for every array field.

Reproducing Failures

When Schemathesis finds a bug, it outputs a seed for reproduction:

Falsifying example:
  case = Case(path='/api/users', method='POST', body={'name': '', 'email': ''})

You can reproduce this failure with:
  --hypothesis-seed=12345

Save the seed to always reproduce the same test run:

st run http://localhost:8080/openapi.json --hypothesis-seed=12345

What to Do With Schemathesis Results

  1. Server errors (5xx): Always bugs. Your API must never return 5xx for valid inputs (inputs that match your schema). Fix the error handling.
  2. Schema violations: Your code doesn't match your spec. Either fix the code or fix the spec — they must agree.
  3. Slow responses: Not a Schemathesis check by default, but you can add custom timing checks.
  4. Security findings: Schemathesis has security-focused checks (path traversal, SQL injection patterns). Enable with --checks security.

Schemathesis finds a category of bugs — schema-invalid inputs causing unexpected behavior — that manual API testing systematically misses. Running it as part of CI adds a fuzzing layer that continuously verifies your API handles everything it claims to handle.

Read more