Testing

Property-Based Testing in CI: Running Hypothesis, fast-check, and jqwik in Pipelines

HelpMeTest

19 Aug 2026 — 6 min read

Property-based tests find bugs that example-based tests miss, but they have different CI requirements: they're statistically driven, they need reproducible failures, and they can be slow if misconfigured. This guide covers CI integration patterns for the major property-based testing frameworks across Python (Hypothesis), JavaScript (fast-check), and Java (jqwik).

The Core CI Challenges

Reproducibility: Property-based tests use randomness. When a test fails on CI, you need the exact seed to reproduce it locally. All three frameworks print the seed on failure — your CI must capture and display this output.

Budget management: More runs find more bugs but cost more CI time. Find the right balance between thoroughness and pipeline speed.

Failure databases: Hypothesis maintains a database of previously failing examples and reruns them first on every subsequent test run. This is invaluable in CI but requires persistent storage between runs.

Parallelization: Property-based tests are CPU-bound and embarrassingly parallel. Run them with as many cores as your CI runner has.

Hypothesis (Python) CI Setup

Persisting the Hypothesis Database

Hypothesis maintains a database at .hypothesis/ in your project root. If CI doesn't persist this directory across runs, Hypothesis forgets previously discovered failures and must rediscover them.

GitHub Actions with cache:

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      
      - name: Restore Hypothesis database
        uses: actions/cache@v4
        with:
          path: .hypothesis
          key: hypothesis-db-${{ runner.os }}-${{ hashFiles('**/requirements*.txt') }}
          restore-keys: |
            hypothesis-db-${{ runner.os }}-
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Run property tests
        run: pytest tests/property/ -v --tb=short
      
      - name: Upload Hypothesis database on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: hypothesis-database
          path: .hypothesis/

Configuring Hypothesis for CI

Use Hypothesis profiles to configure different settings for CI vs local:

# conftest.py
from hypothesis import settings, HealthCheck, Phase

# CI profile — more examples, stricter health checks
settings.register_profile("ci",
    max_examples=500,
    suppress_health_check=[HealthCheck.too_slow],
    deadline=5000,  # 5 second deadline per test
    phases=[Phase.explicit, Phase.reuse, Phase.generate, Phase.shrink],
)

# Local development — fast feedback
settings.register_profile("dev",
    max_examples=50,
    suppress_health_check=[HealthCheck.too_slow, HealthCheck.data_too_large],
    deadline=500,
)

# Thorough nightly run
settings.register_profile("nightly",
    max_examples=2000,
    suppress_health_check=[],
    deadline=30000,
)

# Load profile from environment
import os
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))

Set in CI:

- name: Run property tests
  run: pytest tests/property/ -v
  env:
    HYPOTHESIS_PROFILE: ci

Reproducing Hypothesis Failures

When Hypothesis finds a failure, it prints:

Falsifying example: test_encoding_round_trip(
    data=b'\x00\x01\xff'
)
Traceback (most recent call last):
  ...
You can reproduce this example by temporarily adding @reproduce_failure('6.82.2', b'...') as a decorator.

The @reproduce_failure decorator pinpoints the exact failing case for local debugging:

from hypothesis import reproduce_failure

@reproduce_failure('6.82.2', b'AXicY2BgYBIAAQAB')
def test_encoding_round_trip(data):
    encoded = encode(data)
    assert decode(encoded) == data

In CI, always ensure the full failure output (including this decorator) is captured in build logs.

Running Hypothesis Tests in Parallel

# Install pytest-xdist
pip install pytest-xdist

# Run with 4 workers
pytest tests/property/ -n 4

Hypothesis is safe to run in parallel — each worker has its own randomness stream. The database is shared but designed for concurrent access.

fast-check (JavaScript) CI Setup

Capturing Seeds on Failure

fast-check prints the seed when a property fails:

Property failed after 12 tests
{ seed: 1873450219, path: "11:1:0", endOnFailure: true }
Counterexample: ["hello", 42]

Reproduce locally:

fc.assert(
    fc.property(fc.string(), fc.integer(), myProperty),
    { seed: 1873450219, path: "11:1:0", endOnFailure: true }
);

Make the seed visible in CI by ensuring test output isn't truncated:

# GitHub Actions
- name: Run property tests
  run: npx jest --testPathPattern="property" --verbose 2>&1 | tee test-output.txt
  continue-on-error: true

- name: Upload test output on failure
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: fast-check-output
    path: test-output.txt

Configuring Run Count for CI

// jest.config.js or directly in test files

// Base configuration
const FC_NUM_RUNS = process.env.CI ? 500 : 50;

// In tests
fc.assert(
    fc.property(fc.string(), myProperty),
    { 
        numRuns: FC_NUM_RUNS,
        verbose: !!process.env.CI, // More verbose output in CI
    }
);

Or use a global configuration:

// setup.js (loaded by Jest via setupFilesAfterFramework)
if (process.env.CI) {
    fc.configureGlobal({
        numRuns: 500,
        verbose: true,
    });
}

Parallelizing fast-check with Jest Workers

Jest already parallelizes test files across workers. For within-file parallelism:

// Run the same property multiple times in parallel
test('property under parallel load', async () => {
    const numWorkers = 4;
    
    await Promise.all(
        Array.from({ length: numWorkers }, () =>
            fc.assert(
                fc.asyncProperty(fc.string(), async (s) => {
                    const result = await processString(s);
                    return result !== null;
                }),
                { numRuns: 250 } // 250 * 4 workers = 1000 total
            )
        )
    );
});

GitHub Actions Workflow

name: Property Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  property-test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        # Shard across multiple runners for CI speed
        shard: [1, 2, 3, 4]
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - run: npm ci
      
      - name: Run property tests (shard ${{ matrix.shard }}/4)
        run: npx jest --testPathPattern="property" --shard=${{ matrix.shard }}/4
        env:
          CI: true
          FC_NUM_RUNS: 200

jqwik (Java) CI Setup

Maven Configuration

<!-- pom.xml -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
        <properties>
            <!-- jqwik configuration via system properties -->
            <configurationParameters>
                jqwik.tries.default=500
                jqwik.boundary.default=BOUNDARIES
                jqwik.shrinking.default=FULL
                jqwik.reporting.onlyfailures=false
                jqwik.seeds.fixed=false
            </configurationParameters>
        </properties>
        <!-- Parallel execution -->
        <forkCount>4</forkCount>
        <reuseForks>true</reuseForks>
    </configuration>
</plugin>

Gradle Configuration

// build.gradle
test {
    useJUnitPlatform()
    
    systemProperties([
        'jqwik.tries.default': System.getenv('CI') ? '500' : '100',
        'jqwik.shrinking.default': 'FULL',
        'jqwik.reporting.onlyfailures': 'false',
    ])
    
    maxParallelForks = Runtime.runtime.availableProcessors()
}

jqwik Database (Failure Persistence)

jqwik stores previously failing examples in .jqwik-database. Persist it in CI:

- name: Restore jqwik database
  uses: actions/cache@v4
  with:
    path: .jqwik-database
    key: jqwik-db-${{ runner.os }}-${{ hashFiles('pom.xml') }}
    restore-keys: |
      jqwik-db-${{ runner.os }}-

- name: Run tests
  run: mvn test -pl :my-module

- name: Upload jqwik database on failure
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: jqwik-database
    path: .jqwik-database

Reproducing jqwik Failures

When jqwik fails, it prints:

Property [StringReverseProperties:reversingTwiceGivesOriginal] falsified.
  Timestamp: 2024-01-15T10:23:45.678
  Seed:      -4567890123456789
  
  Original Sample:
  Arg[0]: "abc\u0000def"
  
  Shrunk Sample:
  Arg[0]: "\u0000"

Reproduce with the seed attribute:

@Property(seed = "-4567890123456789")
void reversingTwiceGivesOriginal(@ForAll String anyString) {
    // exact reproduction
}

Configuring CI-Specific Profiles

Use JUnit 5 configuration files for environment-specific settings:

# src/test/resources/junit-platform.properties (for CI)
jqwik.tries.default=500
jqwik.database=.jqwik-database
jqwik.shrinking.default=FULL

Or use environment variables in your build script:

# CI script
TRIES=${CI:+500}
TRIES=${TRIES:-100}

mvn test \
  -Djqwik.tries.default=$TRIES \
  -Djqwik.database=.jqwik-database

Nightly vs PR Runs

Structure property test runs by trigger:

name: Tests

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # 2am daily

jobs:
  property-tests-light:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Quick property tests
        run: pytest tests/property/ -v
        env:
          HYPOTHESIS_PROFILE: ci  # 500 examples

  property-tests-thorough:
    if: github.event_name == 'schedule' || github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Thorough property tests
        run: pytest tests/property/ -v
        env:
          HYPOTHESIS_PROFILE: nightly  # 2000 examples
        timeout-minutes: 60

PRs get fast feedback (500 examples). Main branch merges get thorough coverage (2000 examples). Nightly runs get maximum coverage.

Treating Property Failures as Bugs

When a property test fails in CI, treat it like any other test failure:

Don't dismiss it as flaky: Property tests are deterministic given the seed. If it failed, there's a real bug. Reproduce locally using the seed.
Add the failing example as a regression test: After fixing the bug, add the specific failing input as an example-based test. This ensures the bug doesn't regress even if the property test's random sampling misses it.
Keep the failure database: The property framework's database means that example runs again first on every subsequent run. This is the framework's way of preventing regressions.
Document the invariant: When a property fails, it reveals an invariant you didn't know about. Document it in a comment above the property.

Property-based testing in CI isn't optional once you adopt it locally — local runs with 50 examples find some bugs, but CI with 500+ examples finds the subtle ones. The extra coverage is the point.