Property-Based Testing

Shrinking and Debugging in Property-Based Testing

HelpMeTest

HelpMeTest

— 8 min read
Shrinking and Debugging in Property-Based Testing

One of the first objections people raise about property-based testing is: "If the framework generates random inputs, how do I know what went wrong when a test fails?" The answer is shrinking — and it's the feature that makes property-based testing practical rather than theoretical.

Without shrinking, a failing test would report the raw random input that triggered the failure: a 500-character string, a list of 87 integers, or a complex nested object. That would be nearly useless for debugging. Shrinking transforms that raw failure into the smallest, simplest example that still triggers the bug — often just two or three values — making it as easy to debug as any manually written test case.

This guide explains how shrinking works, how to read and use minimal counterexamples, how to replay failures in CI environments, and how to write property tests that shrink well.

How Shrinking Works

When a property-based testing framework (fast-check, Hypothesis, QuickCheck) finds an input that causes a property to fail, it doesn't stop there. It enters a shrinking phase: it systematically tries smaller and simpler versions of the failing input to see if they also cause a failure.

The exact shrinking algorithm varies by framework, but the general approach is:

  1. Start with the failing input (e.g., the integer 1847 or the list [42, -7, 103, 8, -2, 0, 55, 19])
  2. Try a simpler version (e.g., 923 or [42, -7, 103])
  3. If the simpler version also fails, keep going
  4. If it doesn't fail, try a different simplification
  5. Repeat until no further simplification causes a failure

For integers, "simpler" typically means closer to zero. For lists, it means shorter lists with simpler elements. For strings, it means shorter strings with simpler characters. For objects, it means objects with fewer fields and simpler field values.

The result is a minimal counterexample — the simplest input that still breaks your property.

Reading a Minimal Counterexample

Here's what a fast-check failure output looks like:

Property failed after 34 tests
{ seed: 1829374650, path: "34:0:1:0", endOnFailure: true }
Counterexample: [-2147483648]
Shrunk 8 time(s)
What went wrong: Property returned false

The counterexample [-2147483648] is -(2^31) — the most negative 32-bit integer. The property ran 34 times before finding a failure, then shrunk 8 times to arrive at this minimal case. This immediately tells you: your code has an issue at the boundary of 32-bit integer representation. Without shrinking, you might have seen something like [-1293847562] and had to figure out that the actual issue is at the 32-bit boundary.

Here's the equivalent from Hypothesis in Python:

Falsifying example: test_negate(
    n=-2147483648,
)
AssertionError

Hypothesis gives you a clean function call showing exactly what arguments triggered the failure. You can copy this directly into a debugger session.

A Step-by-Step Shrinking Example

Let's trace through a real example. Suppose you have a function that's supposed to find the maximum element in a list:

function findMax(arr) {
  let max = 0;  // Bug: initializing to 0 instead of -Infinity
  for (const x of arr) {
    if (x > max) max = x;
  }
  return max;
}

And a property test:

import fc from 'fast-check';

test('findMax returns an element present in the array', () => {
  fc.assert(
    fc.property(fc.array(fc.integer(), { minLength: 1 }), (arr) => {
      const max = findMax(arr);
      return arr.includes(max);
    })
  );
});

fast-check might first find this failing input:

[47, -23, 88, -100, 12, -5, -77, 33, -42, 8]

The findMax of this array returns 88. arr.includes(88) is true, so this passes. fast-check keeps searching and eventually finds:

[-5, -8, -3]

findMax of [-5, -8, -3] returns 0 (because all elements are less than the initial max = 0), and arr.includes(0) is false. The property fails.

Shrinking reduces this to:

[-1]

findMax([-1]) returns 0. [-1].includes(0) is false. This is the minimal counterexample — a single-element list containing any negative number. The bug is now completely obvious.

Why Minimal Counterexamples Matter

Consider trying to debug a failure with the raw random input versus the shrunken one:

Raw input: [47, -23, 88, -100, 12, -5, -77, 33, -42, 8, 15, -91, 0, 52, -7]

Debugging this: Why does findMax fail for this specific 15-element list? You'd have to trace through the implementation manually to find the pattern.

Shrunken input: [-1]

Debugging this: findMax([-1]) returns 0, not -1. The initialization let max = 0 is wrong. Takes 10 seconds to diagnose.

Shrinking doesn't just make failures easier to read — it makes bugs immediately obvious in most cases. The minimal counterexample usually points directly at the root cause.

Shrinking in fast-check (JavaScript)

fast-check uses integrated shrinking — every arbitrary knows how to shrink its own values, and composition of arbitraries automatically composes their shrinking behavior. You don't need to write shrinking code manually.

Built-in arbitraries all shrink toward simple values:

  • Integers shrink toward 0
  • Strings shrink toward ''
  • Arrays shrink toward [], then toward arrays with simpler elements
  • Booleans shrink toward false
  • Floats shrink toward 0

When you compose arbitraries with .map(), the shrunken values of the underlying arbitrary are mapped through your transformation:

// This shrinks properly — integers shrink toward 0, then *2 maps them toward 0
const evenIntegers = fc.integer().map(n => n * 2);

// This also shrinks, but shrink space is smaller due to filtering
const positiveIntegers = fc.integer().filter(n => n > 0);

Using .filter() reduces the shrink space because many shrunken candidates are filtered out. Prefer .map() over .filter() when possible for better shrinking.

Shrinking in Hypothesis (Python)

Hypothesis uses a different approach: it maintains an internal byte-string representation of each generated value and shrinks the byte string directly. This means shrinking works automatically for any strategy, including complex custom ones built with @st.composite.

One important nuance: Hypothesis's shrinking is more aggressive than most other tools. It will often find smaller counterexamples than you'd expect, and it runs shrinking in the background even between test runs (the "shrink" phase in CI output).

Replaying Failures

Both fast-check and Hypothesis provide mechanisms to replay the exact sequence of random inputs that caused a failure. This is essential for CI reproducibility.

Replaying in fast-check

fast-check prints the seed and path when a property fails:

Property failed after 34 tests
{ seed: 1829374650, path: "34:0:1:0", endOnFailure: true }
Counterexample: [[-1]]

Replay by passing these parameters back to fc.assert:

fc.assert(
  fc.property(fc.array(fc.integer(), { minLength: 1 }), (arr) => {
    const max = findMax(arr);
    return arr.includes(max);
  }),
  { seed: 1829374650, path: '34:0:1:0', endOnFailure: true }
);

This will reproduce the exact failure. Once you've fixed the bug, run without the replay parameters to verify the fix works across a fresh set of random inputs.

Replaying in Hypothesis

Hypothesis automatically stores failing examples in its database (.hypothesis/examples/). The failing example will be retried on every subsequent test run until it passes. This is the safest approach for CI: you don't need to manage seeds manually.

If you want to hardcode a specific example that should always run:

from hypothesis import given, example
from hypothesis import strategies as st

@given(st.lists(st.integers(), min_size=1))
@example([-1])  # always test this specific case
def test_find_max(arr):
    max_val = find_max(arr)
    assert max_val in arr

The @example decorator runs first, before any random inputs. This documents a known edge case while still running the full property test.

Debugging Strategies for Property Failures

When a property test fails and shrinking gives you a minimal counterexample, here's how to debug efficiently:

Step 1: Reproduce it as a unit test. Take the minimal counterexample and write it as a traditional example-based test. This gives you a stable failing test you can work with:

test('findMax with all-negative array returns minimum element, not 0', () => {
  expect(findMax([-1])).toBe(-1);  // was returning 0
});

Step 2: Run in isolation. With the minimal example as a fixed unit test, you can run it in isolation, attach a debugger, add console.log statements, and iterate without the overhead of property test generation.

Step 3: Fix the bug. Once you understand the root cause, fix it.

Step 4: Re-run the property test. Verify that the property now passes across a fresh set of random inputs. Remove any replay parameters (seed, path) to let it run freely.

Step 5: Keep the example. Add the minimal counterexample as a permanent @example or test.each case to document this specific edge case for future readers.

Writing Shrink-Friendly Properties

Some property tests shrink better than others. Here are principles for writing properties that produce maximally useful minimal counterexamples.

Prefer simple arbitraries. The more complex your input generation, the larger the shrink space. If your property only needs integers, don't use floats. If it only needs non-empty lists, use { minLength: 1 } rather than generating all lists and filtering.

Avoid .filter() on large spaces. Filtered arbitraries shrink poorly because many shrunken candidates are filtered out. Instead, use .map() to construct valid values from simple ones:

// Worse — shrinking is limited by the filter
const nonZero = fc.integer().filter(n => n !== 0);

// Better — shrinks toward 1 and -1 naturally
const nonZero = fc.oneof(
  fc.integer({ min: 1 }),
  fc.integer({ max: -1 })
);

Decompose complex properties. If you have a property with 5 generators and it fails, the minimal counterexample might involve all 5. Break complex properties into simpler ones with fewer generators — each one shrinks to a smaller counterexample.

Use the simplest generator that captures your intent. If your code handles strings, don't use fc.unicodeString() when fc.string() (ASCII) is sufficient. The simpler the generator, the simpler the shrunken counterexample.

Avoid assume()/filter() for common conditions. If you're filtering out more than 10-20% of inputs, restructure your generator to produce only valid inputs. Heavy filtering makes shrinking much slower.

Understanding the Shrink Output

fast-check's failure output includes a Shrunk N time(s) line that tells you how many simplification steps were applied. A high shrink count (20+) means the original random input was very far from minimal. A low count (0-3) means the random input was already close to minimal — which sometimes indicates your generators are already generating simple values.

Hypothesis shows similar information:

Falsifying example: test_find_max(arr=[-1])

When Hypothesis finds a failure, it often shows just the final minimal example without the intermediate steps. You can enable more verbose output with @settings(verbosity=Verbosity.verbose) to see the full shrinking process.

Failure Databases in CI

In CI environments, both tools handle failure persistence differently:

fast-check: Each run is independent by default. CI failures include the seed and path in the output, so you can replay them locally. No persistent database.

Hypothesis: The .hypothesis/ directory stores failing examples persistently. If you commit this directory to your repository, CI will always replay known failures. If you don't commit it, each CI run starts fresh but still builds up its database during the run.

The recommendation for most teams: commit .hypothesis/examples/ to track known failures, and exclude .hypothesis/unicode_data/ which is large and regeneratable.

Connecting to HelpMeTest

When property tests run as part of a larger suite on HelpMeTest, the platform tracks test history and surfaces regressions — when a property that was passing starts failing in a new deployment. HelpMeTest's cloud infrastructure runs your property tests at scale, and the AI-powered test analysis can help identify patterns in which properties are failing and why. For teams running Robot Framework or Playwright alongside property tests, HelpMeTest's usage-based pricing ($0.003/run) provides unified visibility across all test types without a per-seat or per-suite penalty.

Conclusion

Shrinking is what makes property-based testing practical. Without it, failing tests would report giant incomprehensible inputs that take longer to debug than the bug itself. With shrinking, you get a minimal counterexample that often makes the bug immediately obvious.

Understanding how shrinking works helps you write better property tests: use simple generators, prefer .map() over .filter(), and decompose complex properties into simpler ones. When a test fails, use the minimal counterexample as the starting point for a new example-based test, fix the bug, then verify the property passes across fresh random inputs.

The combination of automatic shrinking and failure replay makes property-based testing not just powerful but practical — failures are reproducible, debuggable, and documented as minimal examples that future developers can understand at a glance.

Read more

Start now free