QuickCheck: Property-Based Testing in Haskell and Erlang Deep Dive

QuickCheck: Property-Based Testing in Haskell and Erlang Deep Dive

QuickCheck originated in Haskell in 1999 and remains the foundational property-based testing library. Its ideas have since spread to dozens of languages, but the Haskell version—and Erlang's commercial Quviq QuickCheck—represent the most mature implementations. This post covers advanced QuickCheck patterns that go beyond the introductory examples.

QuickCheck Fundamentals: What Makes It Different

QuickCheck tests properties—universally quantified statements about your code—rather than specific examples. The library generates random inputs, finds failures, and shrinks them to minimal counterexamples.

import Test.QuickCheck

-- Property: reversing a list twice is identity
prop_reverseReverse :: [Int] -> Bool
prop_reverseReverse xs = reverse (reverse xs) == xs

-- Property: sort is idempotent
prop_sortIdempotent :: [Int] -> Bool
prop_sortIdempotent xs = sort (sort xs) == sort xs

main :: IO ()
main = do
  quickCheck prop_reverseReverse
  quickCheck prop_sortIdempotent

Custom Arbitrary Instances

The Arbitrary typeclass controls how QuickCheck generates and shrinks values for a type. Writing good instances is the core skill.

Basic Arbitrary Instance

data Priority = Low | Medium | High | Critical
  deriving (Show, Eq, Ord)

instance Arbitrary Priority where
  arbitrary = elements [Low, Medium, High, Critical]
  -- shrink defaults to [] (no shrinking), which is fine for enums

Structured Data with Invariants

data DateRange = DateRange
  { startDate :: Day
  , endDate   :: Day
  } deriving (Show)

instance Arbitrary DateRange where
  arbitrary = do
    start <- arbitrary
    -- endDate must be >= startDate
    offset <- choose (0, 365)
    let end = addDays offset start
    return $ DateRange start end
  
  shrink (DateRange start end) =
    [ DateRange start' end
    | start' <- shrink start
    , start' <= end
    ]
    ++
    [ DateRange start end'
    | end' <- shrink end
    , end' >= start
    ]

The shrink implementation is critical. Without it, QuickCheck shows you the original random counterexample. With good shrinking, you get the minimal case that triggers the bug.

Weighted Generation

Use frequency to control the distribution of generated values:

data Transaction = Credit Amount | Debit Amount | Refund Amount
  deriving (Show)

instance Arbitrary Transaction where
  arbitrary = frequency
    [ (60, Credit  <$> positiveAmount)
    , (35, Debit   <$> positiveAmount)
    , (5,  Refund  <$> positiveAmount)
    ]
    where positiveAmount = abs <$> arbitrary

  shrink (Credit a)  = Credit  <$> shrink a
  shrink (Debit a)   = Debit   <$> shrink a
  shrink (Refund a)  = Refund  <$> shrink a

60% credits, 35% debits, 5% refunds—matching real-world distribution.

Advanced Generator Combinators

Gen Monad Patterns

-- Generate a valid IP address
genIPv4 :: Gen String
genIPv4 = do
  octets <- replicateM 4 (choose (0, 255) :: Gen Int)
  return $ intercalate "." (map show octets)

-- Generate a non-empty sorted list
genSortedNonEmpty :: (Arbitrary a, Ord a) => Gen [a]
genSortedNonEmpty = do
  xs <- listOf1 arbitrary
  return (sort xs)

-- Generate a valid email (simplified)
genEmail :: Gen String
genEmail = do
  user   <- listOf1 (elements ['a'..'z'])
  domain <- listOf1 (elements ['a'..'z'])
  tld    <- elements ["com", "org", "net", "io"]
  return $ user ++ "@" ++ domain ++ "." ++ tld

Dependent Generation with >>=

When one value depends on another:

-- Generate a list and a valid index into it
genListWithIndex :: Gen ([Int], Int)
genListWithIndex = do
  xs <- listOf1 arbitrary
  i  <- choose (0, length xs - 1)
  return (xs, i)

prop_indexInBounds :: Property
prop_indexInBounds = forAll genListWithIndex $ \(xs, i) ->
  i >= 0 && i < length xs  -- This always holds by construction

prop_indexRoundtrip :: Property
prop_indexRoundtrip = forAll genListWithIndex $ \(xs, i) ->
  xs !! i == xs !! i  -- Trivial, but demonstrates the pattern

suchThat for Filtered Generation

-- Generate even integers (use sparingly—prefer direct generation)
genEven :: Gen Int
genEven = arbitrary `suchThat` even

-- Better: generate directly
genEvenDirect :: Gen Int
genEvenDirect = (2 *) <$> arbitrary

suchThat can cause issues if the predicate is rarely satisfied—QuickCheck will give up after too many failures. Always prefer constructive generation.

Properties Beyond Boolean

=== for Better Failure Messages

import Test.QuickCheck

-- Instead of:
prop_bad :: [Int] -> Bool
prop_bad xs = sort xs == sort (reverse xs)

-- Use === for better output on failure:
prop_good :: [Int] -> Property
prop_good xs = sort xs === sort (reverse xs)
-- Failure shows: [3,1,2] /= [2,1,3]

classify for Test Distribution Insight

prop_insertionSort :: [Int] -> Property
prop_insertionSort xs = 
  classify (null xs) "empty" $
  classify (length xs == 1) "singleton" $
  classify (length xs > 100) "large" $
  sort xs == insertionSort xs

Run with verboseCheck to see what percentage of tests hit each class. If 99% are "empty", your generator isn't exercising the interesting cases.

cover for Enforced Coverage

prop_withCoverage :: [Int] -> Property
prop_withCoverage xs =
  cover 30 (length xs > 10) "lists longer than 10" $
  cover 10 (null xs) "empty lists" $
  sort xs == insertionSort xs

This fails if fewer than 30% of generated test cases have lists longer than 10 elements. Enforces that your property is actually being tested over the interesting range.

Stateful Testing in Haskell QuickCheck

For stateful systems, use Test.QuickCheck.Monadic:

import Test.QuickCheck.Monadic

-- Testing a mutable counter
prop_counter :: [CounterOp] -> Property
prop_counter ops = monadicIO $ do
  counter <- run newCounter
  model   <- run $ newIORef (0 :: Int)
  
  forM_ ops $ \op -> case op of
    Increment -> do
      run $ increment counter
      run $ modifyIORef model (+1)
    Decrement -> do
      run $ decrement counter
      run $ modifyIORef model (subtract 1)
    Reset -> do
      run $ reset counter
      run $ writeIORef model 0
  
  expected <- run $ readIORef model
  actual   <- run $ readCounter counter
  assert (actual == expected)

data CounterOp = Increment | Decrement | Reset
  deriving (Show, Arbitrary via GenericArbitrary CounterOp)

Erlang QuickCheck (Quviq EQC)

Erlang's QuickCheck (commercial, from Quviq) is famous for finding bugs in distributed systems that no other testing approach found—including bugs in Ericsson's telecom stack and the LevelDB/Riak implementation.

Basic EQC Property

-module(prop_queue).
-include_lib("eqc/include/eqc.hrl").

prop_queue_fifo() ->
    ?FORALL(Ops, list(queue_op()),
        begin
            Queue = lists:foldl(fun apply_op/2, queue:new(), Ops),
            Model = lists:foldl(fun apply_op_model/2, [], Ops),
            queue:to_list(Queue) == Model
        end).

queue_op() ->
    frequency([
        {3, {enqueue, int()}},
        {2, dequeue}
    ]).

EQC Stateful Testing (statem)

The eqc_statem module is where Erlang QuickCheck truly shines. It generates sequences of API calls, models the expected state, and checks the implementation matches:

-module(prop_account).
-include_lib("eqc/include/eqc.hrl").
-include_lib("eqc/include/eqc_statem.hrl").

-record(state, {
    accounts = [],  % [{Id, Balance}]
    next_id  = 1
}).

%% Initial state
initial_state() -> #state{}.

%% Command generation
command(S) ->
    frequency([
        {3, {call, account, create, [pos_integer()]}},
        {2, {call, account, deposit,  [account_id(S), pos_integer()]}},
        {2, {call, account, withdraw, [account_id(S), pos_integer()]}},
        {1, {call, account, balance,  [account_id(S)]}}
    ]).

%% Preconditions
precondition(S, {call, _, deposit, [Id, _]}) ->
    lists:keymember(Id, 1, S#state.accounts);
precondition(S, {call, _, withdraw, [Id, Amount]}) ->
    case lists:keyfind(Id, 1, S#state.accounts) of
        {Id, Balance} -> Balance >= Amount;
        false -> false
    end;
precondition(_, _) -> true.

%% State transitions
next_state(S, Id, {call, _, create, [Initial]}) ->
    S#state{
        accounts = [{S#state.next_id, Initial} | S#state.accounts],
        next_id  = S#state.next_id + 1
    };
next_state(S, _, {call, _, deposit, [Id, Amount]}) ->
    S#state{accounts = update_balance(S#state.accounts, Id, Amount)};
%% ... etc

%% Postconditions
postcondition(S, {call, _, balance, [Id]}, Result) ->
    {Id, Balance} = lists:keyfind(Id, 1, S#state.accounts),
    Result == Balance;
postcondition(_, _, _) -> true.

prop_account() ->
    ?FORALL(Cmds, commands(?MODULE),
        begin
            {H, S, Res} = run_commands(?MODULE, Cmds),
            ?WHENFAIL(
                io:format("History: ~p~nState: ~p~nResult: ~p~n", [H, S, Res]),
                Res == ok
            )
        end).

EQC and Distributed Systems

Quviq's most famous work used eqc_statem to find 16 bugs in Riak's eventual consistency implementation—including bugs that only triggered after specific sequences of network partitions and operations. The test modeled Riak's distributed state machine and generated random operation sequences including partition_network, heal_partition, put, get.

The key insight: state machine testing doesn't require deep knowledge of the implementation. You specify what the system should do, and QuickCheck finds sequences where it doesn't.

Comparing QuickCheck Ports

Library Language Shrinking Stateful Commercial
QuickCheck Haskell Excellent Via Monadic No
Quviq EQC Erlang Excellent eqc_statem Yes
Hypothesis Python Excellent RuleBasedStateMachine No
fast-check JavaScript Good ModelBased No
ScalaCheck Scala Good Commands No
PropEr Erlang Good PropEr statem No

Integration with CI

# .github/workflows/property-tests.yml  
- name: Run QuickCheck tests
  run: |
    cabal test --test-option="--quickcheck-tests=10000" \
               --test-option="--quickcheck-replay=0"

Pass --quickcheck-replay=<seed> to reproduce a specific failure:

*** Failed! Falsifiable (after 47 tests and 3 shrinks):
...
Use --quickcheck-replay=1234567890 to reproduce.

Key Takeaways

  • Arbitrary instances are the core skill—write shrink implementations carefully
  • frequency controls distribution to match real-world input patterns
  • classify and cover ensure tests exercise interesting cases, not just trivial ones
  • Monadic QuickCheck handles stateful and IO-heavy systems
  • Erlang's eqc_statem found legendary bugs in distributed systems by modeling state machines
  • Reproduce failures with --quickcheck-replay seed

QuickCheck's 25-year track record proves the value of the approach. The patterns translate across all modern ports—understanding the original deeply makes you a better property-based tester in any language.

Read more

Start now free