Elixir

Property-Based Testing in Elixir with StreamData

HelpMeTest

HelpMeTest

— 5 min read
Property-Based Testing in Elixir with StreamData

Property-based testing generates hundreds of random inputs and finds edge cases you would never think to write by hand. This post covers StreamData generators, the ExUnitProperties module, how shrinking works, and techniques for composing generators to describe realistic domain data.

Example-based tests are assertions about specific inputs you thought of. Property-based tests are assertions about invariants that must hold for any valid input. When you say "sorting a list should produce a list with the same elements in non-decreasing order", you've described a property — and StreamData will generate hundreds of lists to try to prove you wrong.

Installing StreamData

Add to mix.exs:

defp deps do
  [
    {:stream_data, "~> 1.1", only: [:test, :dev]}
  ]
end

StreamData is also bundled with Elixir's standard library since 1.16 via ExUnit.StreamData, but the standalone package offers more generators and is more actively developed.

Your First Property Test

defmodule MyApp.SorterTest do
  use ExUnit.Case, async: true
  use ExUnitProperties

  property "sorting preserves all elements" do
    check all list <- list_of(integer()) do
      sorted = Enum.sort(list)
      assert Enum.sort(sorted) == sorted           # idempotent
      assert length(sorted) == length(list)        # no elements lost
      assert Enum.sort(list) == sorted             # consistent
    end
  end

  property "sorting produces non-decreasing sequence" do
    check all list <- list_of(integer()), length(list) > 1 do
      sorted = Enum.sort(list)
      sorted
      |> Enum.chunk_every(2, 1, :discard)
      |> Enum.each(fn [a, b] -> assert a <= b end)
    end
  end
end

use ExUnitProperties brings in property/2 and check all/2. Inside check all, you bind generators with <-. StreamData runs 100 iterations by default (configurable with max_runs: N).

Core Generators

StreamData ships with generators for all primitive types:

# Integers
integer()           # any integer
integer(1..100)     # bounded integer
positive_integer()  # >= 1

# Floats
float()
float(min: 0.0, max: 1.0)

# Strings
string(:alphanumeric)
string(:printable)
string(:ascii, min_length: 5, max_length: 20)

# Atoms
atom(:alphanumeric)

# Booleans
boolean()

# Lists and maps
list_of(integer())
list_of(string(:alphanumeric), min_length: 1, max_length: 10)
map_of(string(:alphanumeric), integer())

# Fixed-size tuples
tuple({integer(), string(:alphanumeric)})

# One of several values
one_of([constant(:ok), constant(:error)])
member_of([:red, :green, :blue])

Composing Generators with map/2 and bind/2

The real power comes from combining generators to model your domain.

map/2 transforms generator output:

defp email_generator do
  map(
    {string(:alphanumeric, min_length: 1), string(:alphanumeric, min_length: 2)},
    fn {local, domain} -> "#{local}@#{domain}.com" end
  )
end

defp positive_money_generator do
  map(positive_integer(), fn cents -> cents / 100 end)
end

bind/2 chains generators where the second depends on the first:

defp non_empty_sublist_generator(list) do
  bind(integer(1..length(list)), fn size ->
    list
    |> Enum.take_random(size)
    |> constant()
  end)
end

defp date_range_generator do
  bind(integer(2000..2025), fn year ->
    bind(integer(1..12), fn month ->
      max_day = Date.days_in_month(Date.new!(year, month, 1))
      map(integer(1..max_day), fn day ->
        Date.new!(year, month, day)
      end)
    end)
  end)
end

Building Domain Generators

For realistic tests, model your domain structs as generators:

defmodule MyApp.Generators do
  use ExUnitProperties

  def user_generator do
    gen all first <- string(:alphanumeric, min_length: 1, max_length: 50),
            last <- string(:alphanumeric, min_length: 1, max_length: 50),
            age <- integer(18..120),
            email <- email_generator() do
      %MyApp.User{
        first_name: first,
        last_name: last,
        age: age,
        email: email
      }
    end
  end

  def product_generator do
    gen all name <- string(:printable, min_length: 2, max_length: 100),
            price_cents <- integer(1..1_000_000),
            category <- member_of([:electronics, :clothing, :food, :books]) do
      %MyApp.Product{
        name: name,
        price_cents: price_cents,
        category: category
      }
    end
  end

  defp email_generator do
    gen all local <- string(:alphanumeric, min_length: 1, max_length: 30),
            domain <- string(:alphanumeric, min_length: 2, max_length: 20) do
      "#{local}@#{domain}.example"
    end
  end
end

The gen all macro inside a generator definition is syntactic sugar for nested bind and map calls.

Writing Meaningful Properties

Properties are only as useful as the invariants you choose. Common patterns:

Round-trip properties — encode then decode yields the original:

property "JSON encode/decode round-trips a user" do
  check all user <- MyApp.Generators.user_generator() do
    encoded = Jason.encode!(user)
    decoded = Jason.decode!(encoded)
    assert decoded["email"] == user.email
    assert decoded["age"] == user.age
  end
end

Commutativity — order of operations doesn't matter:

property "merging maps is commutative for non-overlapping keys" do
  check all m1 <- map_of(atom(:alphanumeric), integer()),
            m2 <- map_of(atom(:alphanumeric), integer()),
            MapSet.disjoint?(MapSet.new(Map.keys(m1)), MapSet.new(Map.keys(m2))) do
    assert Map.merge(m1, m2) == Map.merge(m2, m1)
  end
end

Boundary conditions — results stay within expected ranges:

property "discounted price is always less than or equal to original" do
  check all product <- MyApp.Generators.product_generator(),
            discount_pct <- float(min: 0.0, max: 100.0) do
    discounted = MyApp.Pricing.apply_discount(product, discount_pct)
    assert discounted.price_cents <= product.price_cents
    assert discounted.price_cents >= 0
  end
end

Idempotency — doing it twice is the same as once:

property "normalizing a string twice is the same as once" do
  check all s <- string(:printable) do
    once = MyApp.Text.normalize(s)
    twice = MyApp.Text.normalize(once)
    assert once == twice
  end
end

Understanding Shrinking

When StreamData finds a failing input, it doesn't just report the raw generated value — it shrinks it to the smallest possible failing case. This is what makes property-based testing so powerful in practice.

Suppose you have a bug that only triggers when a list has two identical adjacent elements. StreamData might initially find a failure with [3, 7, 7, 2, 1, 9, 7], then shrink it to [0, 0] — the minimal reproducing case.

Shrinking works automatically. The smaller the generator's output type, the better shrinking works. Custom generators built from primitives shrink well because StreamData knows how to shrink each primitive component.

You can observe shrinking with a deliberately broken property:

property "all integers are less than 50 (intentionally broken)" do
  check all n <- integer() do
    assert n < 50
  end
end

Output:

  1) property all integers are less than 50 (intentionally broken)
     Failed with generated values (after 3 successful runs):
         * Clause:  n <- integer()
           Generated: 50

     Shrinking...(7 steps)
     The minimum failing value was:
         * Clause:  n <- integer()
           Generated: 50

StreamData shrinks integer() toward 0 but finds that 50 is already minimal (it's the exact boundary).

Filtering with Guards

Use guards in check all to restrict generated values:

property "division is the inverse of multiplication for non-zero divisors" do
  check all a <- integer(),
            b <- integer(),
            b != 0 do
    result = a * b / b
    assert_in_delta result, a * 1.0, 0.0001
  end
end

Avoid over-filtering — if your guard rejects more than 99% of generated values, StreamData will give up. Use filter/2 sparingly:

# Prefer this:
positive_integer()

# Over this:
filter(integer(), fn n -> n > 0 end)

Configuring Check Runs

Override iteration count or seed for reproducibility:

property "expensive property needs fewer runs" do
  check all x <- large_data_structure_generator(),
            max_runs: 20 do
    assert MyApp.analyze(x) != :error
  end
end

# Reproduce a specific failure with a fixed seed
property "reproducing a flaky failure" do
  check all x <- integer(),
            initial_seed: {1, 2, 3} do
    # same sequence every run
    assert x < 1000
  end
end

Mixing Property Tests with Example Tests

Properties complement example tests — they don't replace them. Use examples for known edge cases (empty string, zero, nil) and properties for invariants over arbitrary valid inputs:

defmodule MyApp.SlugTest do
  use ExUnit.Case, async: true
  use ExUnitProperties

  # Example tests for known edge cases
  test "handles empty string" do
    assert MyApp.Slug.generate("") == ""
  end

  test "handles string with only spaces" do
    assert MyApp.Slug.generate("   ") == ""
  end

  # Property tests for invariants
  property "slug contains only lowercase alphanumeric and hyphens" do
    check all title <- string(:printable, min_length: 1) do
      slug = MyApp.Slug.generate(title)
      assert slug =~ ~r/^[a-z0-9-]*$/
    end
  end

  property "slug never starts or ends with a hyphen" do
    check all title <- string(:alphanumeric, min_length: 1) do
      slug = MyApp.Slug.generate(title)
      refute String.starts_with?(slug, "-")
      refute String.ends_with?(slug, "-")
    end
  end
end

Key Takeaways

  • Property-based tests assert invariants over generated inputs; they find bugs you'd never think to write an example for.
  • Use check all with generators for the happy path; use guards (b != 0) to restrict invalid inputs.
  • Build domain generators with gen all, map/2, and bind/2 to produce realistic test data.
  • Shrinking automatically reduces failing cases to their minimal form — no manual bisection needed.
  • Properties complement example tests: use examples for known boundaries, properties for arbitrary valid inputs.
  • Prefer bounded generators (positive_integer(), string(:alphanumeric)) over filtered ones for better shrinking.

Read more

Start now free