Testing

Property-Based Testing in Elixir with StreamData

HelpMeTest

HelpMeTest

— 4 min read
Property-Based Testing in Elixir with StreamData

Unit tests check specific examples: "given input X, the output is Y." Property-based testing asks a different question: "for ALL inputs with this shape, does this property always hold?" StreamData brings this paradigm to Elixir, integrating with ExUnit to automatically generate hundreds of test cases from your specifications.

Why Property-Based Testing?

Consider a function that encodes and decodes a string. You might write:

test "encode then decode returns original" do
  assert decode(encode("hello")) == "hello"
  assert decode(encode("")) == ""
  assert decode(encode("unicode: 日本語")) == "unicode: 日本語"
end

Three examples. Good. But what about strings with embedded newlines? Null bytes? Extremely long strings? Property-based testing generates hundreds of variations automatically and finds the edge cases you didn't think to write.

Installation

Add StreamData to mix.exs:

defp deps do
  [
    {:stream_data, "~> 0.6", only: [:test, :dev]}
  ]
end

Basic Properties

Import ExUnitProperties and use property/2 (analogous to test/2) with check all:

defmodule MyApp.StringUtilsTest do
  use ExUnit.Case
  use ExUnitProperties

  property "encode/decode is a round trip" do
    check all string <- string(:printable) do
      assert decode(encode(string)) == string
    end
  end

  property "length is non-negative" do
    check all string <- string(:utf8) do
      assert String.length(string) >= 0
    end
  end
end

check all runs the block 100 times (configurable) with generated values. If any run fails, StreamData reports the failing example and shrinks it to the smallest possible failure case.

Generators

StreamData comes with generators for common data types:

# Integers
integer()                    # any integer
integer(1..100)              # bounded range
positive_integer()           # > 0

# Strings
string(:ascii)               # ASCII printable
string(:utf8)                # valid UTF-8
string(:printable)           # printable characters

# Booleans
boolean()

# Atoms
atom(:alphanumeric)

# Collections
list_of(integer())           # list of integers
map_of(string(:ascii), integer())  # map with string keys

# Fixed values
constant("hello")            # always returns "hello"
member_of([:a, :b, :c])     # random element from list

# Nullable
one_of([nil, positive_integer()])

Composing Generators

Build complex generators from simpler ones:

defmodule MyApp.Generators do
  use ExUnitProperties

  def email do
    gen all username <- string(:alphanumeric, min_length: 1),
            domain <- string(:alphanumeric, min_length: 1),
            tld <- member_of(["com", "org", "net", "io"]) do
      "#{username}@#{domain}.#{tld}"
    end
  end

  def user do
    gen all name <- string(:printable, min_length: 1, max_length: 50),
            email <- email(),
            age <- integer(18..120) do
      %{name: name, email: email, age: age}
    end
  end
end

Use gen all (like check all but for generator definitions) to compose generators declaratively.

Writing Useful Properties

Not every function has obvious universal properties. Common property patterns:

Round-trip Properties

Encoding, serialization, and transformation functions:

property "JSON encode/decode is identity" do
  check all data <- term() do
    assert Jason.decode!(Jason.encode!(data)) == data
  end
end

property "list sort is idempotent" do
  check all list <- list_of(integer()) do
    sorted = Enum.sort(list)
    assert Enum.sort(sorted) == sorted
  end
end

Invariant Properties

Properties that hold regardless of input:

property "result length always equals input length" do
  check all list <- list_of(integer()) do
    result = Enum.map(list, &(&1 * 2))
    assert length(result) == length(list)
  end
end

property "filter never adds elements" do
  check all list <- list_of(integer()),
            threshold <- integer() do
    filtered = Enum.filter(list, &(&1 > threshold))
    assert length(filtered) <= length(list)
  end
end

Commutativity and Associativity

property "addition is commutative" do
  check all a <- integer(),
            b <- integer() do
    assert MyApp.Math.add(a, b) == MyApp.Math.add(b, a)
  end
end

Model-Based Properties

Compare a simple (possibly slow) reference implementation against your optimized one:

property "optimized sort matches naive sort" do
  check all list <- list_of(integer()) do
    assert MyApp.FastSort.sort(list) == Enum.sort(list)
  end
end

Shrinking

When a property fails, StreamData automatically shrinks the failing case to the smallest example that still fails. Given a property that fails for "abc\0def", StreamData will minimize to "\0" if that's the actual trigger, making debugging much easier:

1) property encode/decode is a round trip (MyApp.StringUtilsTest)
   Failed with generated values:

     * Clause:    string <- string(:printable)
       Generated: "\x00"

   Error: assert decode(encode("\x00")) == "\x00"
   Expected: "\x00"
   Got:      ""

You immediately know the null byte is the problem, not the surrounding characters.

Configuration

Tune the number of runs and other options:

# More runs for critical properties
property "security-critical function" do
  check all input <- string(:utf8), max_runs: 1000 do
    assert safe?(input)
  end
end

# Faster runs during development
property "quick check" do
  check all n <- integer(), max_runs: 20 do
    assert n * 2 == n + n
  end
end

Or configure globally in config/test.exs:

config :stream_data,
  max_runs: 200

Integrating with ExUnit Setup

Properties use the same setup callbacks as regular tests:

defmodule MyApp.ProductTest do
  use ExUnit.Case
  use ExUnitProperties

  setup do
    category = create_category()
    %{category: category}
  end

  property "products always belong to a valid category", %{category: category} do
    check all name <- string(:printable, min_length: 1),
              price <- positive_integer() do
      {:ok, product} = MyApp.Products.create(%{
        name: name,
        price: price,
        category_id: category.id
      })
      assert product.category_id == category.id
    end
  end
end

Stateful Property Testing

For more advanced scenarios, test sequences of operations:

property "stack operations maintain size invariants" do
  check all operations <- list_of(
    one_of([
      {:push, integer()},
      :pop
    ])
  ) do
    final_stack = Enum.reduce(operations, [], fn
      {:push, val}, stack -> [val | stack]
      :pop, [] -> []
      :pop, [_ | rest] -> rest
    end)

    assert length(final_stack) >= 0
  end
end

When to Use Property Tests

Property tests shine for:

  • Pure functions with mathematical properties (sort, encode/decode, parse/format)
  • Data validation — verify validators reject invalid data and accept valid data
  • Protocol implementations — verify a codec handles all inputs correctly
  • Security-sensitive code — test against a wide input space

They're less suited for code with complex side effects or where generating realistic domain data is too expensive. Use them alongside, not instead of, example-based tests.

Continuous Testing with HelpMeTest

Property tests expand coverage for pure Elixir logic. For testing the full user-facing behavior of your Phoenix application, HelpMeTest provides browser-based monitoring that runs in production continuously. Properties verify your code handles all inputs; HelpMeTest verifies your live application handles all users.

Summary

StreamData brings property-based testing to Elixir with minimal ceremony:

  • check all generates random inputs and runs your property 100+ times
  • Generators compose: build complex types from simple primitives
  • Shrinking reduces failures to minimal examples automatically
  • Works with ExUnit.Case, setup, async, and all ExUnit features
  • Finds bugs in edge cases that example-based tests miss

Start small: pick one pure function, identify a single invariant ("decode(encode(x)) == x"), and write your first property. The generator will surprise you with inputs you never would have written by hand.

Read more