Testing

State Machine Testing: FSM-Based Test Design

14 Oct 2026 — 7 min read

Every non-trivial software system has state. A shopping cart is empty, then has items, then is checked out. A door is locked, unlocked, or open. A connection is connecting, connected, or disconnected. Finite state machines (FSMs) are a precise way to model that state — and state machine testing is the discipline of using those models to design tests that actually catch the bugs that stateful systems produce.

This guide walks through the foundations of FSM-based testing, explains the coverage criteria that matter, shows how to model systems with XState, and demonstrates how to implement FSM-driven tests in both Jest and pytest.

What Is a Finite State Machine?

A finite state machine is defined by five components:

States (S) — a finite set of conditions the system can be in
Alphabet (Σ) — the set of valid inputs (events)
Transitions (δ) — a function mapping (state, input) → next state
Initial state (s₀) — the state the machine starts in
Final states (F) — states that represent accepted or terminal conditions (not always present in reactive systems)

As a simple example, consider a user authentication flow:

States: { LoggedOut, LoggingIn, LoggedIn, Locked }
Events: { submitCredentials, credentialsValid, credentialsInvalid, tooManyFailures, logout, unlock }

Transitions:
  LoggedOut   + submitCredentials   → LoggingIn
  LoggingIn   + credentialsValid    → LoggedIn
  LoggingIn   + credentialsInvalid  → LoggedOut (failCount++)
  LoggingIn   + tooManyFailures     → Locked
  LoggedIn    + logout              → LoggedOut
  Locked      + unlock              → LoggedOut

Every path through this FSM is a potential test case. The challenge is choosing which paths to cover and how thoroughly.

Coverage Criteria for State Machine Tests

State Coverage

Every state must be reached by at least one test. This is the minimum useful criterion.

Test 1: LoggedOut → LoggingIn → LoggedIn       (covers: LoggedOut, LoggingIn, LoggedIn)
Test 2: LoggedOut → LoggingIn → Locked         (covers: Locked)

State coverage is weak. You can reach every state without testing any transition more than once.

Transition Coverage

Every transition must be exercised by at least one test. This is the standard criterion for FSM-based testing and catches most implementation bugs.

Transitions to cover:
  T1: LoggedOut   → LoggingIn   (submitCredentials)
  T2: LoggingIn   → LoggedIn    (credentialsValid)
  T3: LoggingIn   → LoggedOut   (credentialsInvalid)
  T4: LoggingIn   → Locked      (tooManyFailures)
  T5: LoggedIn    → LoggedOut   (logout)
  T6: Locked      → LoggedOut   (unlock)

Minimum test paths for 100% transition coverage:
  Path A: LoggedOut → LoggingIn → LoggedIn → LoggedOut    (covers T1, T2, T5)
  Path B: LoggedOut → LoggingIn → LoggedOut               (covers T3)
  Path B extended: ...→ LoggingIn → Locked → LoggedOut    (covers T4, T6)

Transition-Pair Coverage (0-switch)

Every pair of consecutive transitions must appear in at least one test sequence. This catches bugs in how the system recovers after transitions — for example, whether failCount resets correctly after an unlock.

Pairs to cover (sample):
  T1→T2, T1→T3, T1→T4
  T2→T5
  T3→T1 (retry after invalid)
  T4→T6
  T5→T1 (login again after logout)
  T6→T1 (login after unlock)

Transition-pair coverage typically requires 2–3× as many tests as transition coverage but catches a much larger class of bugs.

Comparison of Coverage Criteria

Criterion	Detects missed states	Detects missing transitions	Detects interaction bugs	Typical test count
State coverage	Yes	No	No	Low (= # states)
Transition coverage	Yes	Yes	Partial	Moderate (= # transitions)
Transition-pair coverage	Yes	Yes	Yes	High (= # transition pairs)
All-paths coverage	Yes	Yes	Yes	Very high (often infeasible)

The State Explosion Problem

FSMs scale quadratically — and often exponentially — as systems grow. A system with 10 boolean flags can have 2¹⁰ = 1,024 states. Add parallel regions (concurrent states) and the numbers explode further.

Practical strategies to manage state explosion:

1. Use hierarchical state machines (Harel statecharts) — group related states into superstates. A "Payment" superstate can contain "EnteringCard", "Validating", and "Confirmed" without multiplying the outer states.

2. Model only the behavior under test — a login FSM need not include every detail of what the dashboard does. Keep models scoped.

3. Abstract data from state — instead of separate states for balance=100 and balance=200, use a single Active state with a guard balance > 0. Extended FSMs (EFSMs) carry this data as variables.

4. Use transition coverage, not all-paths — all-paths coverage is impractical for any loop in the model. Transition coverage gives a good risk/cost tradeoff.

Modeling with XState

XState is a JavaScript/TypeScript library for FSMs and statecharts. It produces machine definitions that are easy to test, visualize, and reason about.

import { createMachine, interpret } from 'xstate';

const authMachine = createMachine({
  id: 'auth',
  initial: 'loggedOut',
  context: {
    failCount: 0,
  },
  states: {
    loggedOut: {
      on: {
        SUBMIT: 'loggingIn',
      },
    },
    loggingIn: {
      on: {
        SUCCESS: 'loggedIn',
        FAILURE: [
          {
            target: 'locked',
            guard: ({ context }) => context.failCount >= 3,
            actions: assign({ failCount: ({ context }) => context.failCount + 1 }),
          },
          {
            target: 'loggedOut',
            actions: assign({ failCount: ({ context }) => context.failCount + 1 }),
          },
        ],
      },
    },
    loggedIn: {
      on: {
        LOGOUT: 'loggedOut',
      },
    },
    locked: {
      on: {
        UNLOCK: {
          target: 'loggedOut',
          actions: assign({ failCount: 0 }),
        },
      },
    },
  },
});

XState's createMachine gives you a pure, testable definition. The state machine itself has no side effects — all side effects live in services and actions.

Implementing FSM Tests in Jest

import { createActor } from 'xstate';
import { authMachine } from './authMachine';

describe('Auth FSM - Transition Coverage', () => {

  function buildActor() {
    return createActor(authMachine).start();
  }

  test('T1+T2+T5: successful login and logout', () => {
    const actor = buildActor();

    // T1: loggedOut → loggingIn
    actor.send({ type: 'SUBMIT' });
    expect(actor.getSnapshot().value).toBe('loggingIn');

    // T2: loggingIn → loggedIn
    actor.send({ type: 'SUCCESS' });
    expect(actor.getSnapshot().value).toBe('loggedIn');

    // T5: loggedIn → loggedOut
    actor.send({ type: 'LOGOUT' });
    expect(actor.getSnapshot().value).toBe('loggedOut');
  });

  test('T3: invalid credentials returns to loggedOut and increments failCount', () => {
    const actor = buildActor();

    actor.send({ type: 'SUBMIT' });
    actor.send({ type: 'FAILURE' });

    const snapshot = actor.getSnapshot();
    expect(snapshot.value).toBe('loggedOut');
    expect(snapshot.context.failCount).toBe(1);
  });

  test('T4+T6: lockout after 4 failures, then unlock resets counter', () => {
    const actor = buildActor();

    // Three failures that don't lock
    actor.send({ type: 'SUBMIT' });
    actor.send({ type: 'FAILURE' }); // failCount=1, loggedOut
    actor.send({ type: 'SUBMIT' });
    actor.send({ type: 'FAILURE' }); // failCount=2, loggedOut
    actor.send({ type: 'SUBMIT' });
    actor.send({ type: 'FAILURE' }); // failCount=3, loggedOut

    // Fourth failure locks the account (T4)
    actor.send({ type: 'SUBMIT' });
    actor.send({ type: 'FAILURE' }); // failCount=4, locked

    expect(actor.getSnapshot().value).toBe('locked');

    // T6: unlock resets failCount
    actor.send({ type: 'UNLOCK' });
    const snapshot = actor.getSnapshot();
    expect(snapshot.value).toBe('loggedOut');
    expect(snapshot.context.failCount).toBe(0);
  });

  test('transition-pair T3→T1: can retry after invalid credentials', () => {
    const actor = buildActor();

    actor.send({ type: 'SUBMIT' });
    actor.send({ type: 'FAILURE' }); // → loggedOut

    // Retry (T3→T1)
    actor.send({ type: 'SUBMIT' });
    expect(actor.getSnapshot().value).toBe('loggingIn');

    actor.send({ type: 'SUCCESS' });
    expect(actor.getSnapshot().value).toBe('loggedIn');
  });
});

Implementing FSM Tests in pytest

For Python projects, you can implement the same FSM directly using a dictionary-based transition table:

from dataclasses import dataclass, field
from typing import Dict, Tuple, Optional
import pytest

@dataclass
class AuthMachine:
    state: str = "loggedOut"
    fail_count: int = 0

    TRANSITIONS: Dict[Tuple[str, str], str] = field(default_factory=lambda: {
        ("loggedOut", "SUBMIT"): "loggingIn",
        ("loggingIn", "SUCCESS"): "loggedIn",
        ("loggingIn", "FAILURE"): "loggedOut",  # guard checked separately
        ("loggedIn", "LOGOUT"): "loggedOut",
        ("locked", "UNLOCK"): "loggedOut",
    }, init=False, repr=False)

    def send(self, event: str) -> None:
        key = (self.state, event)
        if event == "FAILURE" and self.state == "loggingIn":
            self.fail_count += 1
            if self.fail_count >= 4:
                self.state = "locked"
            else:
                self.state = "loggedOut"
        elif event == "UNLOCK" and self.state == "locked":
            self.fail_count = 0
            self.state = "loggedOut"
        elif key in self.TRANSITIONS:
            self.state = self.TRANSITIONS[key]
        else:
            raise ValueError(f"No transition for ({self.state}, {event})")


class TestAuthFSMTransitionCoverage:

    def test_t1_t2_t5_happy_path(self):
        m = AuthMachine()
        m.send("SUBMIT")
        assert m.state == "loggingIn"  # T1
        m.send("SUCCESS")
        assert m.state == "loggedIn"  # T2
        m.send("LOGOUT")
        assert m.state == "loggedOut"  # T5

    def test_t3_failure_increments_fail_count(self):
        m = AuthMachine()
        m.send("SUBMIT")
        m.send("FAILURE")
        assert m.state == "loggedOut"  # T3
        assert m.fail_count == 1

    def test_t4_t6_lockout_and_unlock(self):
        m = AuthMachine()
        for _ in range(3):
            m.send("SUBMIT")
            m.send("FAILURE")
        # Still not locked after 3 failures
        assert m.state == "loggedOut"
        assert m.fail_count == 3

        m.send("SUBMIT")
        m.send("FAILURE")  # 4th failure → T4
        assert m.state == "locked"

        m.send("UNLOCK")  # T6
        assert m.state == "loggedOut"
        assert m.fail_count == 0

    def test_invalid_transition_raises(self):
        m = AuthMachine()
        with pytest.raises(ValueError):
            m.send("LOGOUT")  # No transition from loggedOut on LOGOUT

    @pytest.mark.parametrize("events,expected_state", [
        (["SUBMIT", "SUCCESS", "LOGOUT", "SUBMIT"], "loggingIn"),   # T5→T1
        (["SUBMIT", "FAILURE", "SUBMIT"], "loggingIn"),              # T3→T1
    ])
    def test_transition_pairs(self, events, expected_state):
        m = AuthMachine()
        for event in events:
            m.send(event)
        assert m.state == expected_state

Generating Test Paths Automatically

For large FSMs, manually identifying all transitions and pairs is error-prone. You can automate it with a graph traversal:

from itertools import product

def get_all_transitions(machine_def):
    """Extract all (from_state, event, to_state) triples from a machine definition."""
    transitions = []
    for state, events in machine_def.items():
        for event, target in events.items():
            transitions.append((state, event, target))
    return transitions

def get_transition_pairs(transitions):
    """Generate all valid (T1, T2) pairs where T1.target == T2.source."""
    pairs = []
    for t1 in transitions:
        for t2 in transitions:
            if t1[2] == t2[0]:  # T1 target == T2 source
                pairs.append((t1, t2))
    return pairs

machine_def = {
    "loggedOut": {"SUBMIT": "loggingIn"},
    "loggingIn": {"SUCCESS": "loggedIn", "FAILURE": "loggedOut"},
    "loggedIn": {"LOGOUT": "loggedOut"},
}

transitions = get_all_transitions(machine_def)
pairs = get_transition_pairs(transitions)

print("Transitions to cover:")
for t in transitions:
    print(f"  {t[0]} --{t[1]}--> {t[2]}")

print(f"\nTransition pairs to cover: {len(pairs)}")

Common Mistakes in FSM Testing

Testing only the happy path — the most common failure. Testers verify that the system reaches the desired final state but never test invalid transitions or failure recovery.

Missing guard conditions — guards (like failCount >= 4) are often where bugs live. Make sure each guarded transition has at least one test where the guard is true and one where it is false.

Not testing self-transitions — a transition from a state back to itself (e.g., staying on the error page after a failed login) often has side effects (counter increment, error message update) that need verification.

Ignoring context/data — state alone is not enough. After an unlock, failCount should be zero. Tests must verify context, not just state labels.

Conclusion

Finite state machines give you a rigorous vocabulary for describing system behavior and a systematic method for deriving tests from that description. Transition coverage provides a reliable safety net for stateful systems; transition-pair coverage catches the interaction bugs that single-transition tests miss.

XState makes FSM modeling practical for JavaScript and TypeScript projects, providing a testable, visualizable, and framework-independent machine definition. The same concepts translate directly to Python using dictionary-based transition tables or dedicated libraries like python-statemachine.

The state explosion problem is real but manageable through hierarchical modeling, scoped abstractions, and pragmatic coverage criteria. Start with transition coverage, instrument the model to generate test paths automatically, and invest in transition-pair coverage for the highest-risk workflows.