TextTest Framework Deep Dive: Approval Testing for Python CLI Tools

TextTest Framework Deep Dive: Approval Testing for Python CLI Tools

Most testing frameworks ask you to write assertions: explicit statements about what the output should be. TextTest takes the opposite approach. Run the program, capture everything it produces, and make that the approved baseline. Future runs either match it or they don't.

This sounds simple, and it is — which is exactly why it works for a class of problems where assertion-based testing becomes painful. CLI tools, batch processing scripts, code generators, report renderers, anything that produces substantial text output becomes significantly easier to test when you stop trying to assert on every field and start treating the entire output as the artifact under test.

How TextTest Works

TextTest is a Python-based acceptance testing framework built around the idea that the system under test's stdout, stderr, and any files it writes are the ground truth. A test consists of:

  1. A way to invoke the program (command line, arguments, input files)
  2. An "approved" set of expected output files
  3. A diff engine that compares actual vs approved

When a test passes, nothing special happens. When it fails, TextTest shows you a diff between what you got and what you approved. When you decide the new output is correct, you "approve" it — which means copying the actual output over the approved baseline. The test then passes on subsequent runs.

The sandbox model is central to TextTest's design. Each test run gets its own temporary directory (the sandbox) where the program runs and writes output. TextTest manages setup and teardown, so tests don't contaminate each other and you can inspect the sandbox contents after a failure.

Installation and Basic Setup

pip install texttest

TextTest needs a test suite configuration. Create a directory structure:

myproject/
  testsuite/
    config.myapp          # main configuration
    TestSuite             # list of test directories
    FirstTest/
      options             # command line arguments
      stdout              # approved stdout
    SecondTest/
      options
      stdout
      myapp.output        # approved file output

The config.myapp file tells TextTest how to run your application:

executable:/path/to/myapp.py
interpreter:python3

For a CLI tool that processes CSV files and produces a report:

executable:${TEXTTEST_HOME}/../src/report_generator.py
interpreter:python3
checkout_location:${TEXTTEST_HOME}/..

Writing Your First TextTest Test

Say you have a script that reads a data file and outputs statistics:

# src/stats.py
import sys
import csv
from collections import Counter

def main():
    filename = sys.argv[1]
    with open(filename) as f:
        reader = csv.DictReader(f)
        rows = list(reader)
    
    print(f"Total records: {len(rows)}")
    
    if 'status' in rows[0]:
        counts = Counter(r['status'] for r in rows)
        print("Status breakdown:")
        for status, count in sorted(counts.items()):
            print(f"  {status}: {count}")

if __name__ == '__main__':
    main()

Create a test by placing an input file and setting the options:

testsuite/BasicStats/
  options           # contains: input_data.csv
  input_data.csv    # the test data
  stdout            # the approved output

The options file contains the command-line arguments: input_data.csv

Run TextTest once to generate the output, inspect it, and if it looks right, approve it. The stdout file becomes:

Total records: 5
Status breakdown:
  active: 3
  inactive: 2

From this point, any change to stats.py that alters this output will cause the test to fail — and you'll see exactly what changed in the diff.

Handling Dynamic Content

The hard problem with any snapshot/approval testing is dynamic values. Timestamps, process IDs, random identifiers, and dates will break your tests every run if not handled.

TextTest provides several mechanisms. The simplest is text filters defined in config:

[default]
run_dependent_text:{{TIMESTAMP}}:{regexp}[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}

This tells TextTest to replace any matching timestamp pattern with {{TIMESTAMP}} before comparing. Your approved stdout contains the placeholder, not the actual value.

For more complex cases, you can use replace filters that scrub entire lines:

[default]
run_dependent_text:{line}:^Processing batch .*$

This removes any line matching that pattern from both actual and approved output before comparison.

Absolute file paths are another common source of instability. TextTest has built-in support for this — it automatically replaces the sandbox directory path with a placeholder, so your approved output doesn't contain machine-specific paths.

# In approved stdout, TextTest replaces /tmp/texttest-sandbox-12345/ with SANDBOX/
Written output to SANDBOX/results.csv

The Sandbox Model

Each test gets a fresh temporary directory. Any files your program reads or writes should go through this directory. TextTest copies test data files into the sandbox before running the test, so your program always finds them at a relative path.

Configure this in the test's options or the suite config:

copy_test_path:input_data.csv

Files the program writes to the sandbox are compared against approved versions if you've specified them. This extends the coverage beyond stdout — you can approve the exact contents of every output file your program produces.

testsuite/ProcessAndWrite/
  options
  input.csv
  stdout
  output_report.txt     # approved contents of the file written to sandbox

After a test run, the sandbox directory persists until TextTest cleans up, giving you a place to inspect unexpected output before deciding whether to approve it.

Running in CI

TextTest has a batch mode designed for CI:

texttest -b cirun -a myapp

The -b flag runs in batch mode, collecting results and writing a report. Configure batch settings in config.myapp:

[batch_cirun]
batch_result_files:${TEXTTEST_HOME}/results
batch_recipients:team@example.com

For CI pipelines, the key concern is exit codes. TextTest exits non-zero when tests fail, which is what you need for CI to mark the build red.

A typical GitHub Actions step:

- name: Run TextTest suite
  run: |
    texttest -b ci -a myapp -d testsuite/
  env:
    TEXTTEST_HOME: ${{ github.workspace }}/testsuite

Approving new baselines in CI requires care. The typical workflow is:

  1. Developer runs TextTest locally, sees the diff
  2. If the change is intentional, runs texttest -f (or uses the GUI) to approve
  3. Commits the updated approved files alongside the code change
  4. CI verifies the approved files match the actual output

This makes the approved files a first-class artifact of the codebase — they live in version control and change with the code.

Comparison With pytest-snapshot

pytest-snapshot and similar tools (syrupy, pytest-regressions) provide snapshot testing within the pytest ecosystem. The comparison with TextTest reveals a meaningful difference in philosophy.

pytest-snapshot stores snapshots as Python objects serialized to files. It works at the unit test level — you call snapshot.assert_match(some_value) inside a test function and the framework handles serialization and comparison.

def test_report_output(snapshot):
    result = generate_report(data)
    snapshot.assert_match(result, "report_output")

TextTest works at the process level. It runs the actual program and captures everything the process produces. This matters when you're testing:

  • Programs that write to multiple files
  • Programs with complex logging that you want to lock down
  • Programs where the interaction between components is the thing under test
  • Legacy systems where you don't have easy programmatic access to the output

TextTest's GUI is also genuinely useful — it provides a visual diff interface that makes reviewing and approving changes faster than command-line diff tools. For development workflows where you're iterating rapidly, this difference in ergonomics matters.

The tradeoff is that TextTest has more infrastructure overhead than pytest-snapshot. You need to set up the config, manage the testsuite directory structure, and learn the TextTest concepts. pytest-snapshot drops into existing pytest suites with almost no friction.

For Python CLI tools and services with substantial text output, TextTest's process-level approach often wins. For unit testing functions that return structured data, pytest-snapshot or syrupy is usually the better fit.

Organizing a TextTest Suite for a Real Project

A realistic suite structure for a data processing tool:

testsuite/
  config.etl_tool
  TestSuite              # lists: UnitTests IntegrationTests
  UnitTests/
    TestSuite            # lists: ParseCSV TransformData ValidateOutput
    ParseCSV/
      options
      sample.csv
      stdout
    TransformData/
      options
      input.json
      stdout
      transformed.json
    ValidateOutput/
      options
      data.csv
      stdout
      errors.log
  IntegrationTests/
    TestSuite            # lists: FullPipeline ErrorHandling
    FullPipeline/
      options
      raw_data.csv
      stdout
      final_report.csv
      audit_log.txt

The hierarchical structure lets you run subsets: texttest -a etl_tool -t UnitTests runs only the unit tests. This matters as the suite grows.

TextTest also supports test categories and selection by keywords, which becomes important when you have hundreds of tests and need to run only the ones affected by your change.

Approval testing with TextTest doesn't replace unit tests — it adds a different layer of coverage. The approved baselines document what the system actually does, not just what you think individual functions do. When you're working on a CLI tool that's grown beyond what manual testing can cover, that distinction becomes the difference between safe refactoring and guessing.

HelpMeTest's continuous monitoring approach applies the same idea at the live system level — approved behavior becomes the baseline, and deviations trigger alerts rather than test failures.

Start now free