TextTest Framework Deep Dive: Approval Testing for Python CLI Tools
Most testing frameworks ask you to write assertions: explicit statements about what the output should be. TextTest takes the opposite approach. Run the program, capture everything it produces, and make that the approved baseline. Future runs either match it or they don't.
This sounds simple, and it is — which is exactly why it works for a class of problems where assertion-based testing becomes painful. CLI tools, batch processing scripts, code generators, report renderers, anything that produces substantial text output becomes significantly easier to test when you stop trying to assert on every field and start treating the entire output as the artifact under test.
How TextTest Works
TextTest is a Python-based acceptance testing framework built around the idea that the system under test's stdout, stderr, and any files it writes are the ground truth. A test consists of:
- A way to invoke the program (command line, arguments, input files)
- An "approved" set of expected output files
- A diff engine that compares actual vs approved
When a test passes, nothing special happens. When it fails, TextTest shows you a diff between what you got and what you approved. When you decide the new output is correct, you "approve" it — which means copying the actual output over the approved baseline. The test then passes on subsequent runs.
The sandbox model is central to TextTest's design. Each test run gets its own temporary directory (the sandbox) where the program runs and writes output. TextTest manages setup and teardown, so tests don't contaminate each other and you can inspect the sandbox contents after a failure.
Installation and Basic Setup
pip install texttestTextTest needs a test suite configuration. Create a directory structure:
myproject/
testsuite/
config.myapp # main configuration
TestSuite # list of test directories
FirstTest/
options # command line arguments
stdout # approved stdout
SecondTest/
options
stdout
myapp.output # approved file outputThe config.myapp file tells TextTest how to run your application:
executable:/path/to/myapp.py
interpreter:python3For a CLI tool that processes CSV files and produces a report:
executable:${TEXTTEST_HOME}/../src/report_generator.py
interpreter:python3
checkout_location:${TEXTTEST_HOME}/..Writing Your First TextTest Test
Say you have a script that reads a data file and outputs statistics:
# src/stats.py
import sys
import csv
from collections import Counter
def main():
filename = sys.argv[1]
with open(filename) as f:
reader = csv.DictReader(f)
rows = list(reader)
print(f"Total records: {len(rows)}")
if 'status' in rows[0]:
counts = Counter(r['status'] for r in rows)
print("Status breakdown:")
for status, count in sorted(counts.items()):
print(f" {status}: {count}")
if __name__ == '__main__':
main()Create a test by placing an input file and setting the options:
testsuite/BasicStats/
options # contains: input_data.csv
input_data.csv # the test data
stdout # the approved outputThe options file contains the command-line arguments: input_data.csv
Run TextTest once to generate the output, inspect it, and if it looks right, approve it. The stdout file becomes:
Total records: 5
Status breakdown:
active: 3
inactive: 2From this point, any change to stats.py that alters this output will cause the test to fail — and you'll see exactly what changed in the diff.
Handling Dynamic Content
The hard problem with any snapshot/approval testing is dynamic values. Timestamps, process IDs, random identifiers, and dates will break your tests every run if not handled.
TextTest provides several mechanisms. The simplest is text filters defined in config:
[default]
run_dependent_text:{{TIMESTAMP}}:{regexp}[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}This tells TextTest to replace any matching timestamp pattern with {{TIMESTAMP}} before comparing. Your approved stdout contains the placeholder, not the actual value.
For more complex cases, you can use replace filters that scrub entire lines:
[default]
run_dependent_text:{line}:^Processing batch .*$This removes any line matching that pattern from both actual and approved output before comparison.
Absolute file paths are another common source of instability. TextTest has built-in support for this — it automatically replaces the sandbox directory path with a placeholder, so your approved output doesn't contain machine-specific paths.
# In approved stdout, TextTest replaces /tmp/texttest-sandbox-12345/ with SANDBOX/
Written output to SANDBOX/results.csvThe Sandbox Model
Each test gets a fresh temporary directory. Any files your program reads or writes should go through this directory. TextTest copies test data files into the sandbox before running the test, so your program always finds them at a relative path.
Configure this in the test's options or the suite config:
copy_test_path:input_data.csvFiles the program writes to the sandbox are compared against approved versions if you've specified them. This extends the coverage beyond stdout — you can approve the exact contents of every output file your program produces.
testsuite/ProcessAndWrite/
options
input.csv
stdout
output_report.txt # approved contents of the file written to sandboxAfter a test run, the sandbox directory persists until TextTest cleans up, giving you a place to inspect unexpected output before deciding whether to approve it.
Running in CI
TextTest has a batch mode designed for CI:
texttest -b cirun -a myappThe -b flag runs in batch mode, collecting results and writing a report. Configure batch settings in config.myapp:
[batch_cirun]
batch_result_files:${TEXTTEST_HOME}/results
batch_recipients:team@example.comFor CI pipelines, the key concern is exit codes. TextTest exits non-zero when tests fail, which is what you need for CI to mark the build red.
A typical GitHub Actions step:
- name: Run TextTest suite
run: |
texttest -b ci -a myapp -d testsuite/
env:
TEXTTEST_HOME: ${{ github.workspace }}/testsuiteApproving new baselines in CI requires care. The typical workflow is:
- Developer runs TextTest locally, sees the diff
- If the change is intentional, runs
texttest -f(or uses the GUI) to approve - Commits the updated approved files alongside the code change
- CI verifies the approved files match the actual output
This makes the approved files a first-class artifact of the codebase — they live in version control and change with the code.
Comparison With pytest-snapshot
pytest-snapshot and similar tools (syrupy, pytest-regressions) provide snapshot testing within the pytest ecosystem. The comparison with TextTest reveals a meaningful difference in philosophy.
pytest-snapshot stores snapshots as Python objects serialized to files. It works at the unit test level — you call snapshot.assert_match(some_value) inside a test function and the framework handles serialization and comparison.
def test_report_output(snapshot):
result = generate_report(data)
snapshot.assert_match(result, "report_output")TextTest works at the process level. It runs the actual program and captures everything the process produces. This matters when you're testing:
- Programs that write to multiple files
- Programs with complex logging that you want to lock down
- Programs where the interaction between components is the thing under test
- Legacy systems where you don't have easy programmatic access to the output
TextTest's GUI is also genuinely useful — it provides a visual diff interface that makes reviewing and approving changes faster than command-line diff tools. For development workflows where you're iterating rapidly, this difference in ergonomics matters.
The tradeoff is that TextTest has more infrastructure overhead than pytest-snapshot. You need to set up the config, manage the testsuite directory structure, and learn the TextTest concepts. pytest-snapshot drops into existing pytest suites with almost no friction.
For Python CLI tools and services with substantial text output, TextTest's process-level approach often wins. For unit testing functions that return structured data, pytest-snapshot or syrupy is usually the better fit.
Organizing a TextTest Suite for a Real Project
A realistic suite structure for a data processing tool:
testsuite/
config.etl_tool
TestSuite # lists: UnitTests IntegrationTests
UnitTests/
TestSuite # lists: ParseCSV TransformData ValidateOutput
ParseCSV/
options
sample.csv
stdout
TransformData/
options
input.json
stdout
transformed.json
ValidateOutput/
options
data.csv
stdout
errors.log
IntegrationTests/
TestSuite # lists: FullPipeline ErrorHandling
FullPipeline/
options
raw_data.csv
stdout
final_report.csv
audit_log.txtThe hierarchical structure lets you run subsets: texttest -a etl_tool -t UnitTests runs only the unit tests. This matters as the suite grows.
TextTest also supports test categories and selection by keywords, which becomes important when you have hundreds of tests and need to run only the ones affected by your change.
Approval testing with TextTest doesn't replace unit tests — it adds a different layer of coverage. The approved baselines document what the system actually does, not just what you think individual functions do. When you're working on a CLI tool that's grown beyond what manual testing can cover, that distinction becomes the difference between safe refactoring and guessing.
HelpMeTest's continuous monitoring approach applies the same idea at the live system level — approved behavior becomes the baseline, and deviations trigger alerts rather than test failures.