Approval Testing for Complex Outputs: JSON, HTML, and SQL

The problem with complex outputs isn't that they're hard to test — it's that the obvious way to test them is wrong. When your API endpoint returns a JSON object with twenty fields, the instinct is to assert on each one: assertEquals(result.status, "active"), assertEquals(result.count, 42), assertNotNull(result.id). You end up with tests that are fragile (fail whenever the response structure changes), incomplete (only check the fields you remembered to assert on), and tedious to write.

Approval testing inverts this. Capture the entire output once, review it, call it approved, and then any deviation — including fields you didn't think to check — triggers a failure. The test coverage is comprehensive by default.

The Core Pattern

With ApprovalTests.Java, the pattern is:

@Test
public void testUserApiResponse() throws Exception {
    UserResponse response = userService.getUser(42);
    Approvals.verify(response.toJson());
}

On the first run, this fails because no approved file exists. ApprovalTests writes the actual output to UserServiceTest.testUserApiResponse.received.txt and leaves it for you to inspect. When you're satisfied the output is correct, you rename it to UserServiceTest.testUserApiResponse.approved.txt. The test then passes until the output changes.

The approved file lives in version control alongside the test. It documents what the output actually is — not just the few fields you chose to assert on.

Snapshotting JSON API Responses

Raw JSON dumps are hard to review and harder to diff. ApprovalTests.Java has built-in JSON support that normalizes and pretty-prints before comparison:

@Test
public void testOrderApiResponse() throws Exception {
    String json = orderController.getOrder(123).getBody();
    Approvals.verifyJson(json);
}

The approved file becomes nicely formatted:

{
  "id": 123,
  "status": "shipped",
  "items": [
    {
      "sku": "WIDGET-001",
      "quantity": 2,
      "price": 29.99
    }
  ],
  "total": 59.98,
  "shippedAt": "{{SCRUBBED}}"
}

The shippedAt field contains a timestamp that changes every test run. This is where scrubbers come in.

Scrubbing Dynamic Values

ApprovalTests provides scrubbers — functions that transform the output before comparison. You chain them to remove or replace unstable values:

@Test
public void testOrderApiResponse() throws Exception {
    String json = orderController.getOrder(123).getBody();
    Approvals.verifyJson(json, new Options()
        .withScrubber(new RegExScrubber(
            "\"shippedAt\": \"[^\"]+\"",
            "\"shippedAt\": \"{{SCRUBBED}}\""
        ))
    );
}

For IDs and UUIDs that change between test runs, a more systematic approach works better. Create a scrubber that replaces all UUID-shaped strings:

public class UuidScrubber implements StringScrubber {
    private static final Pattern UUID_PATTERN = Pattern.compile(
        "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"
    );
    private final AtomicInteger counter = new AtomicInteger(0);
    private final Map<String, String> seen = new HashMap<>();
    
    @Override
    public String scrub(String input) {
        Matcher m = UUID_PATTERN.matcher(input);
        StringBuffer sb = new StringBuffer();
        while (m.find()) {
            String uuid = m.group();
            String replacement = seen.computeIfAbsent(
                uuid, 
                k -> "UUID-" + counter.incrementAndGet()
            );
            m.appendReplacement(sb, replacement);
        }
        m.appendTail(sb);
        return sb.toString();
    }
}

This replaces UUIDs with stable placeholders (UUID-1, UUID-2) that are consistent within a single test run, so the structure of references between objects is preserved.

Python: The approvals Library

For Python, the approvaltests library follows the same pattern:

from approvaltests import verify, verify_as_json
import json

def test_product_search_response():
    results = product_service.search("widget")
    verify_as_json(results)

For scrubbing:

from approvaltests import verify_as_json
from approvaltests.scrubbers import combine_scrubbers, create_regex_scrubber

def test_product_search_with_scrubbing():
    results = product_service.search("widget")
    scrubber = combine_scrubbers(
        create_regex_scrubber(r'"created_at": "[^"]+"', '"created_at": "<DATE>"'),
        create_regex_scrubber(r'"id": \d+', '"id": <ID>')
    )
    verify_as_json(results, scrubber=scrubber)

Approving HTML Rendering Output

HTML is one of the hardest outputs to assert on with traditional tests. Checking that specific text appears, that specific elements exist, that styles are applied — all of it tends to produce brittle test suites that break whenever the template changes.

Approval testing captures the full rendered HTML and diffs it. Changes to layout, text, or structure all show up as diffs:

@Test
public void testInvoiceHtmlRendering() throws Exception {
    Invoice invoice = createTestInvoice();
    String html = invoiceRenderer.render(invoice);
    Approvals.verifyHtml(html);
}

verifyHtml formats the HTML before comparison, so minor whitespace differences don't cause spurious failures. The approved file shows you exactly what the rendered template looks like:

<html>
  <head>
    <title>Invoice #INV-001</title>
  </head>
  <body>
    <h1>Invoice</h1>
    <table>
      <tr>
        <td>Widget Pro</td>
        <td>$49.99</td>
      </tr>
    </table>
  </body>
</html>

For HTML with dynamic values (generated IDs, inline styles with random values, CSRF tokens), scrubbers work the same way as with JSON:

Approvals.verifyHtml(html, new Options()
    .withScrubber(new RegExScrubber(
        "csrf_token\" value=\"[^\"]+\"",
        "csrf_token\" value=\"{{SCRUBBED}}\""
    ))
);

Approving SQL Query Results

Data pipeline tests often need to verify that a SQL query returns the expected result set. The traditional approach is to assert on specific rows and columns, which misses structural issues and becomes painful to maintain as schemas evolve.

A better pattern: execute the query against a test database, format the results as a string, and approve them:

def test_monthly_sales_report_query():
    db = get_test_database()
    results = db.execute(MONTHLY_SALES_QUERY, month="2024-01")
    
    # Format results as aligned text table
    output = format_results_as_table(results)
    verify(output)

def format_results_as_table(results):
    if not results:
        return "(empty result set)"
    
    headers = results[0].keys()
    col_widths = {h: max(len(h), max(len(str(r[h])) for r in results)) 
                  for h in headers}
    
    separator = "-+-".join("-" * w for w in col_widths.values())
    header_row = " | ".join(h.ljust(col_widths[h]) for h in headers)
    
    lines = [header_row, separator]
    for row in results:
        lines.append(" | ".join(str(row[h]).ljust(col_widths[h]) for h in headers))
    
    return "\n".join(lines)

The approved output looks like:

region    | month   | total_sales | order_count
----------+---------+-------------+------------
Northeast | 2024-01 | 142350.00   | 287
Southeast | 2024-01 | 98420.00    | 201
West      | 2024-01 | 167890.00   | 334

When your query changes — a new aggregation, a different JOIN, a schema migration — the diff shows you exactly what shifted in the result set. This is dramatically easier to review than comparing arrays of objects.

Multi-Line Log Output

Logs are another area where approval testing beats assertion-based approaches. Instead of asserting on specific log lines (which requires knowing exactly what will be logged), capture the full log output and approve it:

import logging
from io import StringIO
from approvaltests import verify

def test_import_process_logging():
    log_capture = StringIO()
    handler = logging.StreamHandler(log_capture)
    handler.setLevel(logging.INFO)
    
    logger = logging.getLogger('import_service')
    logger.addHandler(handler)
    
    try:
        import_service.process_file("test_data.csv")
    finally:
        logger.removeHandler(handler)
    
    log_output = log_capture.getvalue()
    # Scrub timestamps from log lines
    scrubbed = re.sub(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}', '<TIMESTAMP>', log_output)
    verify(scrubbed)

The approved file documents exactly what the import process logs, which doubles as documentation of the process itself.

Combining Approval Testing With Existing Test Infrastructure

Approval tests don't need to replace your existing assertion-based tests — they complement them. A common pattern is to use approval testing for complex output verification while keeping traditional assertions for simple invariants:

@Test
public void testSearchResults() throws Exception {
    SearchResponse response = searchService.search("python testing");
    
    // Traditional assertions for invariants
    assertThat(response.getResults()).isNotEmpty();
    assertThat(response.getTotalCount()).isGreaterThan(0);
    
    // Approval testing for the full structure
    Approvals.verifyJson(response.toJson(), new Options()
        .withScrubber(new UuidScrubber())
    );
}

The traditional assertions catch obvious failures fast. The approval test catches subtle regressions in the full output structure.

Managing approved files in version control is the main operational concern. The *.received.* files should be in .gitignore (they're temporary artifacts from failed runs), while *.approved.* files are checked in. A good workflow is to run approval tests locally, review diffs using a tool like Beyond Compare, approve what's correct, and commit the approved files as part of the same change that modified the behavior.

HelpMeTest's continuous monitoring adds another layer to this — running the same assertions against your live API on a schedule, catching regressions that slip through before users do.