Testing

End-to-End PDF Testing Strategies: Text Extraction, Snapshot Diffs, and Accessibility Checks

HelpMeTest

HelpMeTest

— 7 min read
End-to-End PDF Testing Strategies: Text Extraction, Snapshot Diffs, and Accessibility Checks

PDF testing requires a layered approach: text content assertions, page structure checks, visual regression diffs, and accessibility validation. This guide covers strategies that work regardless of how your PDFs are generated — Puppeteer, WeasyPrint, iText, PDFBox, ReportLab, or any other tool — and shows how to combine them into a CI-ready pipeline.

Key Takeaways

Layer your tests: content → structure → visual → accessibility. Start with the cheapest assertions (text extraction), escalate to visual diffs only for high-risk changes.

Text extraction is fast but fragile. PDF text layers don't preserve HTML structure — text order may differ from visual order. Use region-based extraction for positional assertions.

Visual snapshot tests are the best regression net. For deterministic generators (same input → same bytes), snapshot tests catch CSS and layout regressions instantly.

Test the download flow, not just the bytes. Unit tests verify PDF content; browser tests verify that users can actually download, open, and read the file.

Accessibility is a requirement, not a bonus. Tagged PDFs with proper reading order and alt text are required by law in many jurisdictions. Test it explicitly.

The Four Layers of PDF Testing

PDF testing should be organized into four layers, each with different tools and trade-offs:

Layer 4: E2E / Browser    → Did the user receive a valid, usable PDF?
Layer 3: Accessibility    → Is the PDF tagged, structured, and readable?
Layer 2: Visual / Layout  → Does it look correct? No overflow, no garbling?
Layer 1: Content          → Is the required text present? Is the page count right?

Start with Layer 1 (fastest, cheapest), escalate to higher layers only for riskier changes.

Layer 1: Content Tests

Text Extraction — Universal Pattern

Regardless of generator (Puppeteer, WeasyPrint, iText, PDFBox, ReportLab), the text extraction pattern is the same:

Python (PyMuPDF):

import fitz

def extract_text(pdf_bytes: bytes) -> str:
    doc = fitz.open(stream=pdf_bytes, filetype="pdf")
    text = "".join(page.get_text() for page in doc)
    doc.close()
    return text

# In tests:
def test_invoice_number_present(pdf_bytes):
    assert "INV-2026-001" in extract_text(pdf_bytes)

Java (PDFBox):

import org.apache.pdfbox.text.PDFTextStripper;

String extractText(byte[] pdfBytes) throws IOException {
    try (PDDocument doc = Loader.loadPDF(pdfBytes)) {
        return new PDFTextStripper().getText(doc);
    }
}

// In tests:
@Test
void invoiceNumberPresent() throws IOException {
    assertTrue(extractText(pdfBytes).contains("INV-2026-001"));
}

Node.js (pdf-parse):

import pdfParse from 'pdf-parse';

const { text } = await pdfParse(pdfBuffer);
expect(text).toContain('INV-2026-001');

Page Count Assertions

Page count is the most common structural bug — template changes cause content to overflow or collapse:

# Python
doc = fitz.open(stream=pdf_bytes, filetype="pdf")
assert doc.page_count == 1, f"Expected 1 page, got {doc.page_count}"
// Java
try (PDDocument doc = Loader.loadPDF(pdfBytes)) {
    assertEquals(1, doc.getNumberOfPages());
}
// Node.js
const { numpages } = await pdfParse(pdfBuffer);
expect(numpages).toBe(1);

Region-Based Text Extraction

When you need to verify that a value appears in a specific column or section, region-based extraction prevents false positives from the same number appearing elsewhere:

import fitz

def extract_region_text(pdf_bytes: bytes, page: int, rect: tuple) -> str:
    """
    Extract text from a rectangular region.
    rect = (x0, y0, x1, y1) in PDF points (0,0 = top-left)
    """
    doc = fitz.open(stream=pdf_bytes, filetype="pdf")
    clip = fitz.Rect(*rect)
    text = doc[page].get_text(clip=clip)
    doc.close()
    return text

# A4 dimensions: 595 × 842 pts
# Test that "Total" column (right side) shows the correct total
def test_total_column_shows_correct_value(sample_pdf):
    # Right column: x=450-595, y=200-700
    right_column_text = extract_region_text(sample_pdf, 0, (450, 200, 595, 700))
    assert "7,700" in right_column_text

Layer 2: Visual Snapshot Tests

The Determinism Requirement

Visual snapshot tests only work if your PDF generator is deterministic: the same input always produces the same pixels. This is true for WeasyPrint, iText (with fixed fonts), and server-side Puppeteer with a fixed Chromium version. It's NOT true for generators that embed timestamps or random IDs in PDF metadata.

For non-deterministic generators, normalize before comparing:

import fitz
import re

def normalize_pdf_bytes(pdf_bytes: bytes) -> bytes:
    """Remove timestamps from PDF metadata before comparison."""
    doc = fitz.open(stream=pdf_bytes, filetype="pdf")
    # Clear creation/modification dates
    doc.set_metadata({"creationDate": "", "modDate": ""})

    buf = io.BytesIO()
    doc.save(buf, garbage=4, deflate=True)
    return buf.getvalue()

Rendering PDFs to PNG for Comparison

# Python
import fitz

def render_page_png(pdf_bytes: bytes, page: int = 0, dpi: int = 150) -> bytes:
    doc = fitz.open(stream=pdf_bytes, filetype="pdf")
    mat = fitz.Matrix(dpi / 72, dpi / 72)
    pix = doc[page].get_pixmap(matrix=mat)
    png = pix.tobytes("png")
    doc.close()
    return png
// Node.js
import { fromBuffer } from 'pdf2pic';

const convert = fromBuffer(pdfBuffer, { density: 150, format: 'png' });
const { buffer: pngBuffer } = await convert(1, { responseType: 'buffer' });
// Java
PDFRenderer renderer = new PDFRenderer(doc);
BufferedImage image = renderer.renderImageWithDPI(0, 150);

Snapshot Storage Strategy

Store snapshots in version control alongside tests:

tests/
  snapshots/
    pdf/
      invoice-standard-page1.png   ← baseline
      invoice-large-page1.png
      invoice-large-page2.png
  pdf/
    test_invoice.py

When a snapshot needs updating (intentional change):

  1. Delete the old snapshot file
  2. Run tests — they'll write a new baseline
  3. Review the new baseline visually
  4. Commit

Pixel Comparison Strategies

Strategy Tool Tolerance Use case
Byte-exact Built-in comparison Zero Fully deterministic generators
Pixel diff % pixelmatch (JS), pixel comparison (Py) <0.5% Minor font rendering variation
Perceptual SSIM (structural similarity) >0.95 High tolerance for rendering variation

For most production tests, pixel diff with a 0.1–0.5% tolerance is the right balance.

Layer 3: Accessibility Tests

What PDF Accessibility Requires

Accessible PDFs (PDF/UA standard) require:

  • Document title set
  • Language declaration
  • Marked (tagged) content
  • Logical reading order in the structure tree
  • Alt text on images
  • Proper heading levels

Testing with PyMuPDF

def test_pdf_accessibility_basics(sample_pdf):
    doc = fitz.open(stream=sample_pdf, filetype="pdf")

    # Check PDF is tagged (marked)
    catalog = doc.pdf_catalog()
    markinfo = doc.xref_get_key(catalog, "MarkInfo")
    assert markinfo is not None, "PDF must be marked for accessibility"

    # Check language is set
    lang = doc.xref_get_key(catalog, "Lang")
    assert lang is not None and lang != "(null)", "PDF must declare language"

    # Check title is set
    metadata = doc.metadata
    assert metadata.get("title"), "PDF must have title set for accessibility"

    doc.close()

Testing with PAC (PDF Accessibility Checker)

For full PDF/UA compliance checking, the industry standard is PAC (PDF Accessibility Checker). In CI, use verapdf:

# Install veraPDF
wget https://downloads.verapdf.org/rel/verapdf-installer.zip

<span class="hljs-comment"># Run PDF/UA check
java -jar verapdf.jar --flavour ua1 invoice.pdf
# In CI:
- name: PDF/UA accessibility check
  run: |
    java -jar verapdf.jar --flavour ua1 \
      --format json build/test-output/invoice.pdf > verapdf-report.json
    # Fail if there are validation errors
    python -c "
import json, sys
report = json.load(open('verapdf-report.json'))
errors = report[0].get('details', {}).get('failedRules', [])
if errors:
    print(f'PDF/UA violations: {len(errors)}')
    sys.exit(1)
"

Testing Alt Text on Images

def test_images_have_alt_text(sample_pdf):
    doc = fitz.open(stream=sample_pdf, filetype="pdf")
    page = doc[0]

    # Get all image references on the page
    image_list = page.get_images(full=True)

    for img in image_list:
        xref = img[0]
        # In tagged PDFs, images should have an associated structure element with Alt text
        # This requires parsing the structure tree — a simplified check:
        alt_text_key = doc.xref_get_key(xref, "Alt")
        # For logos/decorative images, alt="" is acceptable
        # For content images, alt text must be non-empty
        # This is a project-specific rule

    doc.close()

Layer 4: Browser E2E Tests

The layers above test the PDF bytes. Layer 4 tests the user experience of receiving a PDF from your application:

  1. User clicks "Download Invoice"
  2. Browser downloads the file
  3. File is a valid PDF (correct MIME type, content-disposition header)
  4. File can be opened (not corrupt)
  5. File contains expected content

These tests can't be done with unit testing — they require browser automation.

*** Test Cases ***
Invoice PDF Downloads Successfully
    Go To    ${APP_URL}/invoices/INV-2026-001
    Click Element    css=[data-testid="download-invoice-btn"]
    Wait For Download    timeout=10s
    Verify Downloaded File    file_extension=.pdf    min_size_kb=10
    Verify PDF Contains Text    INV-2026-001

Choosing the Right Extraction Library

Library Language Speed PDF text fidelity Image support Form fields
PyMuPDF (fitz) Python ★★★★★ ★★★★ ✓ ✓
pdfminer.six Python ★★★ ★★★★★ ✗ ✗
pypdf Python ★★★★ ★★★ Limited ✓
PDFBox Java ★★★★ ★★★★★ ✓ ✓
pdf-parse Node.js ★★★★ ★★★ ✗ ✗
pdfjs-dist Node.js ★★★ ★★★★ ✓ ✓

Recommendation: PyMuPDF for Python projects, PDFBox for Java, pdf-parse for quick Node.js tests (pdfjs-dist for advanced needs).

CI Pipeline Structure

A complete CI pipeline for PDF testing:

name: PDF Test Pipeline
on: [push, pull_request]

jobs:
  content-tests:
    name: Content and structure tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup
        run: pip install -r requirements-test.txt
      - name: Run content tests
        run: pytest tests/pdf/content/ -v

  visual-tests:
    name: Visual regression tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup
        run: pip install -r requirements-test.txt
      - name: Run visual tests
        run: pytest tests/pdf/visual/ -v
      - name: Upload diff artifacts on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-diffs
          path: tests/visual-diffs/

  accessibility-tests:
    name: PDF/UA accessibility tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup veraPDF
        run: |
          wget -q https://downloads.verapdf.org/rel/verapdf-installer.zip
          unzip -q verapdf-installer.zip
      - name: Generate sample PDFs
        run: python scripts/generate-test-pdfs.py
      - name: Run accessibility check
        run: java -jar verapdf/verapdf.jar --flavour ua1 *.pdf

Common PDF Testing Mistakes

1. Testing Only That a PDF Was Generated

# WRONG: only checks file existence
def test_pdf_generated():
    pdf = generate_invoice_pdf(invoice)
    assert pdf is not None  # This tells you nothing

# RIGHT: check content
def test_pdf_contains_required_content():
    pdf = generate_invoice_pdf(invoice)
    text = extract_text(pdf)
    assert "INV-2026-001" in text
    assert "Acme Corp" in text
    assert "7,700" in text

2. Snapshot Tests Without Baseline Review

Creating snapshots without reviewing them means you're snapshotting bugs. Always open the baseline PNG and verify it looks correct before committing.

3. Not Pinning the PDF Library Version

PDF rendering changes between versions. pdf-parse@1.1.1 may produce different text ordering than pdf-parse@1.1.0. Pin all PDF libraries in package.json/requirements.txt and update intentionally.

4. Testing Production Files in Tests

Don't use real customer invoices as test fixtures — they contain PII. Build synthetic fixtures with fake but realistic data.

5. Ignoring Multi-Language Content

If your PDFs contain non-Latin characters (Arabic, Chinese, Japanese), test them specifically. Font embedding, text direction, and encoding are all failure points.

Summary

PDF testing done right:

  1. Content tests — text extraction with pdf-parse, PyMuPDF, or PDFBox — cover required fields, amounts, identifiers
  2. Structure tests — page count, page size, metadata
  3. Visual tests — render to PNG, compare against baseline — catch layout regressions
  4. Accessibility tests — tagged PDF, language, alt text, reading order — use veraPDF
  5. E2E tests — browser download flow, MIME type, file integrity

Each layer catches different bugs. Running all four in CI gives you confidence that your PDF generation works correctly — from the bytes in memory to the file in the user's Downloads folder.

HelpMeTest covers Layer 4 — the browser-level download and open flow that unit tests can't reach — with plain-English test scenarios that run on a schedule and alert you when PDF downloads break.

Read more