Visual Testing

Reg-Suit: Visual Regression CI Integration

HelpMeTest

HelpMeTest

— 5 min read
Reg-Suit: Visual Regression CI Integration

Reg-Suit is a visual regression testing framework built around a plugin architecture. Unlike tools that are opinionated about screenshot capture, Reg-Suit focuses on the comparison and reporting layer — you bring your own screenshots, and Reg-Suit handles storage, comparison, and CI status reporting.

Architecture

Reg-Suit separates three concerns:

  1. Screenshot capture — you handle this (Puppeteer, Playwright, Storybook, anything)
  2. Storage — reg-suit plugins for S3, GCS, or local filesystem
  3. Comparison — reg-cli compares images pixel by pixel
  4. Notification — GitHub, Slack, or custom webhooks

This separation means Reg-Suit integrates with any existing screenshot workflow.

Installation

npm install --save-dev reg-suit
npx reg-suit init

The interactive init wizard asks which plugins to install:

? Which plugins do you use? (Press <space> to select)
 ◯ reg-keygen-git-hash-plugin       - Generates snapshot keys from git hash
 ◯ reg-simple-keygen-plugin         - Generates keys from CLI argument
❯◉ reg-publish-s3-plugin            - Stores screenshots in AWS S3
 ◯ reg-publish-gcs-plugin           - Stores screenshots in GCS
 ◯ reg-notify-github-plugin         - Posts status to GitHub
 ◯ reg-notify-slack-plugin          - Posts results to Slack

For most CI setups, select:

  • reg-keygen-git-hash-plugin
  • reg-publish-s3-plugin (or GCS)
  • reg-notify-github-plugin

Configuration

After init, configure regconfig.json:

{
  "core": {
    "workingDir": ".reg",
    "actualDir": "screenshots",
    "threshold": 0,
    "thresholdRate": 0.01,
    "matchingThreshold": 0,
    "ximgproc": false,
    "enableAntialias": false,
    "concurrency": 4
  },
  "plugins": {
    "reg-keygen-git-hash-plugin": {
      "expectedKey": null,
      "actualKey": null,
      "keyPrefix": ""
    },
    "reg-publish-s3-plugin": {
      "bucketName": "my-visual-regression-bucket",
      "acl": "private",
      "pathPrefix": "reg"
    },
    "reg-notify-github-plugin": {
      "clientId": "your-github-app-client-id",
      "setCommitStatus": true
    }
  }
}

Capturing Screenshots

Reg-Suit doesn't capture screenshots — it compares them. Capture with any tool and put screenshots in the actualDir (default: screenshots/).

With Puppeteer

// capture.js
const puppeteer = require('puppeteer');
const fs = require('fs');
const path = require('path');

const components = [
  { name: 'button-primary',   url: 'http://localhost:6006/iframe.html?id=button--primary' },
  { name: 'button-secondary', url: 'http://localhost:6006/iframe.html?id=button--secondary' },
  { name: 'input-default',    url: 'http://localhost:6006/iframe.html?id=input--default' },
  { name: 'modal-open',       url: 'http://localhost:6006/iframe.html?id=modal--open' },
];

(async () => {
  const browser = await puppeteer.launch({ args: ['--no-sandbox'] });
  const outputDir = 'screenshots';
  
  fs.mkdirSync(outputDir, { recursive: true });
  
  for (const { name, url } of components) {
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });
    await page.goto(url);
    await page.waitForSelector('#storybook-root', { visible: true });
    
    // Disable animations
    await page.addStyleTag({
      content: '*, *::before, *::after { animation: none !important; transition: none !important; }'
    });
    
    await page.screenshot({
      path: path.join(outputDir, `${name}.png`),
      clip: await page.$eval('#storybook-root', el => {
        const { x, y, width, height } = el.getBoundingClientRect();
        return { x, y, width, height };
      }),
    });
    
    await page.close();
  }
  
  await browser.close();
  console.log(`Captured ${components.length} screenshots`);
})();

With Playwright

// capture-playwright.js
const { chromium } = require('playwright');
const path = require('path');
const fs = require('fs');

const viewports = [
  { name: 'desktop', width: 1280, height: 800 },
  { name: 'mobile',  width: 375,  height: 812 },
];

const stories = require('./storybook-static/stories.json');

(async () => {
  const browser = await chromium.launch();
  const outputDir = 'screenshots';
  fs.mkdirSync(outputDir, { recursive: true });
  
  for (const viewport of viewports) {
    const context = await browser.newContext({ viewport });
    
    for (const [storyId, story] of Object.entries(stories.stories)) {
      const page = await context.newPage();
      await page.goto(`http://localhost:6006/iframe.html?id=${storyId}`);
      await page.waitForLoadState('networkidle');
      
      const element = await page.$('#storybook-root');
      await element.screenshot({
        path: path.join(outputDir, `${viewport.name}-${storyId}.png`),
      });
      
      await page.close();
    }
    
    await context.close();
  }
  
  await browser.close();
})();

Running Reg-Suit

With screenshots in screenshots/, run:

npx reg-suit run

This:

  1. Generates a key from the current git hash
  2. Downloads baseline screenshots from S3 for the parent commit
  3. Compares actual vs baseline
  4. Uploads current screenshots to S3
  5. Reports results (console + GitHub status)

Output:

✔ Detected previous snapshot key: abc123f
✔ Downloaded 42 files from S3
✔ Compared 42 files
  ✔ 38 files passed
  ✘ 4 files failed (pixel diff detected)
✔ Uploaded 42 files to S3
✔ Posted status to GitHub

S3 Configuration

Create a dedicated S3 bucket:

aws s3 mb s3://my-company-visual-regression
aws s3api put-bucket-acl --bucket my-company-visual-regression --acl private

IAM policy for CI:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-company-visual-regression",
        "arn:aws:s3:::my-company-visual-regression/*"
      ]
    }
  ]
}

Set credentials in CI as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

GitHub Status Integration

Reg-Suit uses a GitHub App for status checks. Register at https://reg-viz.github.io/reg-suit/ to get a clientId.

After configuration, Reg-Suit posts:

  • ✅ Green check: no visual differences
  • ❌ Red X: differences detected, with link to HTML report
  • 📊 Report link: side-by-side comparison for each changed screenshot

CI Pipeline

name: Visual Regression

on: [pull_request]

jobs:
  visual-regression:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      
      - uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: npm
      
      - run: npm ci
      
      - name: Build Storybook
        run: npm run build-storybook
      
      - name: Start Storybook server
        run: npx serve storybook-static -p 6006 &
      
      - name: Wait for Storybook
        run: npx wait-on http://localhost:6006
      
      - name: Capture screenshots
        run: node capture.js
      
      - name: Run reg-suit comparison
        env:
          AWS_ACCESS_KEY_ID:     ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          REG_NOTIFY_CLIENT_ID:  ${{ secrets.REG_SUIT_CLIENT_ID }}
        run: npx reg-suit run
      
      - name: Upload report on failure
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: reg-suit-report
          path: .reg/report/

Comparison Configuration

Fine-tune comparison in regconfig.json:

{
  "core": {
    "threshold":      0,       // pixel count threshold (absolute)
    "thresholdRate":  0.01,    // percent threshold (1% of pixels can differ)
    "enableAntialias": true,   // ignore antialiasing differences
    "ximgproc":       false,   // use advanced edge-aware comparison
    "concurrency":    8        // parallel comparisons
  }
}

enableAntialias: true reduces false positives from font rendering differences between machines. Start with this enabled.

For stricter comparison (design system enforcement):

{
  "core": {
    "threshold":     0,
    "thresholdRate": 0,
    "enableAntialias": false
  }
}

Local Comparison with reg-cli

For local development without uploading to S3, use reg-cli directly:

npm install --save-dev reg-cli

# Compare two directories
npx reg-cli screenshots/ screenshots-baseline/ -R report

<span class="hljs-comment"># View report
open report/index.html

This generates an HTML report with side-by-side comparison, no S3 or network required.

Workflow for Approving Changes

When a PR changes component appearance intentionally:

  1. Merge the PR — this becomes the new "expected" state for that git hash
  2. The next PR's reg-suit run will use the merged commit as baseline
  3. Visual changes from the merged PR are now the baseline — no explicit approval step needed

This git-hash-based approach means baselines evolve automatically with your codebase. The baseline for any commit is always the closest ancestor in git history that has a stored snapshot.

Summary

Reg-Suit's plugin architecture makes it uniquely flexible:

  • Bring your own screenshots — Puppeteer, Playwright, or any tool
  • Pluggable storage — S3, GCS, or local filesystem
  • Git-hash keying — baselines tied to git history automatically
  • GitHub integration — native PR status checks
  • reg-cli for local comparison without cloud dependencies

The separation between screenshot capture and comparison is Reg-Suit's main advantage — you can plug it into an existing screenshot workflow without changing how screenshots are taken.

Read more