Testing

Automated Accessibility Testing with axe-core and Lighthouse

HelpMeTest

HelpMeTest

— 9 min read
Automated Accessibility Testing with axe-core and Lighthouse

Automated accessibility testing gives you speed and consistency. Run it on every PR and you'll catch the class of violations that tools can reliably detect — missing labels, contrast failures, invalid ARIA — before they reach production. The key is understanding what each tool covers, how to integrate it into your existing test infrastructure, and how to deal with false positives without silencing legitimate failures.

This guide covers axe-core and Lighthouse in depth: the JS API, integration with Playwright and Puppeteer, violation parsing, the Lighthouse Node API, configuration, and a practical comparison of coverage.

axe-core: What It Is and How It Works

axe-core is an open-source accessibility testing engine developed by Deque. It runs in the browser (or in a headless browser context) and checks the live DOM against a rule set that maps to WCAG 2.1/2.2 and other standards.

The core API is simple: you call axe.run() on a DOM context and get back a results object containing violations, passes, incomplete, and inapplicable rules.

Direct API Usage

Install and inject axe-core into a page:

// Install
npm install axe-core

// Inject in a browser context (e.g., a test page)
const { source } = require('axe-core');
// source is the full axe-core script as a string — inject it via <script> or page.evaluate

The results object structure:

{
  violations: [
    {
      id: 'color-contrast',           // rule ID
      impact: 'serious',              // critical | serious | moderate | minor
      description: 'Ensures the contrast between foreground and background colors meets WCAG 2 AA contrast ratio thresholds',
      help: 'Elements must meet minimum color contrast ratio thresholds',
      helpUrl: 'https://dequeuniversity.com/rules/axe/4.9/color-contrast',
      nodes: [
        {
          html: '<p class="subtitle">Loading...</p>',
          failureSummary: 'Fix any of the following:\n  Element has insufficient color contrast of 2.85...',
          target: ['.subtitle'],
          any: [...],   // fix one of these
          all: [...],   // fix all of these
          none: [...]   // none of these should be true
        }
      ]
    }
  ],
  passes: [...],
  incomplete: [...],  // needs manual review
  inapplicable: [...]
}

Running axe in Playwright

Playwright is the recommended environment for production-quality automated accessibility testing because it provides a full browser with real rendering.

// Install
npm install --save-dev @axe-core/playwright axe-core

// accessibility.spec.js
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Homepage accessibility', () => {
  test('should have no WCAG 2.1 AA violations', async ({ page }) => {
    await page.goto('https://example.com');

    const results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
      .analyze();

    expect(results.violations).toEqual([]);
  });

  test('should have no violations in the navigation', async ({ page }) => {
    await page.goto('https://example.com');

    const results = await new AxeBuilder({ page })
      .include('nav')          // only test within <nav>
      .withTags(['wcag2a', 'wcag2aa'])
      .analyze();

    expect(results.violations).toEqual([]);
  });

  test('modal dialog accessibility', async ({ page }) => {
    await page.goto('https://example.com');
    await page.click('[data-testid="open-modal"]');
    await page.waitForSelector('[role="dialog"]');

    const results = await new AxeBuilder({ page })
      .include('[role="dialog"]')
      .analyze();

    expect(results.violations).toEqual([]);
  });
});

Running axe in Puppeteer

const puppeteer = require('puppeteer');
const { default: AxePuppeteer } = require('@axe-core/puppeteer');

async function runA11yAudit(url) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto(url, { waitUntil: 'networkidle0' });

  const results = await new AxePuppeteer(page)
    .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
    .analyze();

  await browser.close();
  return results;
}

runA11yAudit('https://example.com').then(results => {
  if (results.violations.length > 0) {
    console.error(`Found ${results.violations.length} accessibility violations:`);
    results.violations.forEach(violation => {
      console.error(`\n[${violation.impact.toUpperCase()}] ${violation.id}`);
      console.error(`  ${violation.help}`);
      violation.nodes.forEach(node => {
        console.error(`  - Element: ${node.target}`);
        console.error(`    ${node.failureSummary}`);
      });
    });
    process.exit(1);
  } else {
    console.log('No violations found.');
  }
});

Parsing Violations for Useful Output

Raw axe output is verbose. For CI pipelines, you want structured, actionable output:

function formatViolations(violations) {
  return violations.map(v => ({
    id: v.id,
    impact: v.impact,
    description: v.help,
    affectedElements: v.nodes.length,
    elements: v.nodes.map(n => ({
      selector: n.target.join(', '),
      html: n.html.substring(0, 150),
      fix: n.failureSummary.split('\n')[0]
    }))
  }));
}

function groupByImpact(violations) {
  const groups = { critical: [], serious: [], moderate: [], minor: [] };
  violations.forEach(v => groups[v.impact].push(v));
  return groups;
}

// Example output
const formatted = formatViolations(results.violations);
const grouped = groupByImpact(results.violations);

console.log(`
Accessibility Audit Results
===========================
Critical: ${grouped.critical.length} violations
Serious:  ${grouped.serious.length} violations
Moderate: ${grouped.moderate.length} violations
Minor:    ${grouped.minor.length} violations

Details:
`);

formatted.forEach(v => {
  console.log(`[${v.impact.toUpperCase()}] ${v.id} — ${v.affectedElements} element(s)`);
  console.log(`  ${v.description}`);
  v.elements.forEach(el => {
    console.log(`  → ${el.selector}`);
    console.log(`    Fix: ${el.fix}`);
  });
  console.log();
});

axe Tags and Rule Sets

axe organizes rules by tag. The most useful tags for WCAG conformance testing:

Tag Coverage
wcag2a WCAG 2.0 Level A
wcag2aa WCAG 2.0 Level AA
wcag21a WCAG 2.1 additions at Level A
wcag21aa WCAG 2.1 additions at Level AA
wcag22aa WCAG 2.2 additions at Level AA
best-practice Non-WCAG best practices
ACT W3C Accessibility Conformance Testing rules

For most teams, running ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'] covers WCAG 2.1 AA — the standard required by most regulations.

Configuring Custom Rules

axe-core allows you to add custom rules and disable built-in ones:

const results = await new AxeBuilder({ page })
  .options({
    rules: {
      // Disable a rule you've decided is not applicable
      'region': { enabled: false },
      // Enable a rule that's off by default
      'color-contrast-enhanced': { enabled: true }
    }
  })
  .analyze();

Adding a completely custom rule:

const axe = require('axe-core');

axe.configure({
  rules: [{
    id: 'custom-data-attribute',
    selector: '[data-interactive]',
    tags: ['custom'],
    metadata: {
      description: 'data-interactive elements must have an accessible name',
      help: 'Add aria-label or aria-labelledby to data-interactive elements'
    },
    any: ['aria-label', 'aria-labelledby', 'label']
  }],
  checks: [{
    id: 'aria-label',
    evaluate: function(node) {
      return !!node.getAttribute('aria-label');
    }
  }]
});

Lighthouse Accessibility Audits

Lighthouse is Google's web quality tool. Its accessibility category runs a subset of axe-core rules plus some Lighthouse-specific checks. It returns a score (0–100) and itemized audit results.

CLI Usage

# Install
npm install -g lighthouse

# Basic audit — outputs to HTML
lighthouse https://example.com --only-categories=accessibility

# JSON output for parsing
lighthouse https://example.com \
  --only-categories=accessibility \
  --output=json \
  --output-path=./audit.json \
  --chrome-flags="--headless"

# Specify minimum score threshold
lighthouse https://example.com \
  --only-categories=accessibility \
  --output=json \
  --output-path=./audit.json \
  --chrome-flags="--headless" && \
  node -e "
    const report = require('./audit.json');
    const score = report.categories.accessibility.score * 100;
    console.log('Accessibility score:', score);
    if (score < 90) process.exit(1);
  "

Node API

The Lighthouse Node API gives you programmatic control:

const lighthouse = require('lighthouse');
const chromeLauncher = require('chrome-launcher');

async function runLighthouseAudit(url, options = {}) {
  const chrome = await chromeLauncher.launch({
    chromeFlags: ['--headless', '--no-sandbox', '--disable-gpu']
  });

  const config = {
    extends: 'lighthouse:default',
    settings: {
      onlyCategories: ['accessibility'],
      formFactor: options.mobile ? 'mobile' : 'desktop',
      screenEmulation: options.mobile ? undefined : {
        mobile: false,
        width: 1350,
        height: 940,
        deviceScaleFactor: 1,
        disabled: false
      }
    }
  };

  const runnerResult = await lighthouse(url, {
    port: chrome.port,
    output: 'json',
    logLevel: 'error'
  }, config);

  await chrome.kill();

  return runnerResult.lhr; // Lighthouse Result object
}

async function auditPage(url) {
  const lhr = await runLighthouseAudit(url);

  const score = lhr.categories.accessibility.score * 100;
  const audits = lhr.categories.accessibility.auditRefs;

  const failed = audits
    .filter(ref => {
      const audit = lhr.audits[ref.id];
      return audit.score !== null && audit.score < 1;
    })
    .map(ref => {
      const audit = lhr.audits[ref.id];
      return {
        id: ref.id,
        title: audit.title,
        description: audit.description,
        score: audit.score,
        details: audit.details
      };
    });

  return { score, failed, url };
}

// Run audit
auditPage('https://example.com').then(result => {
  console.log(`Score: ${result.score}/100`);
  console.log(`Failed audits: ${result.failed.length}`);
  result.failed.forEach(audit => {
    console.log(`\n[FAIL] ${audit.title}`);
    if (audit.details?.items?.length) {
      audit.details.items.slice(0, 3).forEach(item => {
        console.log(`  → ${item.node?.snippet || JSON.stringify(item)}`);
      });
    }
  });
});

Parsing Lighthouse Results for CI

Lighthouse audit results have a consistent structure for items:

function extractFailedAudits(lhr) {
  const accessibilityAuditIds = lhr.categories.accessibility.auditRefs
    .map(ref => ref.id);

  return accessibilityAuditIds
    .map(id => lhr.audits[id])
    .filter(audit => audit.score !== null && audit.score < 1)
    .map(audit => ({
      id: audit.id,
      title: audit.title,
      score: audit.score,
      impact: audit.details?.debugData?.impact || 'unknown',
      affectedElements: audit.details?.items?.map(item => ({
        snippet: item.node?.snippet,
        selector: item.node?.selector,
        explanation: item.node?.explanation
      })) || []
    }));
}

axe-core vs Lighthouse: Coverage Comparison

Both tools use axe-core as their engine but Lighthouse uses a specific (and smaller) subset of axe rules plus its own additional checks.

Rule axe-core Lighthouse
color-contrast Yes Yes
image-alt Yes Yes
label Yes Yes
button-name Yes Yes
link-name Yes Yes
document-title Yes Yes
html-lang-valid Yes Yes
aria-allowed-attr Yes Yes
aria-required-attr Yes Yes
duplicate-id Yes No
landmark-one-main Yes No
region Yes No
scrollable-region-focusable Yes No
td-headers-attr Yes No
bypass (skip links) No Yes
logical-tab-order No Yes (manual)
focusable-controls No Yes
use-landmarks No Yes

Key differences:

axe-core advantages:

  • More comprehensive rule set (roughly 90+ rules vs Lighthouse's ~30–40 accessibility audits)
  • Better at element-level issues (duplicate IDs, ARIA attribute validation)
  • Direct integration into test frameworks — runs as part of your test suite
  • Can be scoped to specific DOM subtrees

Lighthouse advantages:

  • Gives a single score — useful for tracking trend over time
  • Includes some audits not in axe (logical tab order, bypass blocks)
  • Integrates with Chrome DevTools and PageSpeed Insights
  • Built-in report generation with visual output

Practical recommendation: Use axe-core in your test suite for per-PR violation checking. Use Lighthouse for trend monitoring and score tracking on production. They complement each other.

False Positive Handling

axe-core's incomplete array contains results that need manual review — axe found a potential issue but cannot determine if it's a violation without human judgment. These are not false positives; they require a decision.

True false positives — cases where axe reports a violation but the markup is actually correct — are relatively rare but do occur. Common cases:

1. Color contrast on complex backgrounds

axe computes contrast against a flat background color. If your text sits on a gradient or image, axe may report incorrect contrast because it samples a single background pixel.

Handle by disabling the rule for the specific element and documenting why:

const results = await new AxeBuilder({ page })
  .exclude('.hero-text')  // hero text on image — manually verified 7:1 contrast
  .analyze();

2. region rule with legitimate non-landmark content

The region rule requires all content to be within a landmark. Some pages have intentional off-landmark content (e.g., skip links, notifications). Disable per element:

const results = await new AxeBuilder({ page })
  .options({
    rules: { 'region': { enabled: false } }
  })
  .analyze();

3. Third-party content

Ads, embedded widgets, and analytics scripts can generate violations you can't fix. Exclude their containers:

const results = await new AxeBuilder({ page })
  .exclude('[data-ad-slot]')
  .exclude('#third-party-widget')
  .analyze();

The key principle: every suppression must be documented with a reason. Undocumented suppressions become technical debt that gets silently expanded over time.

A structured approach to suppression:

// a11y-suppressions.js — centralised, code-reviewed suppression registry
module.exports = {
  // Third-party ad iframe — violations outside our control
  excludeSelectors: ['[data-ad-container]', '#helpshift-iframe'],

  // Color contrast reported on canvas element — not applicable (drawn content)
  disabledRules: ['color-contrast'],

  // Verified manually: hero image text has 7.2:1 actual contrast
  // Automated check fails because gradient background
  disabledRulesForElements: {
    '.hero__title': ['color-contrast']
  }
};

Configuring Custom Rule Severity

For projects that want to allow some violations while blocking on others, axe supports impact-based filtering:

test('no critical or serious violations', async ({ page }) => {
  await page.goto('/');

  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
    .analyze();

  // Allow minor and moderate in initial phase, block on critical and serious
  const blockers = results.violations.filter(v =>
    ['critical', 'serious'].includes(v.impact)
  );

  if (blockers.length > 0) {
    const formatted = blockers.map(v =>
      `[${v.impact}] ${v.id}: ${v.nodes.map(n => n.target).join(', ')}`
    ).join('\n');
    throw new Error(`Blocking accessibility violations found:\n${formatted}`);
  }
});

Integrating into a Page Object Model

For teams using Page Object Models in Playwright, axe checks fit cleanly as a method:

// base-page.js
import AxeBuilder from '@axe-core/playwright';

export class BasePage {
  constructor(page) {
    this.page = page;
  }

  async checkAccessibility(options = {}) {
    const builder = new AxeBuilder({ page: this.page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']);

    if (options.include) builder.include(options.include);
    if (options.exclude) options.exclude.forEach(s => builder.exclude(s));
    if (options.disableRules) {
      const rules = {};
      options.disableRules.forEach(r => { rules[r] = { enabled: false }; });
      builder.options({ rules });
    }

    const results = await builder.analyze();
    return results;
  }
}

// checkout-page.js
export class CheckoutPage extends BasePage {
  async verifyAccessibility() {
    // Exclude third-party payment iframe
    const results = await this.checkAccessibility({
      exclude: ['#payment-iframe']
    });

    return results.violations;
  }
}

Running Audits Across Multiple Pages

For a site-wide audit at deployment time:

const lighthouse = require('lighthouse');
const chromeLauncher = require('chrome-launcher');
const fs = require('fs');

const PAGES_TO_AUDIT = [
  '/',
  '/products',
  '/checkout',
  '/account',
  '/blog'
];

const BASE_URL = process.env.SITE_URL || 'https://example.com';

async function auditAll() {
  const chrome = await chromeLauncher.launch({ chromeFlags: ['--headless'] });
  const results = [];

  for (const path of PAGES_TO_AUDIT) {
    const url = `${BASE_URL}${path}`;
    console.log(`Auditing ${url}...`);

    try {
      const lhr = await lighthouse(url, {
        port: chrome.port,
        output: 'json',
        logLevel: 'silent'
      }, {
        extends: 'lighthouse:default',
        settings: { onlyCategories: ['accessibility'] }
      });

      const score = lhr.lhr.categories.accessibility.score * 100;
      results.push({ url, score, status: 'ok' });
    } catch (err) {
      results.push({ url, score: null, status: 'error', error: err.message });
    }
  }

  await chrome.kill();

  // Report
  const sorted = results.sort((a, b) => (a.score || 0) - (b.score || 0));
  sorted.forEach(r => {
    const icon = r.score >= 90 ? '✓' : r.score >= 70 ? '⚠' : '✗';
    console.log(`${icon} ${r.score ?? 'ERR'}/100  ${r.url}`);
  });

  // Fail if any page scores below threshold
  const failing = results.filter(r => r.score !== null && r.score < 90);
  if (failing.length > 0) {
    console.error(`\n${failing.length} page(s) below threshold. Fix before deploying.`);
    process.exit(1);
  }
}

auditAll().catch(console.error);

What Automated Testing Won't Catch

Even with comprehensive axe and Lighthouse integration, you will miss:

  • Whether alt text is meaningful (axe only checks presence)
  • Whether reading order makes sense (structural validation only)
  • Whether keyboard interactions are logical for custom widgets
  • Whether focus management is correct after dynamic content changes
  • Whether error recovery paths are usable
  • Whether the experience is actually understandable to a screen reader user

Automated testing is a floor, not a ceiling. Treat 0 violations as a starting point for manual testing, not a sign-off.

The most effective accessibility programs use automated testing to catch regressions on known-good criteria, freeing human testers to focus on the nuanced judgments that tools cannot make.

Read more

Start now free