Accessibility Testing

Automated Screen Reader Testing: NVDA, JAWS, and VoiceOver CI

HelpMeTest

HelpMeTest

— 4 min read
Automated Screen Reader Testing: NVDA, JAWS, and VoiceOver CI

Axe-core tells you whether your ARIA attributes are structurally correct. A screen reader tells you whether those attributes produce coherent output. These are different questions. A button can have aria-label="Submit form" and pass every automated check while a real screen reader announces it as "Submit form, collapsed, Submit form button" due to redundant text in its children.

Guidepup is a library that drives real screen readers — VoiceOver on macOS, NVDA on Windows — and captures their speech output programmatically. This post covers integrating it into a test suite and CI pipeline.

What Guidepup Actually Does

Guidepup uses OS-level automation APIs to control screen readers:

  • macOS: AppleScript / Accessibility API to drive VoiceOver
  • Windows: COM automation to drive NVDA (requires NVDA installed)

It captures spoken output as strings, letting you write assertions like:

expect(await voiceOver.lastSpokenPhrase()).toContain('Submit button');

This is fundamentally different from axe assertions. Axe checks DOM structure. Guidepup checks what a blind user actually hears.

Setup: VoiceOver on macOS

npm install --save-dev @guidepup/guidepup @guidepup/playwright

VoiceOver must be granted accessibility permissions. On macOS, this means the Terminal (or CI runner process) must be listed under System Settings → Privacy & Security → Accessibility. On GitHub-hosted runners, you'll need a self-hosted runner for VoiceOver testing.

Basic usage without Playwright:

import { voiceOver } from '@guidepup/guidepup';

async function testButton() {
  await voiceOver.start();

  try {
    // VoiceOver is now running — interact with the focused element
    await voiceOver.interact();

    // Press Tab to move focus
    await voiceOver.press('Tab');

    // Get what VoiceOver said about this element
    const spoken = await voiceOver.lastSpokenPhrase();
    console.log('VoiceOver said:', spoken);

  } finally {
    await voiceOver.stop();
  }
}

Playwright + Guidepup Integration

The @guidepup/playwright package provides a voiceOver fixture that integrates with Playwright's test runner:

import { test, expect } from '@guidepup/playwright';

test('navigation menu is correctly announced', async ({ page, voiceOver }) => {
  await page.goto('https://example.com');

  // Focus the navigation
  await page.locator('nav').focus();
  await voiceOver.interact();

  // Move to first nav item
  await voiceOver.press('Tab');
  const firstItemAnnouncement = await voiceOver.lastSpokenPhrase();

  // Should announce the link text and its role
  expect(firstItemAnnouncement).toMatch(/Home.*link|link.*Home/i);

  // Move to second item
  await voiceOver.press('Tab');
  const secondItemAnnouncement = await voiceOver.lastSpokenPhrase();
  expect(secondItemAnnouncement).toMatch(/About.*link|link.*About/i);
});

test('form errors are announced correctly', async ({ page, voiceOver }) => {
  await page.goto('https://example.com/contact');

  // Submit empty form
  await page.locator('[type="submit"]').click();

  // Wait for error to appear
  await page.waitForSelector('[role="alert"]', { timeout: 2000 });

  // Navigate to the error
  await voiceOver.press('Control+Option+Right'); // VoiceOver next item
  const errorAnnouncement = await voiceOver.lastSpokenPhrase();

  // Error must be announced — not just visible
  expect(errorAnnouncement).toMatch(/error|required|invalid/i);
});

Testing Modal Dialogs

Screen readers must hear a modal's role and label when it opens. Many implementations get the ARIA structure right but fail to move VoiceOver's cursor into the dialog, so users have to hunt for it.

test('modal announces role and title when opened', async ({ page, voiceOver }) => {
  await page.goto('https://example.com/dashboard');

  // Open the modal
  await page.locator('[data-testid="open-dialog"]').click();
  await page.locator('[role="dialog"]').waitFor();

  // Allow VoiceOver to process the dialog opening
  await new Promise(resolve => setTimeout(resolve, 500));

  // Check that VoiceOver announced the dialog
  const spokenPhrases = await voiceOver.spokenPhraseLog();
  const dialogAnnounced = spokenPhrases.some(phrase =>
    /dialog|modal/i.test(phrase) || phrase.includes('Settings')
  );

  expect(dialogAnnounced).toBe(true);

  // Navigate through dialog content
  await voiceOver.press('Tab');
  const firstFocusable = await voiceOver.lastSpokenPhrase();
  expect(firstFocusable).toBeTruthy(); // Something inside the dialog is announced
});

NVDA on Windows with GitHub Actions

NVDA testing requires a Windows runner. Here's a complete GitHub Actions job:

jobs:
  screen-reader-nvda:
    runs-on: windows-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install NVDA
        run: |
          $nvdaUrl = "https://www.nvaccess.org/files/nvda/releases/2024.1/nvda_2024.1.exe"
          Invoke-WebRequest -Uri $nvdaUrl -OutFile nvda-installer.exe
          # Silent install
          Start-Process -FilePath nvda-installer.exe -ArgumentList "--silent" -Wait
        shell: pwsh
      
      - run: npm ci
      
      - name: Install Playwright browsers
        run: npx playwright install chromium --with-deps
      
      - name: Run NVDA tests
        run: npx playwright test --config playwright.nvda.config.ts
        env:
          GUIDEPUP_SCREEN_READER: nvda
      
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: nvda-test-results
          path: test-results/
// playwright.nvda.config.ts
import { defineConfig } from '@playwright/test';
import { screenReaderConfig } from '@guidepup/playwright';

export default defineConfig({
  ...screenReaderConfig,
  testDir: './tests/screen-reader',
  testMatch: '**/*.nvda.spec.ts',
  workers: 1, // Screen reader tests must run serially
  timeout: 60000
});

VoiceOver Self-Hosted Runner Setup

For VoiceOver tests in CI, you need a macOS self-hosted runner with accessibility permissions granted. One-time setup on the runner machine:

# Grant Terminal accessibility permissions (requires manual UI step)
<span class="hljs-comment"># System Settings → Privacy & Security → Accessibility → add Terminal.app

<span class="hljs-comment"># Verify VoiceOver can start programmatically
osascript -e <span class="hljs-string">'tell application "VoiceOver" to start'
<span class="hljs-built_in">sleep 2
osascript -e <span class="hljs-string">'tell application "VoiceOver" to stop'

In your GitHub Actions workflow, add the runner label:

jobs:
  voiceover:
    runs-on: [self-hosted, macos, accessibility]
    
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install chromium
      - name: Run VoiceOver tests
        run: npx playwright test --config playwright.voiceover.config.ts

Writing Maintainable Speech Assertions

Speech output varies between screen reader versions and OS versions. Write assertions that tolerate variation:

// Fragile — breaks on minor SR version changes
expect(spoken).toBe('Submit form, button');

// Robust — checks for key information, ignores formatting
expect(spoken).toMatch(/submit/i);
expect(spoken).toMatch(/button/i);

// Even better — test that meaningful content is present
function assertButtonAnnounced(phrase: string, label: string) {
  expect(phrase.toLowerCase()).toContain(label.toLowerCase());
  expect(phrase.toLowerCase()).toContain('button');
}

assertButtonAnnounced(await voiceOver.lastSpokenPhrase(), 'submit form');

Deciding What to Test with Screen Readers

Screen reader tests are slow (30–60 seconds per test), require special infrastructure, and are fragile compared to axe tests. Use them selectively:

Test these with screen readers:

  • Navigation patterns (menus, breadcrumbs, tab lists)
  • Form validation announcements
  • Dynamic content updates (live regions, toasts, loading states)
  • Modal and dialog interactions
  • Custom widgets (date pickers, autocomplete, sliders)

Don't test these with screen readers:

  • Static text content (axe + manual review is sufficient)
  • Color contrast (axe handles this)
  • Every page in your app (use axe for broad coverage, screen readers for critical flows)

The goal is confidence that your key user journeys work with real assistive technology. A login flow, a checkout flow, a settings form — these are candidates for screen reader tests. A marketing page with no interactive elements is not.

Read more