PWA Testing

PWA Performance Testing & Lighthouse Auditing: The Complete Guide

HelpMeTest

HelpMeTest

— 7 min read
PWA Performance Testing & Lighthouse Auditing: The Complete Guide

A PWA that loads slowly or scores poorly on Core Web Vitals fails to deliver on the promise of an app-like experience. Lighthouse is the standard tool for PWA auditing, but running it manually isn't enough — you need automated performance testing in CI, regression detection, and monitoring in production. This guide covers everything from setting up Lighthouse CI to writing custom performance assertions.

Understanding Lighthouse PWA Scoring

Lighthouse evaluates PWAs across five categories, each with its own weight:

Category Weight What It Measures
Performance 1.0 Core Web Vitals, page speed
Accessibility 1.0 ARIA, color contrast, keyboard nav
Best Practices 1.0 HTTPS, console errors, security
SEO 1.0 Meta tags, crawlability
PWA N/A Installability, offline capability

The PWA category doesn't contribute to an overall score — it's pass/fail based on specific criteria. But the Performance score is where most PWAs struggle.

Core Web Vitals for PWAs

Google's Core Web Vitals are especially important for PWAs because they're used in search ranking:

  • LCP (Largest Contentful Paint) — should be ≤ 2.5s. For PWAs with heavy service worker logic, this can be impacted by SW initialization.
  • INP (Interaction to Next Paint) — should be ≤ 200ms. JavaScript-heavy SPAs often struggle here.
  • CLS (Cumulative Layout Shift) — should be ≤ 0.1. Dynamic content loading without reserved space is the primary cause.

Setting Up Lighthouse CI

Lighthouse CI integrates with your CI pipeline for automated performance regression detection:

npm install -g @lhci/cli
# lighthouserc.json
{
  "ci": {
    "collect": {
      "url": ["http://localhost:3000", "http://localhost:3000/dashboard"],
      "numberOfRuns": 3,
      "settings": {
        "preset": "desktop"
      }
    },
    "assert": {
      "preset": "lighthouse:recommended",
      "assertions": {
        "categories:performance": ["error", {"minScore": 0.9}],
        "categories:accessibility": ["error", {"minScore": 0.9}],
        "categories:pwa": ["error", {"minScore": 1}],
        "first-contentful-paint": ["error", {"maxNumericValue": 2000}],
        "largest-contentful-paint": ["error", {"maxNumericValue": 2500}],
        "total-blocking-time": ["error", {"maxNumericValue": 200}],
        "cumulative-layout-shift": ["error", {"maxNumericValue": 0.1}],
        "interactive": ["error", {"maxNumericValue": 3500}]
      }
    },
    "upload": {
      "target": "temporary-public-storage"
    }
  }
}
# .github/workflows/lighthouse.yml
name: Lighthouse CI

on: [push, pull_request]

jobs:
  lighthouse:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Build app
        run: npm run build
      
      - name: Start server
        run: npm run serve &
      
      - name: Wait for server
        run: npx wait-on http://localhost:3000
      
      - name: Run Lighthouse CI
        run: lhci autorun
        env:
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}

Writing Custom Performance Tests with Playwright

Lighthouse is great for static analysis, but real-user scenarios need browser automation:

import { test, expect } from '@playwright/test';

test.describe('PWA Performance', () => {
  test('page loads within performance budget', async ({ page }) => {
    const metrics = {
      navigationStart: 0,
      firstContentfulPaint: 0,
      largestContentfulPaint: 0,
      domInteractive: 0,
    };
    
    // Capture paint timing
    await page.addInitScript(() => {
      const observer = new PerformanceObserver((list) => {
        for (const entry of list.getEntries()) {
          window.__performanceEntries = window.__performanceEntries || [];
          window.__performanceEntries.push({
            name: entry.name,
            startTime: entry.startTime
          });
        }
      });
      observer.observe({ type: 'paint', buffered: true });
      observer.observe({ type: 'largest-contentful-paint', buffered: true });
    });
    
    await page.goto('http://localhost:3000');
    await page.waitForLoadState('networkidle');
    
    const entries = await page.evaluate(() => window.__performanceEntries || []);
    
    const fcp = entries.find(e => e.name === 'first-contentful-paint');
    const lcp = entries.find(e => e.name === 'largest-contentful-paint');
    
    if (fcp) expect(fcp.startTime).toBeLessThan(2000);
    if (lcp) expect(lcp.startTime).toBeLessThan(2500);
  });
  
  test('no layout shift during content load', async ({ page }) => {
    let totalCLS = 0;
    
    await page.addInitScript(() => {
      const observer = new PerformanceObserver((list) => {
        for (const entry of list.getEntries()) {
          if (!entry.hadRecentInput) {
            window.__cls = (window.__cls || 0) + entry.value;
          }
        }
      });
      observer.observe({ type: 'layout-shift', buffered: true });
    });
    
    await page.goto('http://localhost:3000');
    await page.waitForLoadState('networkidle');
    await page.waitForTimeout(1000); // Allow time for dynamic content
    
    totalCLS = await page.evaluate(() => window.__cls || 0);
    
    expect(totalCLS).toBeLessThan(0.1);
  });
  
  test('service worker does not delay first paint', async ({ page }) => {
    // Measure without SW (first visit)
    const startTime = Date.now();
    await page.goto('http://localhost:3000');
    
    const fcp = await page.evaluate(() => {
      const [entry] = performance.getEntriesByName('first-contentful-paint');
      return entry?.startTime;
    });
    
    // FCP should not be delayed by SW registration
    if (fcp) expect(fcp).toBeLessThan(3000);
  });
  
  test('subsequent loads use service worker cache for speed', async ({ page, context }) => {
    // First visit
    await page.goto('http://localhost:3000');
    await page.waitForLoadState('networkidle');
    await page.waitForTimeout(500); // Let SW cache
    
    // Second visit (from cache)
    const start = Date.now();
    await page.goto('http://localhost:3000');
    await page.waitForLoadState('networkidle');
    const cachedLoadTime = Date.now() - start;
    
    // Cached load should be fast
    expect(cachedLoadTime).toBeLessThan(1000);
  });
});

Testing Bundle Size and JavaScript Performance

Large JavaScript bundles are a primary cause of slow PWA performance:

// tests/bundle-size.test.js
import fs from 'fs';
import path from 'path';
import { glob } from 'glob';

describe('Bundle Size Budget', () => {
  const BUILD_DIR = 'dist';
  
  it('main JavaScript bundle is under 200KB (gzipped equivalent)', () => {
    const jsFiles = glob.sync(`${BUILD_DIR}/**/*.js`);
    
    const mainBundle = jsFiles.find(f => f.includes('main') || f.includes('index'));
    
    if (mainBundle) {
      const stats = fs.statSync(mainBundle);
      const sizeKB = stats.size / 1024;
      
      // Raw JS — gzipped will be ~30% of this
      expect(sizeKB).toBeLessThan(600); // ~200KB gzipped
    }
  });
  
  it('total CSS is under 50KB', () => {
    const cssFiles = glob.sync(`${BUILD_DIR}/**/*.css`);
    
    const totalSize = cssFiles.reduce((sum, file) => {
      return sum + fs.statSync(file).size;
    }, 0);
    
    expect(totalSize / 1024).toBeLessThan(150); // ~50KB gzipped
  });
  
  it('images are optimized', () => {
    const imageFiles = glob.sync(`${BUILD_DIR}/**/*.{png,jpg,jpeg}`);
    
    for (const imagePath of imageFiles) {
      const stats = fs.statSync(imagePath);
      const sizeMB = stats.size / (1024 * 1024);
      
      // No image should be over 300KB
      expect(sizeMB).toBeLessThan(0.3);
    }
  });
});

Testing Service Worker Performance Impact

Service workers add latency on first load but improve subsequent loads. Test both:

test('service worker activation does not block first contentful paint', async ({ browser }) => {
  const context = await browser.newContext();
  const page = await context.newPage();
  
  // Capture navigation timing
  const timings = [];
  page.on('response', response => {
    if (response.url().includes('localhost:3000')) {
      timings.push({
        url: response.url(),
        timing: response.timing()
      });
    }
  });
  
  await page.goto('http://localhost:3000');
  
  const swRegistration = await page.evaluate(() => {
    return performance.getEntriesByType('resource')
      .find(e => e.name.includes('sw.js'));
  });
  
  // SW script should load but not block the critical path
  if (swRegistration) {
    // SW fetch should not increase total load time beyond acceptable threshold
    expect(swRegistration.duration).toBeLessThan(200);
  }
});

test('repeat visits load significantly faster with service worker', async ({ browser }) => {
  const context = await browser.newContext();
  const page = await context.newPage();
  
  // Cold load
  const coldStart = Date.now();
  await page.goto('http://localhost:3000');
  await page.waitForLoadState('networkidle');
  const coldTime = Date.now() - coldStart;
  
  // Let SW install
  await page.waitForTimeout(1000);
  
  // Warm load
  const warmStart = Date.now();
  await page.goto('http://localhost:3000');
  await page.waitForLoadState('networkidle');
  const warmTime = Date.now() - warmStart;
  
  // Warm load should be at least 30% faster
  expect(warmTime).toBeLessThan(coldTime * 0.7);
});

Testing App Shell Architecture

PWAs often use an "app shell" pattern — caching the minimal UI shell separately from content. Test that it works:

test('app shell loads instantly from cache', async ({ page, context }) => {
  // Prime the cache
  await page.goto('http://localhost:3000');
  await page.waitForLoadState('networkidle');
  
  // Block all network requests (simulate offline/slow network)
  await context.route('**/*', route => {
    if (route.request().url().includes('/api/')) {
      // Allow API calls but delay them
      setTimeout(() => route.continue(), 2000);
    } else {
      route.continue();
    }
  });
  
  const start = Date.now();
  await page.goto('http://localhost:3000');
  
  // App shell (nav, header) should appear immediately
  await expect(page.locator('nav')).toBeVisible({ timeout: 500 });
  
  const shellLoadTime = Date.now() - start;
  expect(shellLoadTime).toBeLessThan(500);
});

Performance Budget Testing in CI

Define and enforce performance budgets:

// tests/perf-budget.test.js
import lighthouse from 'lighthouse';
import chromeLauncher from 'chrome-launcher';

const PERFORMANCE_BUDGET = {
  'performance': 0.9,
  'accessibility': 0.9,
  'pwa': 1.0,
  'first-contentful-paint': 2000,
  'largest-contentful-paint': 2500,
  'total-blocking-time': 200,
  'cumulative-layout-shift': 0.1,
  'speed-index': 3000,
};

describe('Performance Budget', () => {
  let chrome;
  let lhr;
  
  beforeAll(async () => {
    chrome = await chromeLauncher.launch({ chromeFlags: ['--headless'] });
    const result = await lighthouse('http://localhost:3000', {
      port: chrome.port,
      onlyCategories: ['performance', 'accessibility', 'pwa'],
    });
    lhr = result.lhr;
  }, 60000);
  
  afterAll(async () => {
    await chrome.kill();
  });
  
  it('meets performance score budget', () => {
    expect(lhr.categories.performance.score)
      .toBeGreaterThanOrEqual(PERFORMANCE_BUDGET['performance']);
  });
  
  it('meets accessibility score budget', () => {
    expect(lhr.categories.accessibility.score)
      .toBeGreaterThanOrEqual(PERFORMANCE_BUDGET['accessibility']);
  });
  
  it('passes all PWA checks', () => {
    expect(lhr.categories.pwa.score)
      .toBeGreaterThanOrEqual(PERFORMANCE_BUDGET['pwa']);
  });
  
  it('LCP within budget', () => {
    const lcp = lhr.audits['largest-contentful-paint'].numericValue;
    expect(lcp).toBeLessThanOrEqual(PERFORMANCE_BUDGET['largest-contentful-paint']);
  });
  
  it('TBT within budget', () => {
    const tbt = lhr.audits['total-blocking-time'].numericValue;
    expect(tbt).toBeLessThanOrEqual(PERFORMANCE_BUDGET['total-blocking-time']);
  });
  
  it('CLS within budget', () => {
    const cls = lhr.audits['cumulative-layout-shift'].numericValue;
    expect(cls).toBeLessThanOrEqual(PERFORMANCE_BUDGET['cumulative-layout-shift']);
  });
});

Runtime Performance Testing

Beyond load time, test that your PWA performs well under interaction:

test('scrolling does not cause janky experience', async ({ page }) => {
  await page.goto('http://localhost:3000/long-list');
  await page.waitForLoadState('networkidle');
  
  // Measure frame rate during scroll
  const frameData = await page.evaluate(async () => {
    const frames = [];
    
    const observer = new PerformanceObserver((list) => {
      frames.push(...list.getEntries().map(e => e.duration));
    });
    observer.observe({ type: 'frame', buffered: false });
    
    // Scroll down
    window.scrollTo({ top: 5000, behavior: 'smooth' });
    await new Promise(resolve => setTimeout(resolve, 1000));
    
    return frames;
  });
  
  // Check that we don't have too many frames > 16ms (60fps = 16ms/frame)
  const slowFrames = frameData.filter(duration => duration > 16);
  const slowFrameRatio = slowFrames.length / frameData.length;
  
  expect(slowFrameRatio).toBeLessThan(0.1); // Less than 10% slow frames
});

Continuous Performance Monitoring with HelpMeTest

Performance doesn't just matter at release time — it can degrade with every deployment. HelpMeTest provides continuous monitoring:

*** Test Cases ***
PWA Performance Health Check
    [Documentation]    Monitor Core Web Vitals in production
    Run Lighthouse Audit
    ...    https://your-pwa.com
    ...    minPerformance=90
    ...    maxLCP=2500
    ...    maxCLS=0.1
    ...    maxTBT=200
    Log Lighthouse Score    ${score}
    Should Pass PWA Requirements    ${score}

Schedule this to run after every deployment and daily in production. Performance regressions are often introduced incrementally — a slightly larger bundle here, a missing cache header there — and only surface when they've compounded enough to fail the Lighthouse budget.

Reading Lighthouse Reports Programmatically

When Lighthouse fails, you need to know exactly what changed:

async function compareLighthouseScores(baseUrl, compareUrl) {
  const [baseReport, compareReport] = await Promise.all([
    runLighthouse(baseUrl),
    runLighthouse(compareUrl)
  ]);
  
  const metrics = [
    'first-contentful-paint',
    'largest-contentful-paint',
    'total-blocking-time',
    'cumulative-layout-shift',
    'speed-index',
    'interactive'
  ];
  
  return metrics.map(metric => ({
    metric,
    base: baseReport.audits[metric]?.numericValue,
    compare: compareReport.audits[metric]?.numericValue,
    delta: compareReport.audits[metric]?.numericValue - baseReport.audits[metric]?.numericValue,
    regression: compareReport.audits[metric]?.numericValue > baseReport.audits[metric]?.numericValue * 1.1
  }));
}

Conclusion

PWA performance testing has multiple layers: automated Lighthouse CI to catch regressions, custom Playwright tests for real-world scenarios, bundle size budgets to prevent gradual bloat, and production monitoring to catch what CI misses.

The key insight: performance isn't a one-time achievement. Every feature addition, dependency update, and configuration change is a potential regression. Automated performance testing in CI — with hard failure thresholds — is the only reliable way to keep a PWA fast over time.

Set your budgets conservatively, fail the build when you miss them, and treat a performance regression as seriously as a functional bug. Your users' time is worth it.

Read more

Start now free