Testing

GrowthBook Tutorial: Open-Source A/B Testing with Jest and Playwright

HelpMeTest

HelpMeTest

— 5 min read
GrowthBook Tutorial: Open-Source A/B Testing with Jest and Playwright

GrowthBook is an open-source A/B testing and feature flag platform with a JavaScript SDK that works in both Node.js and browsers. This tutorial covers SDK setup, writing Jest unit tests for flag evaluation, Playwright tests for UI experiment variants, and setting up GrowthBook in CI without a live backend.

What GrowthBook Provides

GrowthBook is a self-hostable alternative to LaunchDarkly and Optimizely. Its key differentiator: feature definitions and experiment configurations live in JSON files (or are fetched from an API), and the SDK evaluates them entirely client-side without network round-trips for every flag check.

This architecture makes GrowthBook very testable—you can inject feature definitions directly into tests without running a GrowthBook instance.

Installing the SDK

npm install @growthbook/growthbook

Basic SDK Usage

// src/feature-flags.js
const { GrowthBook } = require('@growthbook/growthbook');

let gbInstance = null;

async function initGrowthBook(userAttributes = {}) {
  const gb = new GrowthBook({
    apiHost: process.env.GROWTHBOOK_API_HOST,
    clientKey: process.env.GROWTHBOOK_CLIENT_KEY,
    attributes: userAttributes,
    trackingCallback: (experiment, result) => {
      // Send experiment impressions to your analytics
      analytics.track('Experiment Viewed', {
        experimentId: experiment.key,
        variationId: result.variationId,
      });
    },
  });

  await gb.loadFeatures();
  gbInstance = gb;
  return gb;
}

function isFeatureEnabled(featureKey) {
  if (!gbInstance) throw new Error('GrowthBook not initialized');
  return gbInstance.isOn(featureKey);
}

function getFeatureValue(featureKey, defaultValue) {
  if (!gbInstance) throw new Error('GrowthBook not initialized');
  return gbInstance.getFeatureValue(featureKey, defaultValue);
}

module.exports = { initGrowthBook, isFeatureEnabled, getFeatureValue };

Unit Testing with Jest — No Backend Required

The GrowthBook SDK accepts feature definitions inline, so tests don't need a live GrowthBook instance. Inject the features directly:

// test/feature-flags.test.js
const { GrowthBook } = require('@growthbook/growthbook');

function createGrowthBook(attributes, features) {
  const gb = new GrowthBook({ attributes });
  gb.setFeatures(features);
  return gb;
}

describe('feature flag evaluation', () => {
  it('returns true for enabled boolean flag', () => {
    const gb = createGrowthBook(
      { userId: 'user_123', plan: 'pro' },
      {
        new_dashboard: { defaultValue: false, rules: [{ force: true }] },
      }
    );

    expect(gb.isOn('new_dashboard')).toBe(true);
  });

  it('returns false for disabled flag', () => {
    const gb = createGrowthBook(
      { userId: 'user_123' },
      {
        experimental_editor: { defaultValue: false },
      }
    );

    expect(gb.isOn('experimental_editor')).toBe(false);
  });

  it('returns default value for missing feature', () => {
    const gb = createGrowthBook({ userId: 'user_123' }, {});
    expect(gb.getFeatureValue('nonexistent_feature', 'fallback')).toBe('fallback');
  });

  it('evaluates targeting rules by attribute', () => {
    const features = {
      beta_ui: {
        defaultValue: false,
        rules: [
          {
            condition: { plan: { $eq: 'enterprise' } },
            force: true,
          },
        ],
      },
    };

    const proUser = createGrowthBook({ userId: 'u1', plan: 'pro' }, features);
    const enterpriseUser = createGrowthBook({ userId: 'u2', plan: 'enterprise' }, features);

    expect(proUser.isOn('beta_ui')).toBe(false);
    expect(enterpriseUser.isOn('beta_ui')).toBe(true);
  });
});

Testing A/B Experiment Assignment

GrowthBook uses a deterministic hash for experiment assignment. Test that your experiment setup assigns variants consistently:

// test/experiments.test.js
const { GrowthBook } = require('@growthbook/growthbook');

const checkoutExperiment = {
  key: 'checkout_button_color',
  variations: ['control', 'blue', 'green'],
  weights: [0.34, 0.33, 0.33],
  coverage: 1,
};

function runExperiment(userId) {
  const gb = new GrowthBook({
    attributes: { id: userId },
    features: {},
  });

  // Set a feature that maps to the experiment
  gb.setFeatures({
    checkout_cta: {
      defaultValue: 'control',
      rules: [
        {
          key: 'checkout_button_color',
          variations: ['control', 'blue', 'green'],
          weights: [0.34, 0.33, 0.33],
          coverage: 1,
        },
      ],
    },
  });

  return gb.getFeatureValue('checkout_cta', 'control');
}

describe('checkout A/B experiment', () => {
  it('assigns same user to same variant consistently', () => {
    const userId = 'stable_user_abc';
    const first = runExperiment(userId);
    const second = runExperiment(userId);
    expect(first).toBe(second);
  });

  it('distributes variants across users', () => {
    const variants = new Set();
    for (let i = 0; i < 200; i++) {
      variants.add(runExperiment(`user_${i}`));
    }
    // With 200 users and 3 variants at ~33% each, all should appear
    expect(variants.size).toBeGreaterThanOrEqual(2);
  });

  it('respects coverage — users outside coverage get default', () => {
    // At coverage=0, no user should be in the experiment
    const gb = new GrowthBook({ attributes: { id: 'any_user' } });
    gb.setFeatures({
      checkout_cta: {
        defaultValue: 'control',
        rules: [
          {
            key: 'checkout_button_color',
            variations: ['control', 'blue', 'green'],
            coverage: 0, // 0% coverage
          },
        ],
      },
    });

    expect(gb.getFeatureValue('checkout_cta', 'control')).toBe('control');
  });
});

Testing the Tracking Callback

Impressions (experiment exposures) must be tracked accurately for results to be valid. Test that the callback fires correctly:

// test/tracking.test.js
const { GrowthBook } = require('@growthbook/growthbook');

it('fires tracking callback when experiment is evaluated', () => {
  const trackingCallback = jest.fn();

  const gb = new GrowthBook({
    attributes: { id: 'tracked_user' },
    trackingCallback,
  });

  gb.setFeatures({
    checkout_cta: {
      defaultValue: 'control',
      rules: [
        {
          key: 'checkout_cta_experiment',
          variations: ['control', 'treatment'],
          weights: [0.5, 0.5],
          coverage: 1,
        },
      ],
    },
  });

  // First evaluation triggers the callback
  gb.getFeatureValue('checkout_cta', 'control');

  expect(trackingCallback).toHaveBeenCalledTimes(1);
  const [experiment, result] = trackingCallback.mock.calls[0];
  expect(experiment.key).toBe('checkout_cta_experiment');
  expect(['control', 'treatment']).toContain(result.value);
});

it('does not fire tracking callback for feature flags without experiments', () => {
  const trackingCallback = jest.fn();

  const gb = new GrowthBook({
    attributes: { id: 'user_1' },
    trackingCallback,
  });

  gb.setFeatures({
    simple_flag: { defaultValue: true, rules: [{ force: true }] },
  });

  gb.isOn('simple_flag');

  // Force rules don't count as experiments
  expect(trackingCallback).not.toHaveBeenCalled();
});

Playwright Tests for Experiment Variants

For UI experiments, use Playwright to verify each variant renders correctly. Inject the feature state via the GrowthBook SDK before the test page loads:

// playwright.config.js
const { defineConfig } = require('@playwright/test');

module.exports = defineConfig({
  use: {
    baseURL: 'http://localhost:3000',
  },
});
// tests/checkout-experiment.spec.js
const { test, expect } = require('@playwright/test');

// Override the GrowthBook features via localStorage or a test cookie
async function setFeature(page, featureKey, value) {
  await page.addInitScript(({ key, val }) => {
    window.__TEST_FEATURES__ = window.__TEST_FEATURES__ || {};
    window.__TEST_FEATURES__[key] = val;
  }, { key: featureKey, val: value });
}

test('control variant shows original CTA button', async ({ page }) => {
  await setFeature(page, 'checkout_cta', 'control');
  await page.goto('/checkout');

  const button = page.locator('[data-testid="checkout-btn"]');
  await expect(button).toBeVisible();
  await expect(button).toHaveCSS('background-color', 'rgb(0, 0, 0)'); // black
  await expect(button).toContainText('Checkout');
});

test('treatment variant shows blue CTA button', async ({ page }) => {
  await setFeature(page, 'checkout_cta', 'blue');
  await page.goto('/checkout');

  const button = page.locator('[data-testid="checkout-btn"]');
  await expect(button).toBeVisible();
  await expect(button).toHaveCSS('background-color', 'rgb(37, 99, 235)'); // blue
});

test('both variants lead to order confirmation', async ({ page }) => {
  for (const variant of ['control', 'blue', 'green']) {
    await setFeature(page, 'checkout_cta', variant);
    await page.goto('/checkout');

    await page.fill('[name="card_number"]', '4242424242424242');
    await page.fill('[name="expiry"]', '12/28');
    await page.fill('[name="cvc"]', '123');
    await page.click('[data-testid="checkout-btn"]');

    await expect(page).toHaveURL(/\/order-confirmation/);
    await expect(page.locator('[data-testid="order-id"]')).toBeVisible();
    
    // Go back for next iteration
    await page.goto('/checkout');
  }
});

The addInitScript approach injects test overrides before any JavaScript runs. Your application reads window.__TEST_FEATURES__ in the GrowthBook initialization and uses it to override live feature values.

CI Without a Live GrowthBook Instance

For CI, use GrowthBook's JSON payload directly (no API call):

// src/feature-flags.js — CI-aware initialization
async function initGrowthBook(userAttributes = {}) {
  const gb = new GrowthBook({ attributes: userAttributes });

  if (process.env.GROWTHBOOK_FEATURES_JSON) {
    // In CI, load features from the environment variable (JSON string)
    gb.setFeatures(JSON.parse(process.env.GROWTHBOOK_FEATURES_JSON));
  } else {
    await gb.loadFeatures({
      apiHost: process.env.GROWTHBOOK_API_HOST,
      clientKey: process.env.GROWTHBOOK_CLIENT_KEY,
    });
  }

  return gb;
}
# .github/workflows/test.yml
env:
  GROWTHBOOK_FEATURES_JSON: >
    {
      "checkout_cta": {"defaultValue": "control"},
      "new_dashboard": {"defaultValue": false}
    }

Production Monitoring With HelpMeTest

Testing that experiments work locally is only part of the picture. For continuous monitoring of experiment variants in production—detecting if a rollout accidentally excludes users, or if a variant change breaks a downstream conversion step—HelpMeTest runs browser-based monitoring tests on a schedule that can target specific experiment variants and assert on the full user journey.

Summary

  • GrowthBook SDK evaluates features client-side from JSON definitions—no backend needed in tests
  • Inject features with gb.setFeatures() in unit tests for fast, isolated flag evaluation
  • Test tracking callbacks to verify experiment impressions are recorded correctly
  • Use Playwright's addInitScript to inject feature overrides before page load for UI variant tests
  • In CI, pass a JSON blob via GROWTHBOOK_FEATURES_JSON instead of hitting the live API
  • Test deterministic assignment, coverage behavior, and targeting rule evaluation in Jest

Read more

Testing Atlantis Terraform PR Automation: Workflows, Plan Verification, and Policy Enforcement

Testing Atlantis Terraform PR Automation: Workflows, Plan Verification, and Policy Enforcement

Atlantis automates Terraform plan and apply through pull requests. But Atlantis itself needs testing: workflow configuration, plan output validation, policy enforcement, and server health checks. This guide covers testing Atlantis workflows locally with atlantis-local, validating plan outputs with custom scripts, enforcing Terraform policies with OPA and Conftest, and monitoring Atlantis

By HelpMeTest