Testing Tools

PostHog Open-Source Product Analytics and A/B Testing: A QA Engineer's Guide

HelpMeTest

HelpMeTest

— 6 min read
PostHog Open-Source Product Analytics and A/B Testing: A QA Engineer's Guide

PostHog is unusual in the analytics tooling landscape: it's open-source, self-hostable, and combines session replay, product analytics, feature flags, A/B testing, and error tracking in a single platform. For QA engineers, this combination means behavioral analysis and experimentation infrastructure live in the same place.

This guide covers PostHog's architecture, its testing-relevant features, and how to integrate it into a QA workflow — whether you're using PostHog Cloud or self-hosting.

PostHog's Feature Set

PostHog's product covers ground that typically requires multiple separate tools:

Session replay: Video-like recordings of user sessions with DOM capture, similar to LogRocket or FullStory.

Product analytics: Event-based analytics with funnels, retention analysis, cohort tracking, and custom dashboards — functionally comparable to Mixpanel or Amplitude.

Feature flags: Boolean and multivariate flags that control feature rollout, usable from client-side SDKs or server-side integrations.

A/B testing and experiments: Hypothesis-driven feature experiments with statistical significance calculations built in.

Error tracking: JavaScript error capture and grouping, with session replay integration showing the session context around errors.

Heatmaps: Click and scroll heatmaps generated from captured interaction data.

Surveys: In-app surveys triggered by user behaviors.

Data pipelines: Export events to external systems (data warehouses, Slack, webhooks).

For a QA team, the most relevant combination is session replay, error tracking, and feature flags — with A/B testing becoming relevant when you're validating that experimental features work correctly for different user cohorts.

Self-Hosting vs. PostHog Cloud

PostHog is available as a managed cloud service or as a self-hosted deployment (Docker, Kubernetes via Helm).

Cloud: Easier setup, managed infrastructure, generous free tier (1M events/month free). Data leaves your infrastructure.

Self-hosted: Your data stays on your infrastructure, relevant for compliance requirements. Requires operational overhead — database management, scaling, updates.

The self-hosting option is what distinguishes PostHog from most competitors. For organizations with strict data residency requirements or that want to avoid per-event pricing at scale, self-hosting makes sense.

Session Replay

PostHog's session replay captures DOM mutations and user interactions, similar to other tools in this category. Notable features:

Error correlation: JavaScript errors in a session automatically appear in the session timeline, with the ability to filter recordings to sessions containing specific errors.

Network request capture: Recorded sessions include XHR/Fetch requests — important for debugging API-related issues. Network request capture requires opt-in configuration to avoid inadvertently capturing sensitive data.

Event correlation: PostHog events you track from your application appear in session replays, so you can see which analytics events correlate with which user interactions.

Masking: Input masking is configurable at the element level. PostHog supports ph-no-capture class for element-level suppression and global form masking.

Enabling session replay:

posthog.init('YOUR_API_KEY', {
    api_host: 'https://app.posthog.com',
    session_recording: {
        recordCrossOriginIframes: false,
    },
    capture_pageview: true,
});

Feature Flags

Feature flags are where PostHog's QA value extends beyond behavioral analysis. Flags let you:

Roll out gradually: Enable a feature for 5% of users, then 25%, then 100% — giving you staged exposure with the ability to roll back.

Target specific cohorts: Enable a feature for users in a specific plan tier, country, or with a specific property value.

Toggle in QA environments: Enable features for your QA team's user accounts regardless of production rollout status.

Kill switch: Disable a broken feature instantly without a deployment.

Using feature flags from JavaScript:

// Boolean flag
if (posthog.isFeatureEnabled('new-checkout-flow')) {
    // show new checkout
} else {
    // show old checkout
}

// Multivariate flag
const variant = posthog.getFeatureFlag('checkout-experiment');
if (variant === 'control') {
    // show control
} else if (variant === 'variant-a') {
    // show variant A
}

// Async check (waits for flag values to load)
posthog.onFeatureFlags(() => {
    if (posthog.isFeatureEnabled('new-feature')) {
        initializeNewFeature();
    }
});

From a QA perspective, feature flags enable you to test features in production before they're visible to all users. You can run your QA process against a production environment with the flag enabled for your test accounts — testing against real data, real infrastructure, real third-party integrations.

Server-side flag evaluation is also supported:

# Python
from posthog import Posthog
client = Posthog('YOUR_API_KEY')

if client.is_feature_enabled('new-feature', 'user-123'):
    # feature is enabled for this user

A/B Testing with Experiments

PostHog's experiment framework runs A/B tests against product metrics, with statistical significance calculated automatically. For QA engineers, experiments create a specific validation challenge: you need to verify that both control and treatment variants work correctly.

PostHog experiments define:

  • A control variant (existing behavior)
  • One or more test variants (new behavior)
  • A primary metric (goal event to optimize)
  • Secondary metrics (guardrail metrics to monitor for regression)
  • Minimum statistical significance threshold

The experiment assigns users to variants based on the feature flag system — so the same flag evaluation API applies.

QA checklist for experiments:

  1. Verify control variant renders correctly and matches existing behavior
  2. Verify each test variant renders correctly
  3. Verify variant assignment is consistent for a given user (a user who sees variant A on first visit should always see variant A)
  4. Verify that goal events are firing correctly for both variants
  5. Verify that guardrail metric events are firing correctly
  6. Test edge cases specific to the variant differences

PostHog provides a debugger overlay that shows which flags are active for your current session, making variant verification straightforward:

// Enable PostHog toolbar for visual debugging
posthog.loadToolbar({
    apiURL: 'https://app.posthog.com',
});

Error Tracking

PostHog's error tracking captures JavaScript exceptions and groups them by error signature. For each error group, you can:

  • See how many users are affected and the error frequency over time
  • Filter to sessions where the error occurred
  • View stack traces with source map support (if you upload source maps)
  • See the distribution of errors across browsers, countries, and device types

Integration is automatic with the JavaScript SDK — uncaught exceptions and promise rejections are captured by default. You can also capture errors explicitly:

try {
    riskyOperation();
} catch (error) {
    posthog.captureException(error, {
        extra_context: 'Payment processing',
        user_action: 'checkout_submit',
    });
}

For QA, error tracking provides a monitoring layer after automated tests pass. Your test suite verifies that known flows work. Error tracking alerts you to exceptions in production that your tests didn't anticipate.

Event Tracking for QA Validation

PostHog's event system lets you verify that specific application behaviors are occurring correctly in production:

// Track meaningful user actions
posthog.capture('checkout_completed', {
    order_value: 149.99,
    payment_method: 'credit_card',
    coupon_applied: false,
    item_count: 3,
});

// Track errors and edge cases
posthog.capture('form_validation_error', {
    form: 'checkout',
    field: 'expiry_date',
    error_type: 'format_invalid',
});

In PostHog's analytics interface, you can create funnel reports to verify that users who reach checkout are completing it at the expected rate, and that error events are occurring at acceptable frequencies.

This gives you a production monitoring layer: if a deployment causes checkout completion rates to drop by 5%, PostHog surfaces this before enough users report it for it to become a visible support issue.

PostHog for QA: A Practical Workflow

Setup:

  1. Instrument your application with PostHog's SDK
  2. Track key conversion events (signup, checkout, feature activation)
  3. Enable session replay
  4. Configure error tracking

For each release:

  1. Review error tracking dashboard for new error groups within hours of deployment
  2. Check funnel metrics for key flows to verify no regression
  3. Review session recordings for features that changed in the release
  4. If using feature flags: verify controlled rollout is working as intended

For experiments:

  1. Write test plans covering both control and treatment variants
  2. Use PostHog debugger to force-assign yourself to each variant
  3. Execute test plan against each variant
  4. Verify goal events fire correctly in PostHog's live event stream
  5. Monitor for the first 48 hours post-launch: error spikes, funnel drops in either variant

Self-Hosting Quick Start

For teams that need data sovereignty:

# docker-compose.yml (simplified)
version: '3'
services:
  db:
    image: postgres:14
    environment:
      POSTGRES_DB: posthog
      POSTGRES_PASSWORD: posthog
  
  redis:
    image: redis:7
  
  web:
    image: posthog/posthog:latest
    environment:
      SECRET_KEY: your-secret-key
      DATABASE_URL: postgres://postgres:posthog@db:5432/posthog
      REDIS_URL: redis://redis:6379/
    ports:
      - "8000:8000"
    depends_on:
      - db
      - redis

The full production deployment uses Helm on Kubernetes, with PostHog's official Helm chart. For serious deployments, the Helm chart handles Kafka (for high-volume event ingestion), ClickHouse (for analytics queries), and other components of the stack.

When PostHog Makes Sense

PostHog's all-in-one design is its key advantage. If you need session replay, analytics, feature flags, and A/B testing, implementing all of these in PostHog is simpler than managing four separate vendors with four separate SDKs and four separate billing relationships.

The open-source and self-hosting option is particularly valuable for:

  • GDPR/HIPAA contexts requiring data residency
  • Organizations that want to avoid per-event pricing at scale
  • Teams that prefer vendor-independent infrastructure

The tradeoff is operational complexity if you self-host, and less depth in any individual feature area compared to dedicated tools. LogRocket's session replay is more technically detailed; Amplitude's product analytics are more mature. But for most QA workflows, PostHog's depth is sufficient — and having everything in one place reduces context-switching cost significantly.

Read more

Start now free