A/B Test Experiment Design: How to Set Up Tests That Produce Reliable Results

The failure mode for most A/B tests isn't statistical — it's design. Teams write bad hypotheses, measure the wrong metrics, run tests too short, or contaminate the test population. By the time they realize the error, weeks of data are worthless.

Good A/B test design happens before the first user is enrolled. This guide covers the decisions that determine whether your experiment produces signal or noise.

Step 1: Write a Hypothesis Worth Testing

A testable A/B hypothesis has three parts: the change you're making, the expected effect, and the mechanism.

Bad hypothesis: "Making the CTA button green will increase conversions."

Good hypothesis: "Changing the CTA button from gray to green will increase checkout page conversion rate by at least 5%, because green signals action and our user research shows current users perceive the gray button as inactive."

The mechanism matters because it determines whether you should run the test at all. If you can't explain why a change should improve a metric, you're fishing. Fishing experiments are expensive and rarely replicate.

The minimum detectable effect (5% in the example) matters because it determines sample size. If you wouldn't ship a change that lifts conversion by less than 5%, don't test for effects smaller than 5%.

Step 2: Choose a Primary Metric (and Stick to It)

Specify your primary metric before you start the test. This is the single number that determines whether the experiment succeeds or fails.

Primary metric criteria:

• Directly represents user value or business outcome
• Moves in a reasonable timeframe (within your test window)
• Can be measured unambiguously
• Is not easily gamed

Common primary metrics:

• Conversion rate (% users who complete a target action)
• Revenue per user
• Retention rate at N days
• Task completion rate
• Error rate

Pick one. Analyzing 10 metrics and finding one significant is a multiple comparisons error — statistically guaranteed to produce false positives.

You can monitor secondary metrics (guardrail metrics) to ensure your primary metric improvement doesn't come at the cost of something else. But they don't determine success/failure — they determine whether a winning experiment is safe to ship.

Step 3: Define Guardrail Metrics

Guardrail metrics protect against optimization that's harmful overall:

If your primary metric improves but a guardrail metric degrades significantly, don't ship. You've optimized locally at the cost of the broader experience.

Step 4: Define the Unit of Randomization

The unit of randomization should match the unit of analysis. This sounds obvious but is violated constantly.

Correct: Randomize by user ID, analyze per-user conversion rate.

Incorrect: Randomize by session, analyze per-session conversion rate, then conclude about users.

The problem with session randomization: the same user can be in both variants across sessions. Their behavior in one session is influenced by their experience in the previous session (variant contamination). You don't have independent observations.

Always randomize at the level of the experimental unit that will experience the change continuously:

• User-level changes (UI, algorithm) → randomize by user ID
• Request-level changes (latency, API response) → can randomize by request
• Content changes (email subject line) → randomize by email recipient
• Geographic tests → randomize by geography (and analyze accordingly)

Step 5: Define the Enrollment Criteria

Who gets enrolled in your test? Be explicit before launching:

# Experiment spec
name: checkout-cta-color
enrolled_when:
  - user visits /checkout
  - user is logged in
  - user has items in cart
  
excluded:
  - users who have already placed an order this session
  - users in other active checkout experiments
  - internal company accounts
  - users assigned to the experiment before their first checkout visit

"Enrolled when" matters because users should be enrolled on their first exposure to the experiment, not retroactively. Post-enrollment exclusions contaminate your sample.

Novelty effect consideration: When testing new UI patterns, early users may behave differently than later users simply because of novelty. Consider excluding users enrolled in the first 48 hours from analysis, or extending test duration to let novelty wear off.

Step 6: Calculate Test Duration (Not Just Sample Size)

Sample size calculators give you a number of users. Test duration requires additional context:

Duration = Required sample size per variant × 2 / Daily new user enrollments

But this minimum should be adjusted upward for:

• Day-of-week effects: Run at least 1 full week (7 days) to capture weekly seasonality
• Day-of-month effects: Some metrics (billing, subscription renewals) cycle monthly — run for a full month
• Marketing calendar: Don't start a test the week of a major campaign or holiday
• Novelty decay: Add 2-3 days to let initial novelty behavior normalize

A test with a 3-day statistical minimum should run for at least 7 days. A test with a 10-day minimum should run for at least 14.

Step 7: Pre-Register Your Analysis Plan

Before enrolling users, document in writing:

1. Primary metric and success threshold
2. Secondary/guardrail metrics
3. Planned end date (or end sample size)
4. Which user segments will be analyzed (if any)
5. Statistical significance threshold (usually p < 0.05)
6. What "ship it" looks like: e.g., "primary metric improves by ≥ 3%, p < 0.05, no guardrail metric degrades by > 5%"

Pre-registration prevents post-hoc rationalization. When you write down "we'll analyze mobile vs desktop" before the test, you've committed to that analysis. When you do it after seeing the data, you're data dredging.

Common Design Errors

The Interaction Effect

Running multiple A/B tests on the same population simultaneously creates interaction effects. User X might be in test 1 variant B AND test 2 variant A. If both tests modify the checkout flow, you can't cleanly attribute changes to either.

Mitigation: maintain an experiment exclusivity layer. Experiments that modify the same user journey run sequentially, not simultaneously.

Sample Ratio Mismatch

You set up a 50/50 split but your actual split is 48/52. This indicates something is wrong with your randomization or enrollment tracking — possibly users switching variants, enrollment bugs, or logging gaps.

Check for sample ratio mismatch before analyzing results:

from scipy.stats import chi2_contingency

observed_control = 4823
observed_treatment = 5177
expected_each = (observed_control + observed_treatment) / 2

chi2, p_value, _, _ = chi2_contingency([
    [observed_control, expected_each - observed_control],
    [observed_treatment, expected_each - observed_treatment]
])

if p_value < 0.01:
    print(f"WARNING: Sample ratio mismatch detected (p={p_value:.4f})")
    print("Investigate before analyzing results")

If you have significant sample ratio mismatch, your experiment data is unreliable. Fix the randomization and re-run.

Survivorship Bias in Enrollment

If you enroll users based on completing an action, you're selecting for engaged users. Your test results won't generalize to new or less-engaged users.

Example: enrolling users in an A/B test only after they reach the checkout page selects for users who were already close to converting. The effect size you measure won't apply to users who abandon earlier.

Test at the earliest meaningful exposure point, not the last.

Carryover Effects

In sequential tests (running multiple experiments over time), users carry behavioral patterns from previous experiments. A user who converted in a winning experiment in Q1 will behave differently in Q2 than a user who was in the control group.

Mitigation: introduce washout periods between sequential experiments on the same user population, or use holdout groups (users permanently excluded from all experiments) as a long-term baseline.

Instrumentation Checklist

Before launching:

• Assignment logic is deterministic (same user always gets same variant)
• Exposure event fires at first user interaction with the variant (not on initialization)
• Conversion events include experiment ID and variant
• No logging gaps in mobile/native clients (especially offline scenarios)
• Test helper accounts are excluded from experiment enrollment
• Bot traffic is filtered from analysis
• Health dashboard shows ~50/50 split within 24 hours of launch

Stopping an Experiment Early

Sometimes you should stop an experiment before the planned end date:

Stop for harm: A guardrail metric drops significantly (p < 0.01, magnitude > 10%). Stop immediately, investigate.

Stop for futility: The observed effect is so small that even reaching full sample size won't achieve practical significance. Pre-define: "if after 50% of sample size collected the effect is less than 1%, stop — we won't reach 5%."

Never stop for success: Stopping early because the primary metric looks good is the peeking problem. Wait for the pre-specified end date. The one exception is continuous monitoring with corrected sequential testing methods (mSPRT) — but you must set this up before the test starts.

After the Test: Decision Framework

"No significant effect" is a valid result. It tells you the change didn't matter enough to detect. That's useful information — document it and move on.

A properly designed experiment that produces a null result is a success. You didn't waste engineering time shipping a change that doesn't work.