A/B Testing

What is A/B testing?

A/B testing is a controlled way to answer a decision question such as: Does changing the checkout layout increase completed purchases without harming payment success or page performance?

The current experience is usually called A, the control, or the baseline. The proposed experience is B, the treatment, variant, or variation. Both experiences run during the same period for comparable users. A primary metric is selected before the test, and the difference between A and B is estimated after enough data has been collected.

The method is used in marketing, ecommerce, SaaS, mobile apps, media, advertising, pricing, recommendation systems, search ranking, and feature development. A testable change can be visual—such as a headline—or technical, such as a ranking algorithm or model prompt.

Why randomization matters

Suppose visitors who see the new page convert at 6% while visitors who see the old page convert at 5%. Without randomization, the groups might differ in device, geography, intent, acquisition channel, loyalty, or timing. The observed difference could reflect those pre-existing differences rather than the page.

Randomization assigns eligible units—usually users, accounts, households, or sessions—to variants using a rule unrelated to their likely outcome. It does not guarantee identical groups in every sample, but it makes systematic imbalance less likely and provides the basis for causal inference.

The five core components of a valid A/B test

1. A testable hypothesis

A hypothesis states the change, expected effect, and reason:

A hypothesis is not a prediction that B must win. It is a falsifiable explanation that can be supported or not supported by data.

2. A control and a treatment

The control is normally the current experience. The treatment contains the planned change. Document the exact difference, including copy, targeting, eligibility, code version, and exposure event. If multiple elements change, the test estimates the effect of the package, not of each individual element.

3. A primary metric

The primary metric is the main outcome used to evaluate the hypothesis. Use secondary metrics to explain behavior and guardrail metrics to detect harm.

4. Random traffic allocation

A 50/50 split is common because it gives both arms similar sample sizes and generally maximizes power for a two-arm comparison. It is not mandatory. A risky change may start with 90/10 or 95/5, then ramp gradually. The allocation must be planned, logged, and checked.

5. A predeclared analysis and stopping rule

Before launch, define the baseline metric, minimum detectable effect, significance level, desired power, expected traffic, sample size, duration, primary comparison, and stopping rule. Changing these after seeing favorable results makes the evidence harder to interpret.

A/B testing workflow

1. Identify a business or user problem.
2. Form a testable hypothesis.
3. Define population, eligibility, and randomization unit.
4. Select primary and guardrail metrics.
5. Estimate baseline, MDE, sample size, and duration.
6. Implement, QA, and run the experiment.
7. Analyze effect, uncertainty, quality checks, and segments.
8. Ship, iterate, roll back, or collect more evidence.

A/B testing formulas

For a binary conversion metric:

If A converts at 5% and B at 5.5%, the absolute change is 0.5 percentage points, while the relative uplift is 10%. Always report both; they answer different questions.

Sample size, power, and MDE

Sample-size planning is a pre-test activity. The main inputs are baseline conversion rate, minimum detectable effect (MDE), significance level, desired power, number of variants, and allocation.

Power is commonly expressed as 1 − β. An 80% powered test has a 20% chance of missing an effect of the planned size in repeated experiments. If required traffic is unrealistic, choose a higher-volume surface, use variance reduction, or reconsider whether this decision is suitable for A/B testing.

Statistical significance: what it does and does not say

In a frequentist analysis, a p-value describes how surprising the observed result would be under the null hypothesis of no difference. A p-value below a predeclared threshold such as 0.05 is often called statistically significant.

Report conversion rate for every arm, absolute and relative effect, an interval estimate, sample and conversion counts, primary and guardrail outcomes, quality checks, and the decision.

How long should an A/B test run?

Run until the planned sample size is reached and the test has covered relevant operating cycles. A weekly cycle often matters because weekday and weekend behavior can differ. Do not stop simply because a dashboard temporarily shows a positive result.

Repeatedly checking results and stopping when p < 0.05—known as peeking or optional stopping—can inflate the false-positive rate. If early stopping is operationally important, use sequential testing with valid boundaries rather than ordinary fixed-horizon inference.

Frequentist and Bayesian A/B testing

Both approaches still require valid randomization, reliable metrics, and business judgment. A statistical framework does not repair faulty exposure logging or contaminated groups.

A/B testing versus related methods

Common failure modes

• No randomization.
• Sample-ratio mismatch.
• Identity switching between variants.
• Denominator drift.
• Post-treatment segmentation.
• Multiple comparisons without correction.
• Peeking and optional stopping.
• Novelty effects.
• Underpowered design.
• Metric mismatch.
• Asymmetric event logging.
• Winner’s curse.

Pre-launch checklist

• Problem and decision are documented.
• Hypothesis states change, outcome, and rationale.
• Control and treatment are precisely specified.
• Unit of randomization is appropriate.
• Primary and guardrail metrics are fixed.
• Baseline, MDE, alpha, power, and sample size are documented.
• Allocation and identity persistence are tested.
• Event logging and denominators are QA’d.
• SRM and data-quality checks are available.
• Stopping and multiple-testing rules are documented.
• Rollout and rollback plans exist.

FAQ

Is 50/50 traffic splitting required?

No. It is common and efficient for a low-risk two-arm test, but unequal allocation can be appropriate for staged or high-risk rollouts.

How many users do I need?

There is no universal number. Calculate it from baseline conversion, MDE, significance level, power, allocation, and the number of variants.

Can A/B testing prove causation?

A properly randomized, well-instrumented experiment can support a causal interpretation. Randomization alone cannot fix exposure errors, contaminated assignments, faulty metrics, or interference.

What should I do when the result is not significant?

Check the confidence interval, achieved sample, power, data quality, and practical effect. “Not significant” does not automatically mean “no effect.”

Summary

A/B testing is not a button-color contest or a dashboard status. It is a causal decision process. The strongest experiments connect a real problem to a testable hypothesis, randomize the right unit, measure a meaningful outcome, plan sample size and stopping rules in advance, validate the data, and interpret statistical evidence together with business value.

Sources

1. Optimizely: A/B testing
2. GrowthBook: What is A/B testing?
3. Statsig: A/B testing methods and metrics
4. AB Tasty: The Ultimate A/B Testing Guide
5. Optimizely: MDE and experiment prioritization
6. Mida: Power statistics for A/B testing
7. Omniconvert: Statistical power analysis
8. Statsig: Statistical significance
9. GrowthBook: Open Guide to A/B Testing

Related terms

• A/B Test
• A/B/n Testing
• A/A Testing
• Split Testing
• Bucket Testing