You have a hypothesis: a new checkout flow will increase conversions. You run an A/B test for four days, see the variant beating the control by 7%, and call it a win. Six weeks later, the change is live — and conversions are flat.
What went wrong? Almost certainly, you didn't have enough data. Running experiments without pre-calculating sample size is one of the most expensive mistakes in product development, and it's also one of the most preventable.
This guide explains how to calculate the right sample size for an A/B test, what parameters drive the math, and how to avoid the traps that invalidate most experiments.
Why sample size is a decision, not a coincidence
Sample size is not something you discover after the fact — it's something you decide before the experiment begins. Running a test and stopping when the result "looks good" is called peeking, and it dramatically inflates your false positive rate.
A properly powered experiment answers a specific question: if a real effect of a given size exists, what is the probability I will detect it? That probability is called statistical power. Industry standard is 80%, meaning you accept a 20% chance of missing a real effect.
📌 The sample size is always computed before data collection begins. Changing it mid-experiment is p-hacking.
The three inputs you must define
Every sample size calculation requires three parameters. Get these wrong and the formula output is meaningless.
1. Significance level (α)
Alpha is the probability you're willing to accept a false positive — declaring a winner when there is none. The convention is α = 0.05 (5%), giving you a 95% confidence threshold. For high-stakes decisions, some teams use 0.01.
2. Statistical power (1 − β)
Power is the probability of detecting a real effect if it truly exists. The standard is 80% power (β = 0.20). Higher power requires more users: a 90% power test needs roughly 30% more observations than an 80% power test.
3. Minimum Detectable Effect (MDE)
The MDE is the smallest improvement you care about detecting. This is the most critical — and most misunderstood — parameter. A smaller MDE (detecting a 2% lift) requires far more data than a larger one (detecting a 20% lift).
⚠️ Setting your MDE based on what effect you hope to see is motivated reasoning. Set it based on what effect would be business-meaningful.
The sample size formula
Sample size is computed per comparison, then rolled up across arms given how you split traffic. For two proportions with allocation ratio \( \kappa = n_{\mathrm{B}} / n_{\mathrm{A}} \) (how many users B gets for each user in A), a standard power formula is:
where \( \delta = p_{\mathrm{B}} - p_{\mathrm{A}} \) (absolute MDE), \( z_{\alpha/2} \approx 1.96 \) for \( \alpha = 0.05 \) (two-tailed), \( z_{\beta} \approx 0.84 \) for 80% power (or \( \approx 1.28 \) for 90% power). With three or more arms you size each pairwise contrast you care about, then take the traffic plan that covers the slowest arm.
Notice that \( \delta \) appears squared in the denominator — halving your MDE roughly quadruples the required sample. Small effects are expensive to detect.
Special case: classic 50/50 A/B
When you have exactly two groups and an equal split (\( \kappa = 1 \)), a common pooled-variance shortcut collapses to the textbook formula — sample size per group, with a hard-coded factor of 2:
where \( \bar p = (p_{\mathrm{A}} + p_{\mathrm{B}}) / 2 \). Total users ≈ \( 2n \). This is the clean teaching case — not the full design space.
📌 Treat the 50/50 A/B formula as a baseline example. In production you often run A/B/C (or more), protect control with an 80/20 or 70/15/15 split, or power several comparisons at once. Those choices change \( n \) per arm — a calculator that only assumes equal two-way splits understates real experiments. The AB-Labz sample size calculator handles multi-variant designs and arbitrary traffic allocation, not just 50/50.
A worked example (50/50 A/B)
Stick with the simple case so the arithmetic stays readable. Suppose checkout conversion is 3.2% and you want to detect a 15% relative lift — 0.032 × 0.15 = 0.48 pp, so 3.2% → 3.68%. With α = 0.05, 80% power, and equal traffic:
Total under 50/50: ~45,000 users across both variants.
So you need ~45,000 users across both groups if traffic is split evenly. At 5,000 unique visitors per week → 2,500 per arm → about 9 weeks to fill. Unequal splits or a third variant change that calendar: the thinner arm becomes the bottleneck. That's planning input, not a reason to stop early when a p-value flickers green.
The four most common mistakes
1. Stopping early when you see significance
The most dangerous mistake. Repeatedly checking p-values while an experiment runs inflates your false positive rate far beyond 5%. An experiment checked at every 10% of data accumulation can have an effective false positive rate closer to 30%. Use sequential testing methods if early stopping is required.
2. Not accounting for novelty effects
New experiences often see short-term engagement spikes simply because they're new. Run experiments for at least one full business cycle — and a minimum of one week to account for day-of-week variation.
3. Changing the experiment while it runs
Mid-experiment changes — fixing a bug in the variant, adjusting traffic allocation, or changing the success metric — invalidate the results. Document your hypothesis, metrics, and sample size before starting. Don't touch the test.
4. Ignoring variance in continuous metrics
If you're measuring revenue per user rather than conversion rate, variance matters enormously. CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance reduction technique that can cut the required sample size by 30–50% when you have pre-experiment covariates on the same users.
The AB-Labz sample size calculator
AB-Labz includes a standalone sample size calculator — an extra planning tool. You type in the inputs yourself: baseline (conversion rate, or mean + standard deviation for numeric metrics), MDE, significance level, power, how many variants, and how you split traffic.
It returns sample size per variant (including unequal splits) and the total across groups. That covers the designs the textbook 50/50 formula skips: A/B/C, 80/20, 70/15/15, conversion or continuous metrics — so you can size the test you actually plan to run.
Practical rules of thumb
- Always calculate sample size before starting. Never run "until it looks good."
- Set your MDE based on business impact, not what you expect to see.
- Plan for at least one full business cycle (minimum one week).
- If traffic is limited, test bigger changes — small MDEs are expensive.
- Use CUPED if you have pre-experiment data on the same users.
- Use sequential testing only if your platform supports valid stopping boundaries.
Summary
Sample size is the foundation of a valid experiment. Under-powered tests produce misleading results that can ship the wrong change. Over-powered tests waste time and traffic. The goal is to compute exactly the sample size you need for the effect size you care about — then collect exactly that much data before deciding.
The math isn't complicated once you understand the three inputs: significance level (α), power (1 − β), and minimum detectable effect (MDE). Everything else follows from there.
Calculate sample size in AB-Labz
A standalone planner for multi-variant designs, unequal splits, and conversion or numeric metrics — not just 50/50.