Most A/B test sample-size calculators begin and end with conversion rate. That works when the outcome is yes or no. It is the wrong model when your success metric is revenue per assigned user, user-level spend, number of items per assigned user, or another numeric value.
Numeric metrics need one extra ingredient: variance. Two products can have the same average revenue and the same meaningful lift, but the product with more spread-out orders needs far more users to distinguish signal from noise. This guide explains the calculation, shows what unequal traffic does to it, and includes a small Python function you can copy.
When to use a numeric-metric calculation
Use this approach when each experiment unit contributes a number and you plan to compare group means with a two-sample test. Typical examples include revenue per assigned user, user-level spend, task-completion time when it is defined for every randomized unit, or support contacts per account. Metrics such as average order value may require ratio-metric methods depending on their definition.
Do not confuse a numeric metric with a ratio metric. “Revenue per user” is usually safe when revenue is aggregated per randomized user first. Average order value is revenue divided by orders, and “revenue per session” has a similar denominator problem: users may contribute different numbers of orders or sessions. Choose the unit of randomization first, then calculate the metric at that same unit.
The four inputs that drive the sample
- Baseline mean \(\mu\): the historic average at the same unit you will analyse.
- Standard deviation \(\sigma\): the spread of those user-level values. This is what conversion calculators do not ask for.
- Minimum detectable effect \(\delta\): the smallest absolute change worth detecting, or a relative change converted into an absolute amount.
- Alpha and power: commonly 0.05 (two-sided) and 80%, chosen before the experiment starts.
Use historical data, not a guess: calculate the mean and standard deviation on a recent, representative period. Include zeros for assigned users who did not purchase when the metric is revenue per user. Excluding non-purchasers turns it into a conditional metric and can bias the experiment comparison.
The numeric sample-size formula
For a balanced two-arm test with a common standard deviation, the standard normal approximation expresses the requirement through standardized effect size, also called Cohen’s \(d\):
At two-sided \(\alpha = 0.05\), \(z_{1-\alpha/2} \approx 1.96\). At 80% power, \(z_{\text{power}} \approx 0.84\).
If a metric averages $40 per user with a standard deviation of $120, a meaningful +$4 change has \(d = 4/120 = 0.033\). That is a very small standardized effect, so the required sample will be large. The formula is not being pessimistic: it is quantifying how much individual spending varies.
What unequal traffic allocation costs
A 50/50 split is statistically efficient for a single A/B comparison because both groups contribute equally to the standard error. Product constraints sometimes require an unequal allocation: protect an existing experience with 80/20 traffic, expose a risky change gradually, or reserve more users for a control shared by several variants.
Let \(w_T\) be the treatment share and \(w_C = 1-w_T\) the control share. Relative to a 50/50 A/B test, the total sample is inflated by:
The factor is 1.0 at 50/50. At 80/20 it is 1.5625, so you need about 56% more total users for the same power and effect. Giving one arm too little traffic lengthens the test because each group has its own required observation count at the chosen allocation. Translate those per-group requirements into calendar time using the expected traffic to each group.
“Weights” are not always traffic allocation: the formula above is for planned group shares. Analysis weights, inverse-propensity weights, or revenue weights change the estimator and its variance. Do not plug them into this allocation formula as if they were traffic percentages.
Worked example: revenue per user at an 80/20 split
Suppose historical revenue per assigned user has a mean of $40 and standard deviation of $120. Your business threshold is a +10% lift, or +$4 per user. With a two-sided 5% alpha, 80% power, and 20% treatment allocation:
That is approximately 35,320 control users and 8,830 treatment users after rounding. Treat it as a planning approximation; calculate the final design with the exact assumptions used by your analysis.
Copyable Python example
This small function uses only the Python standard library. It implements the normal approximation above for a two-arm numeric-metric test. Pass the treatment traffic share as a decimal: 0.2 means 80% control and 20% treatment.
from math import ceil
from statistics import NormalDist
def numeric_ab_sample_size(
mean, sd, relative_mde,
alpha=0.05, power=0.80, treatment_share=0.50,
):
"""Return planned users in control and treatment.
mean, sd: historical user-level metric summary
relative_mde: 0.10 means a 10% relative change
treatment_share: planned fraction assigned to treatment
"""
if mean == 0 or sd <= 0:
raise ValueError("mean must be non-zero and sd must be positive")
if not 0 < treatment_share < 1:
raise ValueError("treatment_share must be between 0 and 1")
absolute_mde = abs(mean * relative_mde)
effect_size = absolute_mde / sd
z = NormalDist()
z_alpha = z.inv_cdf(1 - alpha / 2)
z_power = z.inv_cdf(power)
n_equal_per_arm = 2 * ((z_alpha + z_power) / effect_size) ** 2
control_share = 1 - treatment_share
design_factor = 1 / (4 * control_share * treatment_share)
total = ceil(2 * n_equal_per_arm * design_factor)
return {
"control": ceil(total * control_share),
"treatment": ceil(total * treatment_share),
"total": total,
}
print(numeric_ab_sample_size(40, 120, 0.10, treatment_share=0.20))
# {'control': 35320, 'treatment': 8830, 'total': 44150}The exact final integers can differ slightly with rounding. The important part is the model: mean and standard deviation are measured per randomized user, the MDE is absolute after conversion from relative lift, and allocation changes the total through the design factor.
Assumptions and when to use a stronger method
The calculation is a useful large-sample approximation for a mean comparison. It assumes independent experiment units, a reasonably stable variance estimate, and a distribution for which the sampling distribution of the mean is well behaved at the planned sample. It is not a promise that an individual revenue distribution is normal.
- Heavy-tailed revenue: calculate per-user values and inspect outliers. A transformation or robust procedure can be appropriate, but it may change the estimand; plan and analyse the same question. See our guide to what a log transform changes.
- Repeated observations: aggregate to the randomized unit or account for correlation; sessions from one user are not independent users.
- Several variants or primary comparisons: size each planned pairwise contrast and account for the multiple-testing rule you actually plan to use. A proportional multi-variant allocation is a planning shortcut, not a full k-arm power calculation; see our guide to multiple comparisons.
- Pre-experiment covariates: variance reduction such as CUPED can reduce the required sample, but only estimate that reduction from historical data rather than assuming it.
Plan the design you will actually run
The AB-Labz Sample Size Calculator accepts conversion and numeric metrics, and supports multiple variants and custom allocation. That matters when the design is not the textbook 50/50 conversion test.
Start with a business-relevant MDE and representative variance, calculate the required observations for each group at your chosen allocation, then translate each into calendar time from its expected traffic. Check that the realized traffic follows your sample-allocation plan. Once the experiment starts, use the same metric definition and analysis method you planned for. For a broader primer on alpha, power, and conversion experiments, see our A/B test sample-size guide.
Summary
Numeric A/B test sizing depends on the effect you care about relative to the metric’s standard deviation. The smaller that standardized effect, the more users you need. Unequal traffic is sometimes necessary, but it raises the total through a simple allocation penalty. Use the copyable calculation for a transparent first estimate, then validate the final design against your real metric distribution and analysis plan.
Calculate the sample for your actual experiment
Plan numeric and conversion metrics with multiple variants and your intended traffic allocation.