Statistics·Glossary term

Mean

Mean A/B testing Reference guide

Mean is a concept used in statistical inference.

Quick definition: The mean is the arithmetic average: the sum of all numeric observations divided by their number. In experiments, it often estimates expected value per randomized user.

What is the mean?

The arithmetic mean summarizes a set of values with one number. For observations x1, x2, …, xn, the sample mean is x̄ = (Σxi)/n. It is the point at which the total deviation above and below balances to zero. For example, values 2, 4, 5, and 9 have a mean of 5 because their sum, 20, divided by four is 5.

“Average” is often used as a synonym for mean, but it can also refer to a median, mode, or weighted average. Name the statistic in a report. The mean treats every observation equally in the sum, although a high-value observation contributes more numerically than a low-value one. That feature makes it a natural summary for total outcomes such as revenue, cost, or total support minutes.

A sample mean estimates a population mean: the long-run expected value for a defined population and measurement window. It is not necessarily the experience of a typical user. With a right-skewed metric, most users can have values below the mean while a small upper tail raises it substantially. That distinction is important when a product decision concerns total economic value rather than the middle individual experience.

Mean formulas and related quantities

The unweighted formula is x̄ = Σxi/n. If observations represent groups of unequal size, use a weighted mean: x̄w = Σwixi/Σwi. Combining a mean of 10 for 100 users with a mean of 20 for 10 users yields (100 × 10 + 10 × 20)/110 = 10.91, not 15. Averaging daily averages without daily user counts is a common source of incorrect dashboards.

The mean difference between treatment and control is Δ = x̄T − x̄C. Relative lift is often reported as Δ/x̄C, but the absolute difference conveys scale and is usually easier to connect to an operational decision. A mean of 5.40 dollars versus 5.00 dollars is a $0.40 per-user change and an 8% relative increase.

Variance measures how spread out values are around the mean. For a sample it is commonly estimated with s² = Σ(xi − x̄)²/(n − 1). Larger variance makes a mean less precise for a fixed sample size. The standard error of the mean is approximately s/√n for independent observations; for a difference in independent group means, standard errors from both groups are combined. These formulas describe sampling uncertainty, not whether a metric or experiment was implemented correctly.

Assumptions and boundaries

A mean exists and can be calculated whenever finite numeric values are available. Drawing inferences from it requires more. The analysis unit should match randomization or account for clustering, the observation window and denominator should be defined, and measurement should be comparable across arms. A page-event table with several rows per user does not supply several independent randomized users.

Individual data do not have to be normally distributed for a mean comparison. With sufficiently many independent units, the sampling distribution of a mean often becomes approximately normal under the central limit theorem. But small samples, severe skew, sparse zero-inflated revenue, extreme tails, and dependence can make standard errors or normal approximations unreliable. Bootstrap intervals, robust methods, or a model appropriate to the outcome can be useful, provided they preserve the randomized unit and answer the intended question.

Outliers deserve investigation rather than automatic removal. A duplicate event, bot, currency-conversion bug, or data-entry error is not a legitimate outcome and should be corrected under a consistent rule. A genuine large enterprise order may be exactly the revenue value the mean is meant to include. Predefine capping, winsorization, exclusion, and refund policies; do not choose them after learning which version wins.

Mean in A/B testing

Many experiment metrics are means: revenue per assigned user, orders per visitor, time to complete a task, support contacts per account, and page-load time. For revenue per user, assign zero revenue to randomized users who do not buy. Average order value answers a different, conditional question because it excludes nonbuyers; if the treatment changes conversion, comparing only purchasers can introduce post-treatment selection bias.

The primary analysis compares treatment and control means at the unit of assignment. A two-sample t procedure or regression is commonly used for a continuous mean metric. A confidence interval gives a plausible range for the treatment-control difference under the method’s assumptions. It is more informative than a pass/fail p-value because it shows whether a result is both directionally credible and large enough to matter.

Use the mean alongside distributional diagnostics. Compare zero rates, quantiles, top-value contribution, missingness, and duplicate rates. A client-side change that loses low-value sessions can raise observed mean revenue without raising actual revenue. The metric definition should state the numerator, denominator, unit, currency, refunds, eligibility, and outcome maturity so readers know exactly what the average represents.

Worked example: revenue per assigned user

An ecommerce team assigns 12,000 eligible visitors to each checkout experience. After fourteen days, control records $60,000 in validated net revenue and treatment records $66,000. The control mean is $5.00 per assigned visitor; treatment is $5.50. The estimated treatment effect is $0.50 per visitor, a 10% relative lift.

A user-level analysis estimates a standard error of $0.19 for the difference, producing an approximate 95% confidence interval from $0.13 to $0.87. The team has evidence that treatment increased expected revenue in the eligible population. It also checks conversion, refunds, payment errors, and the distribution of order values. Two high-value treatment orders are legitimate but account for some lift, so the readout reports both the all-user mean and a preplanned capped-value sensitivity analysis.

The team does not claim that every user spends $0.50 more. Most users still spend zero. The correct interpretation is an expected additional $0.50 net revenue per eligible assigned visitor over the chosen window, subject to the experiment’s validity checks. That quantity can be scaled cautiously to traffic forecasts and weighed against implementation cost and guardrails.

How to interpret a mean

Always attach a unit and population: “mean seven-day net revenue per eligible randomized visitor” is interpretable; “average revenue improved” is not. Pair the estimate with sample sizes, uncertainty, absolute and relative effects, and the decision threshold. A very small mean increase can be statistically significant in a large test but still fail to pay for complexity or harm another metric.

The mean is often the right estimand even when it is not typical. Expected total revenue equals mean revenue per user multiplied by users, so the upper tail is economically relevant. If the question is about the ordinary experience, pair it with a median, percentiles, or the fraction exceeding a service threshold. Changing from mean to median changes the causal quantity being estimated; it is not merely a more robust version of the same answer.

Limitations and common mistakes

  • Leaving the denominator implicit: per purchaser, order, visitor, and assigned user are different means.
  • Averaging averages equally: combine subgroup means using their relevant weights.
  • Calling the mean typical: skewed outcomes often put most users below it.
  • Dropping legitimate tail values: this can remove real business impact.
  • Ignoring dependence: repeated events or cluster assignment require suitable analysis.
  • Choosing handling rules after results: post hoc transformations undermine trustworthy uncertainty.

Frequently asked questions about the mean

Is mean the same as average?

Usually in casual speech, but “average” can also mean median, mode, or weighted mean. Use “arithmetic mean” when precision matters.

Should non-converters have zero revenue?

Yes when measuring revenue per eligible or assigned user. Excluding them changes the metric to revenue conditional on conversion or purchase.

Do mean-based A/B tests require normal data?

No. The appropriate inference depends on sample size, independence, tails, and the method. Normality of individual outcomes is not a universal requirement.

When is a weighted mean needed?

Use it when combining summaries from groups of different sizes or exposures, or when weights are part of the defined estimand.

Can a mean be negative?

Yes. Means of signed measures such as profit change, net balance, or change in latency can be negative, even if some individual values are positive.

Summary

The mean is total value divided by the number of observations and commonly estimates expected value for a defined population. In A/B testing, compare user-level means with an explicit denominator, appropriate uncertainty, distribution checks, and predeclared data rules. Use a median or percentiles as complementary context when the typical experience matters.

Sources