Quick definition: A normal distribution is a continuous, symmetric, bell-shaped probability distribution described by a mean and standard deviation. It is a useful model and approximation, not a default description of every product metric.
What is a normal distribution?
The normal distribution, also called the Gaussian distribution, describes continuous values concentrated around a center with probability decreasing symmetrically as values move away from it. Its familiar bell shape has one peak at the mean. The distribution is completely characterized by its mean μ, which sets location, and standard deviation σ, which sets spread. A larger standard deviation creates a wider, flatter bell.
Its probability density function is f(x) = [1/(σ√(2π))] exp(−(x − μ)²/(2σ²)). A density is not the probability of one exact measurement; for continuous outcomes, the probability of an exact point is zero. Probability is the area under the curve over an interval, and the total area is one. The distribution is symmetric, so mean, median, and mode coincide at μ.
For a normal variable, roughly 68% of values lie within one standard deviation of the mean, about 95% lie within two, and about 99.7% lie within three. These percentages are useful rules of thumb only when the normal model is reasonably appropriate. They should not be applied mechanically to skewed revenue, bounded conversion rates, count data, or metrics with a spike at zero.
Standardization and the standard normal
Any normal variable can be converted to the standard normal distribution by calculating a z-score: Z = (X − μ)/σ. The standard normal has mean zero and standard deviation one. A z-score says how many standard deviations an observation lies above or below the mean. Tables or software use the standard normal cumulative distribution to calculate tail probabilities and critical values.
For example, if normally distributed task time has mean 10 minutes and standard deviation 2, a task completed in 14 minutes has a z-score of 2. Under that model, it is near the upper 2.5% one-sided tail. The calculation is only as meaningful as the distributional assumption and measurement definition. A mixture of novice and returning users may be bimodal rather than normal, making this tail interpretation misleading.
The normal distribution is closed under addition for independent normal variables: sums and averages remain normal. This algebraic convenience explains its role in many statistical procedures. But product data are not made normal by calling them continuous. Inspect the empirical distribution and understand the process that generated it before relying on a normal model for individual values.
Assumptions and why normality appears
A normal model assumes a continuous, unbounded, symmetric distribution with light tails of a particular shape. Negative values are possible, so it is often unsuitable as a literal model for nonnegative duration, order value, count, or probability data. It can still approximate a measurement away from a boundary, such as aggregated measurement noise or a carefully transformed outcome, but the approximation needs evidence and a decision-relevant rationale.
The central limit theorem is a different idea. Under broad conditions, the distribution of a sample mean or another suitably regular estimator becomes approximately normal as the number of independent observations grows, even when individual observations are not normal. This supports many confidence intervals and tests for mean differences. It does not say that user-level revenue, conversions, or page loads are normally distributed.
Independence matters. Repeated events from the same user, users in one household, stores in a region, or observations during the same outage can be correlated. Treating them as independent inflates the effective sample size and makes a normal approximation appear more precise than warranted. Aggregate at the randomized unit, use cluster-aware methods, or choose an experimental design that matches the dependence.
Normal distribution in A/B testing
Normal approximations commonly enter A/B testing through sampling distributions. A treatment-control difference in mean revenue per user may be analyzed with a t procedure or regression; at an adequate independent sample size, its standardized estimate is often approximately normal. Confidence intervals use this approximate distribution to express sampling uncertainty. For binary conversion, a normal approximation to the difference in proportions can work with enough successes and failures, although exact or alternative methods may be preferable for rare events or small samples.
Do not assume every primary metric should use a normal raw-data model. Revenue is often zero-inflated and right-skewed, latency has long upper tails, and count metrics can be sparse. A distribution check should compare zeros, quantiles, extreme values, missingness, and sample counts by arm. The choice of estimator comes first: mean revenue per assigned user may still be the business target, while bootstrap or robust inference can complement a t approximation.
Normality also matters for diagnostic reasoning. A histogram or Q–Q plot can reveal strong skew, multiple populations, truncation, and tail problems. These visuals do not supply a pass/fail test for validity. With a huge sample, tiny harmless deviations can be statistically detectable; with a small sample, a normality test can miss important deviations. Focus on whether the chosen method produces reliable uncertainty for the actual design and decision.
Worked example: mean order-processing time
A fulfillment team randomly assigns orders to two packing workflows. The primary outcome is order-processing minutes per assigned order. Control has a mean of 18.0 minutes and treatment 17.2 minutes, so the estimated mean difference is −0.8 minutes. Individual processing times are right-skewed because a small number of orders require manual review.
With 30,000 independently assigned orders per arm, the sampling distribution of the mean difference is likely much closer to normal than the raw times. A regression or t-based interval gives −1.20 to −0.40 minutes. The team also bootstraps order-level differences and obtains a similar interval, increasing confidence that the conclusion is not driven by the normal approximation. It checks the 90th and 99th percentiles, cancellation rate, and manual-review rate to ensure the average improvement did not hide severe delays.
If assignment had occurred by warehouse shift rather than order, 60,000 order rows would not be 60,000 independent units. Shifts share staffing and demand conditions. The analysis would need shift-level or cluster-robust uncertainty, and the number of shifts—not just orders—would determine precision. This issue is more consequential than a minor deviation from a perfect bell curve.
Interpreting normal-based results
A 95% normal-based confidence interval is a procedure with repeated-sampling coverage under its assumptions and approximations. It is not a 95% probability that the fixed true effect lies inside one computed interval. More practically, it communicates which mean effects remain compatible with the observed data and method. Pair it with the absolute effect, business threshold, and design checks.
When analysts say “the data are normal,” ask which data: individual outcomes, residuals from a model, daily aggregates, or the estimator’s sampling distribution. These are distinct. A t-test can be useful for a non-normal individual outcome if its assumptions and sample size make the mean difference estimate reliable. Conversely, a visually bell-shaped chart does not repair biased assignment, missing exposure, or an undefined denominator.
Limitations and common mistakes
- Assuming continuous means normal: continuity does not imply symmetry, light tails, or an unbounded range.
- Confusing raw and sampling distributions: the central limit theorem concerns estimates, not every user observation.
- Using normal probabilities for binary outcomes blindly: rare events and small cells can make approximations poor.
- Ignoring bounds and zeroes: conversion, revenue, and duration need outcome-aware modeling.
- Treating rows as independent: clustering can invalidate standard errors even with bell-shaped values.
- Using a normality test as a validity certificate: experiment design and telemetry remain essential.
Frequently asked questions about the normal distribution
Are A/B-test metrics normally distributed?
Often not at the individual level. Binary conversion, revenue, counts, and latency frequently depart from normality, while an estimate such as a group mean can be approximately normal under suitable conditions.
What is the difference between normal and standard normal?
A normal distribution can have any mean and positive standard deviation. The standard normal has mean zero and standard deviation one.
Does the central limit theorem remove all concerns about skew?
No. It needs appropriate independence and enough effective sample size. Very heavy tails, sparse outcomes, clustering, and incorrect units can still make inference unreliable.
Should I transform revenue to make it normal?
Only if the transformed estimand answers the business question and zero handling is prespecified. A log transform changes interpretation; it is not a neutral cleanup step.
How can I check whether a normal approximation is adequate?
Inspect distributions and effective sample size, assess the assignment and dependence structure, compare reasonable methods such as bootstrap sensitivity analysis, and document the choice before interpreting results.
Summary
The normal distribution is a symmetric continuous model defined by its mean and standard deviation. It is fundamental because many estimates become approximately normal under repeated independent sampling, not because every product metric has a bell shape. In A/B testing, distinguish raw outcomes from sampling distributions, match analysis to the randomization unit and estimand, and validate conclusions with distributional and data-quality checks.
Sources
- NIST/SEMATECH e-Handbook: Normal Distribution
- OpenIntro Statistics
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments