Statistics·Glossary term

Distribution

Distribution A/B testing Reference guide

Distribution is a concept used in statistical inference.

Quick definition: A probability distribution describes the possible values of a random variable and the probability or relative frequency associated with each value or range of values.

What is a distribution in statistics?

A distribution is a model or description of how values of a variable are spread across possible outcomes. It answers more than “what is typical?”: it shows which values are possible, how often they occur, where values concentrate, how variable they are, and whether unusually large or small observations are plausible. A histogram of checkout values, a table of conversion outcomes, and a mathematical probability model are all ways to describe a distribution, although they serve different purposes.

In statistics, a random variable maps an uncertain outcome to a number or category. A distribution then assigns probabilities to that variable. For a discrete variable, such as whether an eligible user converts, it assigns probability to individual values. For a continuous variable, such as page-load time or order value, it assigns probability to ranges; the probability of one exact measured value is normally zero. The distribution is about the process that could generate observations, while a sample is the finite set of observations actually collected.

Distribution is also used informally for the empirical distribution: the observed pattern in a data set. That pattern can reveal skew, spikes, gaps, outliers, rounding, or data-quality problems before any formal test is run. It does not automatically establish a population model. A two-week sample of campaign traffic, for example, may not represent normal traffic after the campaign ends.

Core components and notation

A discrete probability mass function gives P(X = x) for each possible value x, with probabilities that are nonnegative and sum to one. A continuous probability density function f(x) is nonnegative and has total area one; probability over an interval is its area: P(a ≤ X ≤ b) = ∫ab f(x) dx. A cumulative distribution function works for either type: F(x) = P(X ≤ x).

Useful summaries include the mean or expected value, median, variance, standard deviation, quantiles, and tail probabilities. No single summary fully characterizes every distribution. Two revenue distributions can have the same mean while one is stable and the other has many zeroes plus a few very large orders. A median can better represent a typical observation in a skewed distribution, whereas the mean may remain the right quantity for a revenue-per-user decision because it incorporates all value.

Common named distributions encode different outcome structures. A Bernoulli distribution models one binary outcome; a binomial distribution models a count of binary successes under additional assumptions. A normal distribution is symmetric and bell shaped. Poisson models are often considered for event counts, while lognormal or gamma-like models can describe positive, right-skewed values. A named model is an assumption to assess, not a decorative label chosen after looking at a chart.

Shape, variability, and interpretation

Location describes where values tend to fall; spread describes how dispersed they are; shape describes asymmetry, multimodality, and tails. Right skew means a long upper tail, as is common for spending and latency. Bimodality may indicate two user populations, an implementation change, a mix of device types, or an erroneous merge of units. A sharp spike at zero can be real, such as users with no purchases, or a sign that missing values were silently converted to zero.

Sampling distributions are a separate but important idea. If analysts repeatedly drew comparable samples and calculated an estimator, such as the difference in conversion rates, the distribution of those estimates is its sampling distribution. Standard errors, confidence intervals, and hypothesis tests rely on a model for that repeated-sampling behavior. It is not the same as the distribution of individual user outcomes. A binary user conversion can be highly non-normal even when the average conversion rate is approximately normal at a large sample size.

Interpret a chart in the context of the unit, population, and time window. One row per event gives a different distribution from one row per user; a seven-day outcome differs from a same-session outcome. Aggregating too early can hide dependence or changes in composition. Keeping the raw unit and a clear metric definition makes it possible to diagnose whether a surprising shape reflects users, instrumentation, or a transformation.

Why distributions matter in A/B testing

An A/B test compares distributions or summaries of distributions between randomized groups. The precise estimand matters. For a conversion metric, the usual target is the difference in the mean of a user-level 0/1 outcome. For revenue per assigned user, it is often the difference in means of a zero-inflated, right-skewed variable. For latency, a team may care about a high percentile as well as the mean. Random assignment supports causal comparison only when assignment, eligibility, exposure, and measurement are credible.

Inspect distributions before trusting a single dashboard number. Compare sample counts, zero rates, duplicate rates, missingness, extreme values, and quantiles by arm. A treatment that fails to log a portion of its low-value users can create an artificial mean increase. A client-side variant that logs an outcome after a different delay can change the observed distribution without changing customer behavior. These are measurement problems, not evidence of a treatment effect.

Distributional checks also guide analysis. Repeated observations from the same user, account, store, or household are not independent merely because the table has many rows. Heavy tails can make a mean estimate noisy; pre-specified robust methods, winsorization rules, transformations, or bootstrap procedures may be appropriate depending on the estimand. Such choices should be defined before results determine the preferred answer. See log-transforming revenue metrics for a related trade-off.

Worked scenario: revenue per visitor

Consider 10,000 assigned visitors in each arm. In both arms, most visitors spend nothing. Control has mean revenue of $2.00 per visitor, while treatment has $2.12. The estimated effect is $2.12 − $2.00 = $0.12 per assigned visitor. The treatment distribution also has a slightly larger upper tail because several customers placed large orders.

The result should not be interpreted from the mean alone. The team checks whether the extra tail comes from legitimate completed orders, duplicate purchase events, one enterprise account mistakenly counted as many visitors, or a changed currency conversion rule. It reports uncertainty for the mean difference and separately checks purchase conversion, refund rate, and order-value quantiles. If a small number of real high-value orders drive the change, that may be commercially meaningful; it is not automatically an “outlier problem” to erase.

If the decision concerns a typical customer experience rather than expected revenue, median order value could answer a different question. Replacing the mean with the median just because the mean is inconvenient would change the estimand. A sound readout names both the outcome distribution and the business quantity that the chosen estimator is meant to represent.

Assumptions and analysis choices

Every inferential method makes assumptions, explicit or implicit. A two-sample t procedure concerns means and can work well with non-normal individual observations when the sample is sufficiently large and independent at the analyzed unit, but severe clustering, unstable variance, or a sparse heavy-tailed metric can require a different approach. A proportion procedure depends on the binary outcome and denominator being correctly defined. Resampling methods still require representative units and a design that respects dependence.

Transformations can improve numerical stability or make a model fit an operational question, but they alter interpretation. Taking the logarithm of positive revenue emphasizes relative differences and needs a rule for zeroes. Capping values limits the influence of tails but must be justified by a business or data rule applied equally in both arms. Analysts should report the primary estimand, transformation, exclusion rules, and sensitivity analyses rather than presenting a favorable specification as inevitable.

Limitations and common mistakes

  • Assuming normality from a familiar chart: many product metrics are zero inflated, skewed, or mixtures of populations.
  • Confusing an empirical distribution with a causal result: differences can arise from traffic composition without random assignment.
  • Analyzing event rows as independent users: repeated behavior can understate uncertainty.
  • Deleting real tail values automatically: extreme observations can be valid customer value or harm.
  • Changing transformations after seeing the winner: it creates selection bias and unclear interpretation.
  • Ignoring outcome maturity: late conversions and refunds can change the observed distribution.

Frequently asked questions about distributions

Is a distribution the same as a histogram?

No. A histogram is one visualization of observed values. A probability distribution is a mathematical description of possible values and probabilities; a histogram may help assess a candidate model.

Why can a mean be misleading?

The mean is sensitive to the entire distribution, including tails. It can be the correct expected-value metric while still failing to describe a typical observation. Pair it with quantiles, counts, and a clear decision question.

Do A/B tests require normally distributed data?

No. The appropriate method depends on the outcome, estimator, sample size, and dependence structure. Binary conversion, counts, and skewed revenue should not be treated as identical data types.

What is the difference between a population and sampling distribution?

A population distribution describes individual outcomes. A sampling distribution describes how an estimate would vary across repeated comparable samples.

How should I handle outliers?

First determine whether they are errors, fraud, duplicate records, or legitimate observations. Predefine any exclusion or treatment rule and use it consistently across experiment arms.

Summary

A distribution describes possible outcomes and their probabilities or observed frequencies. In experimentation, its shape, unit, and tails determine what a metric means and which analysis is credible. Compare distributions across randomized arms, validate measurement, choose an estimator that matches the decision, and communicate uncertainty rather than relying on one summary statistic.

Sources