Descriptive Statistics

What are descriptive statistics?

Descriptive statistics turn a collection of observations into an auditable account of what happened in the data. For an experiment, that account may include assigned users, exposure counts, conversion rates, mean revenue per assigned user, medians, percentiles, missing-event rates, and the distribution of time to complete a task. A table or chart is descriptive when it summarizes observed values without attaching a probability calculation about an underlying population or a causal claim.

The distinction matters in product work. If treatment conversion is 5.4% and control conversion is 5.0%, the observed difference is 0.4 percentage points. That is a useful descriptive result, but its reliability depends on sample size, random variation, assignment, and measurement. Inferential methods add a model for those sources of uncertainty. Randomization supports a causal interpretation only when eligibility, exposure, and telemetry preserve the intended comparison.

Core measures and when to use them

Counts and proportions answer basic coverage questions: how many users were assigned, exposed, eligible, converted, or missing an outcome? A conversion rate is conversions / eligible assigned users when that is the declared denominator. Always state the numerator, denominator, attribution window, and whether repeated events are deduplicated. Counts alone can conceal unequal traffic allocation; proportions alone can conceal a very small sample.

Location describes a typical value. The arithmetic mean, mean = Σx_i / n, is the average contribution per analysis unit and is essential for additive business quantities such as revenue per assigned user. The median is the middle observation and is more resistant to isolated extreme values. The mode identifies the most frequent category but is rarely sufficient for product decisions. A mean and median that differ sharply signal skew and should prompt distribution review rather than an automatic choice of the more flattering number.

Spread describes variability. The range uses only the minimum and maximum and is unstable in large samples. Quantiles, such as the 25th, 50th, 90th, and 99th percentiles, show where most observations lie. The interquartile range is Q3 − Q1. Sample variance is Σ(x_i − x̄)²/(n − 1), and standard deviation is its square root. Standard deviation concerns variation among individual observations; the standard error concerns uncertainty in an estimated mean and should never be substituted silently.

Shape includes asymmetry, tails, spikes at zero, multiple modes, and impossible values. Revenue commonly has many zeroes and a long right tail. Page-load time can have a long tail caused by a minority of failed or slow sessions. A mean can be the correct estimand even in those cases, but a histogram or percentile table reveals operational behavior that one average hides. Report a fixed, documented clipping or winsorization rule if one is used.

Data requirements and boundaries

Descriptive calculations require fewer distributional assumptions than hypothesis tests, but they still require coherent data. The unit must be defined: user, account, session, order, or cluster. If users are randomized but events are summarized at event level, highly active users receive more weight. If revenue per purchaser is reported without revenue per assigned user, a variant that changes purchasing incidence can be obscured. The aggregation rule is part of the metric, not a formatting choice.

Check the observation window and data maturity. A seven-day revenue metric cannot be interpreted on day two as though all users had seven days to purchase. Identify late-arriving events, bot filtering, duplicate identifiers, currency conversion, and changes to the tracking implementation. A clean-looking mean from an incomplete join is still a misleading summary. Before comparing variant totals, investigate sample ratio mismatch and unexpected eligibility differences.

Descriptive statistics in A/B testing

Start an experiment readout with a descriptive data-quality table. Include users assigned and exposed by variant, allocation share, dates, exclusions, primary-metric numerator and denominator, missingness, and important guardrails. Then show the primary outcome in business units: for example, control conversion 5.00% (2,500/50,000) and treatment conversion 5.35% (2,675/50,000), an observed lift of 0.35 percentage points or 7.0% relative to baseline.

For continuous outcomes, show sample size, mean, median, selected percentiles, standard deviation, and zero share. This makes metric sensitivity visible. If treatment mean revenue is higher because of three enterprise purchases, that could be a real commercial gain, a random tail event, or a logging anomaly; the appropriate response is investigation and an uncertainty analysis suited to the predeclared estimand, not deleting inconvenient purchasers. The discussion of log-transforming revenue metrics explains why a transformation changes the quantity being summarized.

Segment tables are useful for monitoring exposure and understanding mechanisms, but selected segments are not automatically confirmatory findings. Define key segments in advance, report their denominators, and distinguish a descriptive pattern from a segment-specific treatment-effect estimate. Testing many segments can create attractive chance patterns; see multiple comparisons in A/B testing for the decision risk.

Worked example: onboarding completion

A team randomizes 20,000 eligible new users equally between the existing onboarding flow and a shorter flow. During the planned 14-day window, 9,980 control and 10,020 treatment users are recorded. Completion is 2,396/9,980 = 24.01% in control and 2,556/10,020 = 25.51% in treatment. The descriptive absolute difference is 25.51% − 24.01% = 1.50 percentage points; relative lift is 1.50 / 24.01 = 6.25%.

The team also reports median time to first value: 11.2 minutes in control and 9.1 minutes in treatment; the 90th percentiles are 53 and 49 minutes. A small increase in treatment-side missing completion events is found. The result is not ready for a launch claim until the event definition is reconciled, because differential missingness can manufacture or conceal a conversion difference. After validating data, the team applies its preplanned inference method and reviews retention and support-contact guardrails, consistent with the distinction between primary and guardrail metrics.

Interpretation and reporting workflow

1. Name the population, assignment unit, analysis unit, eligibility rule, and observation window.
2. Reconcile assigned, exposed, and analyzed counts before viewing outcome summaries.
3. Show outcome numerators, denominators, central tendency, spread, and distribution features appropriate to the metric.
4. Express the observed treatment-control difference in an absolute operational unit and, where helpful, a relative scale with its baseline.
5. Use the prespecified inferential and decision method for uncertainty; label post-hoc slices as exploratory.

Interpret summaries in context. A 20-millisecond mean latency improvement may be precise but irrelevant, while a small average revenue change can matter at high volume. Conversely, a favorable mean can coexist with a harmful tail experience. Connect summaries to a practical threshold, guardrails, implementation cost, and the target population rather than treating a single dashboard card as a decision.

Limitations and common mistakes

• Equating a sample difference with proof. Descriptive differences quantify what was observed, not how likely it is under chance or whether the treatment caused it.
• Using the wrong denominator. A purchaser-only average answers a different question from revenue per eligible user.
• Reporting only a mean. Skew, zero inflation, outliers, and user harm can disappear in an average.
• Comparing immature windows. Variants need comparable opportunity for delayed outcomes.
• Hiding exclusions or transformations. Record them before analysis and report their effect on the population and metric.
• Mining segments. A selected high-lift subgroup is a hypothesis unless it was protected by a plan or confirmed independently.

Frequently asked questions about descriptive statistics

Are descriptive statistics the same as analytics?

They are a core part of analytics: summaries, tables, and visualizations that characterize data. Analytics can also include forecasting, causal inference, and optimization, which add other models and goals.

Should I report mean or median revenue?

Report the measure matching the decision. Mean revenue per assigned user estimates expected revenue contribution; the median describes a typical user. With skewed revenue, show both plus zero share and tail percentiles.

Can a histogram replace a statistical test?

No. A histogram reveals distributional features but does not quantify uncertainty about an experiment effect. It complements the planned analysis and often exposes reasons to revisit the metric or data pipeline.

What is the difference between standard deviation and standard error?

Standard deviation describes the variation of observations. Standard error describes estimated sampling variation of a statistic, such as a mean. They answer different questions and have different denominators.

Why show raw counts with percentages?

Counts show information volume and expose small denominators; percentages make groups with different sizes comparable. Reporting both makes a summary easier to audit.

Summary

Descriptive statistics make an experiment’s data, metric behavior, and observed treatment-control differences visible. They are necessary for quality checks and transparent reporting, but causal and decision claims require valid design, a suitable uncertainty method, and practical thresholds.

Sources

• NIST/SEMATECH e-Handbook: Exploratory Data Analysis
• NIST/SEMATECH e-Handbook: Measures of Location and Spread
• Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments

Related terms

• Two-Proportion Z-Test
• Z-Test
• T-Test
• Independent Samples T-Test
• Paired T-Test