Statistics·Glossary term

Median

Median A/B testing Reference guide

Median is a concept used in statistical inference.

Quick definition: The median is the middle value after observations are ordered. Half the observations are at or below it and half are at or above it, making it a robust summary of central location.

What is the median?

The median describes the center of ordered data by rank rather than by total. With an odd number of observations, it is the single middle value. With an even number, it is conventionally the average of the two middle values. For the sorted values 1, 3, 4, 7, and 90, the median is 4. For 2, 5, 7, and 11, it is 6 because the two central values are 5 and 7.

Unlike the arithmetic average, the median is not pulled upward or downward by the magnitude of extreme values. Replacing 90 with 9000 in the first example changes the mean dramatically but leaves the median at 4. This robustness makes the median useful for describing a typical transaction, session duration, or latency observation when the data have long tails.

The median is the 50th percentile, also written as the 0.5 quantile. It says something about position, not total value: at least half the values are no greater than the median and at least half are no smaller. It need not be an observed value, and for tied or discrete data it may not split the sample into exactly equal counts. A report should specify the quantile convention and measurement unit when those details matter.

How the median is calculated

First order the n values from smallest to largest. If n is odd, the sample median is the value with rank (n + 1)/2. If n is even, many introductory definitions use the midpoint of ranks n/2 and n/2 + 1. Software can implement quantiles differently, particularly when estimating percentiles from small samples, so use a documented method for reproducible dashboards and experiments.

For grouped or streaming data, an exact median may require keeping values or an appropriate quantile sketch. A median cannot generally be reconstructed from a mean, count, minimum, and maximum. Nor can medians from separate groups be simply averaged to get the overall median. The distribution of values and group sizes matter; retain unit-level data or a suitable quantile summary for the population of interest.

The population median is any value m for which P(X ≤ m) ≥ 0.5 and P(X ≥ m) ≥ 0.5. Continuous distributions often have one unique median. A discrete distribution can have an interval or multiple valid median values. This is not an error; it reflects that rank-based centers can be less uniquely defined when many observations tie.

Assumptions and appropriate use

The median requires an ordered scale. It can summarize numerical values and ordered categories, but not nominal labels such as browser names. It is meaningful only after the unit, eligibility rules, and time window are fixed. “Median checkout time” might mean all started checkouts, completed checkouts only, or completed sessions within a timeout; each has a different interpretation and can be affected differently by a product change.

Robustness is a benefit, not a cure for data-quality problems. A treatment-specific event loss, duplicated records, or shifted clock can alter the median just as it can alter a mean. Inspect counts, missingness, censoring, and definitions across groups. For duration metrics, decide how to handle sessions that do not complete. Excluding treatment users who abandon can make their remaining completion times look artificially good.

Statistical inference for medians is possible through quantile regression, rank-based procedures, bootstrap methods, or other techniques. A standard t-test estimates a mean difference, not a median difference. Do not calculate medians and then apply an arbitrary mean-based formula to them. Select an interval and test whose assumptions match the estimand, the randomization unit, sample size, ties, and dependence structure.

Median in A/B testing

Teams often use median page-load time, median task-completion time, median order value, or median support-response time to describe a central user experience. It can reveal an improvement for the middle user even when a few slow sessions make the mean noisy. For service-quality metrics, pair it with upper percentiles: reducing median latency while worsening the 95th percentile can leave a vulnerable minority with a worse experience.

Decide whether the median itself is the primary causal estimand. For expected total revenue, the mean revenue per assigned user usually better represents the financial outcome because every dollar contributes to total revenue. Median order value among purchasers is conditional on purchasing and can change simply because the variant changes which users buy. It is useful descriptive context, but it should not be substituted for revenue per assigned user without changing the business question.

Predefine the outcome population. If user-level assignment is used, calculate one value per user according to the metric definition or use an analysis that preserves user-level clustering. Analyze all eligible assigned users when possible. The denominator and censoring rule deserve explicit review, especially for time-to-event metrics where an uncompleted task has no natural completion time.

Worked example: time to finish onboarding

A product team tests a simplified onboarding flow. Among users who complete onboarding within seven days, control has a median completion time of 8.4 minutes and treatment has a median of 6.9 minutes. The observed median difference is −1.5 minutes. A bootstrap interval, constructed by resampling randomized users rather than event rows, ranges from −2.2 to −0.7 minutes.

That result alone is incomplete because it conditions on completion. Completion is 61% in control and 56% in treatment. The shorter median may mean the simplified flow helps completers, but it may also reflect that users who would take longer are now abandoning. The team reports both the completion rate and the time distribution, investigates funnel exits, and treats completion as a primary guardrail. It does not announce that onboarding became universally faster.

Suppose completion remains equal while the treatment lowers the median and 90th percentile with no increase in errors. The evidence then supports a better typical and tail completion experience for the studied users. The exact rollout decision still considers the value of faster activation, engineering cost, and whether the test population matches the intended launch.

Interpreting median results

Phrase results as rank statements: “the estimated median seven-day completion time was 1.5 minutes lower,” not “every user saved 1.5 minutes.” The median says nothing directly about the mean or total time saved. Two arms can have equal medians and very different tails; conversely, a small median change can coexist with a large mean change driven by rare expensive failures.

Show sample sizes, the share with missing or censored outcomes, selected percentiles, and uncertainty. A confidence interval for a median difference expresses uncertainty under the stated sampling method; it does not demonstrate that users were comparable if assignment or logging failed. Link the reading to the experiment’s primary outcome and guardrails rather than presenting the median as a standalone success metric.

Limitations and common mistakes

  • Equating median with mean: they answer different questions in skewed data.
  • Averaging subgroup medians: this does not recover the overall median.
  • Ignoring censored outcomes: excluding non-completers can bias time metrics.
  • Using a mean test for a median claim: analysis must match the estimand.
  • Hiding tails: medians cannot show severe experiences in the upper distribution.
  • Changing the population after assignment: post-treatment filtering can destroy causal interpretation.

Frequently asked questions about the median

When is the median better than the mean?

It is better for a robust typical-value description when outliers or skew dominate the mean. It is not automatically better for decisions about totals, such as expected revenue.

Can the median be calculated for categories?

Only if categories have a meaningful order, such as satisfaction levels. It is not meaningful for unordered labels.

Why is median order value risky as a primary test metric?

It includes only purchasers. If a treatment changes purchase probability, the purchaser population differs between arms, so it does not measure overall revenue impact.

Does a lower median latency guarantee a faster experience?

No. It describes the middle observation. Check high percentiles, failures, and the population included in the calculation.

How should incomplete tasks be handled?

Define the policy before launch. Consider completion rate, time-to-event methods, a fixed follow-up window, and a user-level treatment of censoring that matches the product decision.

Summary

The median is the middle ordered value and the 50th percentile. Its resistance to extreme values makes it valuable for typical experience metrics, but it does not measure total value or show tails. In A/B tests, predefine the population and censoring rules, use inference designed for quantiles, and interpret the median beside completion, percentiles, and other guardrails.

Sources