Quick definition: The Mann–Whitney U test is a rank-based test for comparing two independent groups. It tests whether observations from one group tend to be larger than observations from the other under its distributional assumptions; it is not automatically a test of mean or median difference.
What is the Mann–Whitney U test?
The Mann–Whitney U test, also called the Wilcoxon rank-sum test, compares two independent samples without assuming normally distributed outcomes. It replaces raw outcome values with their ranks in the pooled sample. If one group tends to have higher values, its observations tend to receive larger ranks and the rank sums differ more than expected under a common distribution.
The test is often described as “nonparametric,” but that does not mean assumption-free. Its basic null hypothesis is that the two groups have the same distribution. Under additional assumptions that distributions differ only by a location shift and have similar shape and spread, it can be interpreted as a test of a location difference, often loosely called a median difference. Without those conditions, a significant result may reflect differences in spread, shape, tails, or the probability that a random treatment outcome exceeds a random control outcome.
Ranks, U statistics, and effect sizes
Pool all n1 + n2 observations, sort them, and assign ranks from one through N. Tied values receive their average rank. Let R1 be the rank sum for group one. Its U statistic is U1 = R1 − n1(n1 + 1)/2; equivalently, U counts ordered cross-group pairs for which a group-one observation exceeds a group-two observation, with tied pairs split equally. The other U is n1n2 − U1.
For small samples, exact distributions of U can yield an exact p-value. For larger samples, software commonly uses a normal approximation with a tie correction and possibly a continuity correction. State whether the test is one- or two-sided, how ties were handled, and whether the p-value was exact or asymptotic. A one-sided test must be chosen from the product hypothesis before outcomes are seen; choosing direction after inspection doubles the opportunity for a misleading result.
A useful effect summary is the probability of superiority: the probability that a randomly selected treatment outcome is greater than a randomly selected control outcome, with half weight for ties. Depending on direction, it can be derived from U divided by n1n2. Cliff’s delta is a related effect size ranging from −1 to 1. These quantify stochastic ordering, not average dollar lift or an individual user’s probability of improving.
Assumptions and interpretation limits
Observations must be independent within and between groups at the level analyzed. If users are randomized but each user contributes many session durations, ranking sessions treats correlated measurements as new independent evidence. Aggregate to one user-level summary or use a repeated-measures or clustered approach. Likewise, account-level assignment requires account-level or cluster-aware inference.
The outcome must be at least ordinal so that ordering is meaningful. The test handles skew and outliers differently from a t-test because magnitudes are discarded after ranking. That is sometimes beneficial for an ordinal satisfaction score or heavily contaminated measurement, but it can be a serious limitation for revenue: the difference between $1 and $100,000 receives no more rank distance than the difference between $1 and $2. If expected revenue per user is the decision target, a rank test does not directly estimate it.
Many zeroes and ties are common in product data. Tie corrections exist, but a rank test can have low information when most values are identical and can be hard to explain. A binary conversion metric should normally use a binomial method rather than Mann–Whitney. For time-to-event data with censoring, use survival analysis rather than ranking observed times as if censored users completed at their last recorded time.
Mann–Whitney U in A/B testing
Use the test only when its estimand aligns with the decision. It can be reasonable for a predeclared ordinal outcome such as a five-point post-task rating, or for a user-level latency measure when the question is whether the treatment generally shifts the distribution and an appropriate rank-based effect is useful. It is not a generic substitute for checking model assumptions after a t-test produces an unwanted result.
For an experiment with task-time outcomes, first decide how to handle users who abandon the task, users with no qualifying event, timeout behavior, bots, and duplicate sessions. Excluding treatment users who never complete can induce post-treatment selection bias. A median among completers and a probability-of-superiority result may both be relevant descriptive or secondary analyses, but neither automatically answers overall product success. Consider a joint metric or a planned survival analysis if completion and time both matter.
Randomization and measurement remain prerequisites. Verify the assignment population, exposure log, and analysis denominator. A rank test can be mathematically flawless yet infer a delivery artifact if one variant fails to record an event. This is why experiment quality checks, including sample-ratio mismatch diagnostics, precede outcome inference.
Worked example: support-resolution time
A support team randomizes 4,000 eligible tickets to an existing triage flow or a new routing rule. It predefines the outcome as hours from assignment to resolution among tickets with a recorded resolution, while separately monitoring unresolved-ticket rate. There are 1,980 control and 2,020 treatment tickets. Treatment times tend to rank lower; the two-sided Mann–Whitney test gives p = 0.012 and the probability that a randomly selected treatment ticket resolves faster than a control ticket is 0.535.
The result supports evidence of a distributional shift toward faster treatment resolution under the specified analysis. The 0.535 effect means treatment beats control in about 53.5% of random cross-group pairs after tie handling, not that every ticket is 53.5% faster. The team reports medians and 90th percentiles too: 6.2 versus 6.8 median hours and 31 versus 30 hours at the 90th percentile. The improvement in typical time coexists with a slightly worse tail, so the rollout decision includes the unresolved rate and high-priority-ticket guardrail.
If resolution logging differs between flows, the test cannot support a causal claim. If the actual goal is average labor cost, the team should analyze cost per assigned ticket, because ranks deliberately ignore the magnitude of expensive cases. The method follows the decision question; it should not select the question after viewing a favorable p-value.
Practical workflow
- Specify an ordinal or continuous user-level outcome and state whether the decision concerns distributional ordering, a location shift, or another estimand.
- Define the independent analysis unit, treatment of ties, zeros, censoring, missing outcomes, and one- or two-sided alternative.
- Validate assignment, exposure, data completeness, and comparability of the observation window.
- Report group sizes, medians and quantiles, U statistic, p-value method, probability of superiority or Cliff’s delta, and the relevant uncertainty interval where available.
- Apply predeclared multiplicity rules and interpret results with guardrails and the business quantity actually being optimized.
Risks and common mistakes
- Calling it a median test unconditionally. A median-shift interpretation needs similar distribution shapes; otherwise it tests a broader distributional hypothesis.
- Using it for repeated events. Rank-based methods still require independent units.
- Ignoring magnitudes. Ranking is unsuitable when the size of a revenue, cost, or harm difference is central.
- Switching tests after seeing data. Select the primary method and estimand before outcomes to avoid researcher degrees of freedom.
- Dropping non-completers. Post-treatment exclusion can bias a time comparison and hide a treatment-induced completion change.
- Reporting only p-values. Give distribution summaries and an interpretable rank-based effect size.
Frequently asked questions about the Mann–Whitney U test
Is Mann–Whitney U a test of medians?
Only under a location-shift interpretation with similarly shaped distributions. In general it tests whether the two distributions are the same, or whether one tends to produce larger values.
When is it preferable to a t-test?
It can be useful for an ordinal metric or when distributional ordering is the predeclared target and independence holds. It is not automatically preferable merely because a continuous metric is skewed.
Can it compare conversion rates?
A binary outcome creates many ties and is usually more directly analyzed with a two-proportion test or logistic regression. Those methods estimate a probability difference or ratio more transparently.
What does a probability of superiority of 0.50 mean?
It indicates no tendency for a random observation from one group to exceed one from the other, after ties receive half credit. It does not prove identical distributions.
Does the test require equal sample sizes?
No. Unequal group sizes are allowed, although very small or imbalanced groups affect precision. The randomization and data-collection plan should still explain the allocation.
Summary
The Mann–Whitney U test compares ranks from two independent groups and tests whether their outcome distributions differ. It is useful for a clearly defined ordinal or distributional A/B-test question, but it does not inherently estimate a mean effect or prove a median difference. Choose it before analysis, use the randomized unit, report an interpretable effect size, and retain metric and data-quality safeguards.
Sources
- Mann and Whitney (1947), On a Test of Whether One of Two Random Variables Is Stochastically Larger Than the Other.
- NIST/SEMATECH e-Handbook of Statistical Methods, Rank Sum Test.
- Hollander, Wolfe, and Chicken, Nonparametric Statistical Methods.