Quick definition: Multiple comparisons are the individual pairwise or planned contrasts evaluated after or within an analysis involving several groups, outcomes, or hypotheses. They require a defined comparison family and an adjustment strategy when any favorable comparison could be treated as a finding.
What are multiple comparisons?
Multiple comparisons arise when analysts compare more than two groups or evaluate several contrast questions. In a control-plus-three-treatment experiment, possible comparisons include each treatment versus control and each treatment versus every other treatment. With four groups there are six pairwise contrasts. Testing each one at an unadjusted 5% level makes a false positive more likely than testing only one specified contrast.
The phrase is sometimes used interchangeably with multiple testing, but a comparison is specifically a relationship between estimates. A contrast can be a simple difference, such as treatment B minus control, or a weighted combination, such as the average of two treatments minus control. The right set of comparisons comes from the experiment’s decision question. An omnibus ANOVA asks whether any group means differ; it does not identify which experience wins or authorize every pairwise claim.
Plan comparisons before the experiment
Before launch, identify the comparison that matters most. If the business question is whether any new checkout design beats the current experience, predeclare treatment-versus-control contrasts. If the question is whether all design elements have different mean effects, all-pair comparisons may be relevant. If a product hypothesis specifies that treatments A and B should be pooled because they share a mechanism, define the weighted contrast and its rationale before results are known.
Planned comparisons are generally more interpretable and more powerful than an indiscriminate post-hoc search. They are not automatically exempt from multiplicity: three planned contrasts that can each trigger a win still form a family. However, a limited, well-justified set needs less severe adjustment than every imaginable pair, metric, and segment. Document the population, outcome, contrast coefficients, direction, alpha allocation, and practical threshold alongside the experiment plan.
Unplanned comparisons are valuable for exploration, diagnosis, and future hypothesis generation, but their label matters. Do not show only the pair with the largest observed lift and imply it was the pre-specified winner. Selection inflates apparent effects and changes p-value interpretation. A new comparison can inform a follow-up experiment rather than becoming the sole basis for rollout.
Methods for controlling comparison error
Bonferroni divides a family alpha by the number of comparisons. It is simple and valid under arbitrary dependence but can be conservative. The Holm procedure tests ordered p-values sequentially and is uniformly at least as powerful as Bonferroni while retaining family-wise error control. Both can be applied to diverse valid contrast p-values, including regression coefficients, proportions, and nonparametric tests.
Dunnett’s procedure is designed for several treatments compared with one common control. Because all comparisons share the control estimate, it uses their correlation and is usually more powerful than treating them as unrelated pairwise tests. Tukey’s honestly significant difference method is designed for all pairwise mean comparisons in an ANOVA setting. Scheffé’s method protects a very broad class of possible contrasts but is often conservative for a small planned set. The choice should reflect the intended claim, model, and assumptions—not whichever produces the smallest adjusted p-value.
For large discovery collections, false discovery rate procedures such as Benjamini–Hochberg control the expected proportion of false discoveries rather than the probability of any false comparison. This may suit screening, but it is not a replacement for strong family-wise protection when a single false treatment claim is unacceptable. The article on multiple comparisons in A/B testing explains how these trade-offs affect product decisions.
Multiple comparisons in A/B/n testing
Suppose an experiment has one control and four variants. A common primary decision is “which, if any, treatment should replace control?” The natural comparison family is the four treatment-control differences on the primary metric. A Dunnett adjustment or Holm correction can control family-wise false-positive risk. If the team instead also compares all variants against one another to name the numerically best experience, those extra contrasts require inclusion in the plan or a clearly exploratory label.
Multiple comparisons also occur across endpoints. A treatment could increase conversion, revenue, activation, and retention while changing latency and cancellation. Separate comparison families may be defensible if the roles are distinct: one primary success contrast, a gatekept secondary family, and specified harm guardrails. The framework in primary and guardrail metrics helps prevent a team from declaring success solely because one of many secondary outcomes crossed 0.05.
Data quality is upstream of adjustment. Compare assigned, exposed, and analyzed units by variant; confirm the analysis unit matches randomization; use common attribution windows; and investigate unexpected allocation. A sample ratio mismatch or variant-specific event loss can invalidate every contrast. A corrected p-value is not a certificate that the underlying comparison is causal.
Worked example: three pricing pages
A SaaS team randomizes 90,000 eligible visitors equally among control and three pricing pages. Its primary metric is 14-day purchase conversion, and the plan specifies the three treatment-versus-control comparisons with family alpha 0.05. The observed conversion rates are 4.00% for control, 4.12% for A, 4.47% for B, and 3.96% for C. The unadjusted p-values for A, B, and C versus control are 0.20, 0.001, and 0.55.
Applying Holm, order the p-values: 0.001, 0.20, 0.55. Compare 0.001 with 0.05/3 = 0.0167 and reject B’s null. Compare 0.20 with 0.05/2 = 0.025 and stop. The team can claim evidence that B differs positively from control under the planned family-wise rule. It cannot make a confirmatory claim for A merely because its point estimate is positive, and it does not need to compare B with C to make the defined rollout decision.
B’s absolute estimated lift is 0.47 percentage points, with an adjusted confidence interval that should be reported alongside expected incremental purchases, support contacts, revenue per visitor, refund behavior, and latency. If B has the largest observed lift, the estimate is still subject to winner’s-curse exaggeration: selected winners tend to look better than their true effects. A staged rollout or confirmation experiment can reduce that risk when the consequence is large.
Interpretation and reporting workflow
- State the groups, primary outcome, population, randomization unit, and all comparisons capable of supporting a decision.
- Choose contrasts that represent the decision: treatment-control, all-pairs, factorial effects, or a predeclared weighted contrast.
- Select a family-wise, FDR, or hierarchical error strategy before outcomes and verify that individual p-values are valid.
- Report every comparison in the family with effect estimates, raw and adjusted uncertainty measures, and the named adjustment method.
- Base rollout on effect size, practical threshold, guardrails, data quality, and implementation evidence—not merely the lowest adjusted p-value.
Confidence intervals deserve the same care as p-values. A collection of ordinary 95% intervals does not usually provide simultaneous 95% coverage for all comparisons. Where practical, report intervals adjusted to match the family-wise procedure or clearly distinguish ordinary per-comparison intervals from the formal decision rule. Always present effects in original product units: percentage points, dollars per eligible user, or milliseconds, rather than presenting a ranking alone.
Risks and common mistakes
- Running every pairwise test after a significant ANOVA. The omnibus result does not remove the need for a comparison procedure.
- Choosing a correction after viewing outcomes. Method selection must follow the comparison objective and error tolerance, not favorable output.
- Reporting only adjusted p-values. Effects, intervals, denominators, and business thresholds are necessary to evaluate a comparison.
- Equating a ranked winner with a proven winner. The largest observed estimate can be noise and often overstates the selected treatment’s true effect.
- Forgetting dependence and clustering. Shared users, accounts, or stores can make standard contrast errors too small.
- Combining unrelated claims indiscriminately. Overly broad families can destroy useful power; define families around coherent decisions.
Frequently asked questions about multiple comparisons
Do I need an omnibus ANOVA before pairwise comparisons?
No. Preplanned contrasts can be tested directly with an appropriate multiplicity procedure. An omnibus test can be useful, but it does not itself answer treatment-versus-control questions.
When should I use Dunnett’s procedure?
Use it when several treatments are compared with one common control and those are the only confirmatory comparisons needed. It exploits the shared control and is often more powerful than generic corrections.
What is the difference between Tukey and Holm?
Tukey is specialized for all pairwise mean comparisons in an ANOVA-type setting. Holm is a flexible step-down adjustment for any defined set of valid p-values and can be used for selected contrasts.
Can I compare the winning variant with control after selecting it?
Not with an ordinary unadjusted test if selection occurred among several variants. Include all candidate treatment-control comparisons in the family or use a design and analysis that accounts for selection.
Are adjusted p-values enough to choose a rollout?
No. They control a statistical error rate under assumptions. Choose using absolute effect size, uncertainty, guardrails, cost, data integrity, and the target rollout population.
Summary
Multiple comparisons are the contrast claims made among several groups or outcomes. A valid A/B/n analysis defines the comparisons that can support action, controls the relevant joint error rate with a suitable procedure, and reports all planned effects rather than selecting a favorable pair after the fact. Adjustment strengthens a well-designed experiment; it cannot cure invalid randomization, measurement, or product reasoning.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, Multiple Comparisons.
- Dunnett, C. W. (1955), A Multiple Comparison Procedure for Comparing Several Treatments with a Control.
- Tukey, J. W., The Problem of Multiple Comparisons.