Quick definition: Multiple testing is the statistical problem created when a team evaluates more than one hypothesis and treats favorable results as discoveries. Without a planned error-control strategy, the chance of at least one false positive rises as the number of tested claims increases.
What is multiple testing?
Multiple testing occurs whenever several statistical questions can produce a result that the team might act on: several variants versus control, several primary outcomes, a set of segments, repeated interim analyses, or a screen of many product ideas. If every true null hypothesis is tested at alpha 0.05, each individual test has a 5% false-positive probability. Across 20 independent null tests, the chance of at least one false positive is 1 − 0.9520, about 64%.
The issue is selection. A report containing a single predeclared primary test has a different error structure from a dashboard where analysts examine fifty metrics and highlight the smallest p-value. The latter has many opportunities for chance to create an apparently convincing winner. Counting only the result eventually mentioned in a slide deck does not remove the opportunities that led to its selection.
Error rates and correction goals
The family-wise error rate (FWER) is the probability of one or more false rejections in a defined family. It is appropriate when any false positive is costly, such as a high-impact release claim. Bonferroni controls FWER by testing each of m hypotheses at alpha/m. Holm’s step-down procedure also controls FWER and is more powerful. Procedures such as Dunnett’s can be efficient for several treatment-versus-control comparisons, while Tukey’s method addresses all pairwise mean comparisons.
The false discovery rate (FDR) is the expected proportion of false positives among reported rejections. Benjamini–Hochberg control is useful in discovery-oriented work where identifying promising signals matters and selected findings will be validated. It is less conservative than FWER control, but it intentionally permits some false discoveries in expectation. It is not an assurance that any one flagged feature has a 95% probability of working.
Multiplicity can also be addressed with a pre-specified hierarchy or gatekeeping strategy. Test the primary outcome first; test a secondary outcome only if the primary passes, perhaps with alpha recycled according to a documented rule. This preserves an error-rate guarantee while reflecting how decisions actually proceed. The method must be designed before results, not used to create a favorable path after a primary result fails.
Defining a hypothesis family
A family is the collection of hypotheses whose joint false-positive risk matters for a decision. There is no universal mechanical answer. For one A/B/n experiment, a family may contain all three treatment-versus-control comparisons for a primary metric. For a product research program, it may contain all 30 mechanism metrics evaluated as a discovery screen. For a release decision, it may include every claim that could independently trigger rollout.
Define families based on the questions and actions, then document the rationale. Do not split a coherent family merely because one set of outcomes looks unfavorable. Conversely, correcting every number across unrelated experiments forever can be needlessly conservative. A transparent analysis plan distinguishes confirmatory, guardrail, diagnostic, and exploratory metrics. The principles in primary and guardrail metrics make it easier to state which results can actually establish a win.
Multiple testing in A/B testing
Online experiments create multiplicity in more ways than teams expect. A control plus four treatments creates four treatment-control claims; measuring conversion, revenue, retention, and latency creates additional endpoints; breaking each result down by country, device, and acquisition channel adds more; and checking daily results creates repeated looks. One unadjusted p-value may be valid in isolation but not as part of a broad winner-selection process.
Start by protecting one primary metric tied to the product hypothesis. State the comparison, direction if justified, alpha, and stopping rule. Classify other metrics: guardrails may have predeclared harm thresholds, secondary outcomes may be confirmatory only under a hierarchy, and unplanned slices should be exploratory. For multi-variant experiments, decide whether the objective is to identify any difference, select a treatment over control, or compare every pair. Each objective supports a different contrast family and correction.
Multiplicity adjustment assumes the underlying p-values are valid. It cannot fix optional stopping, broken treatment assignment, data leakage, or a changed denominator. Reconcile assigned and exposed users, verify the full outcome window, and investigate anomalies such as sample ratio mismatch. Applying Benjamini–Hochberg to a dashboard built from invalid event-level observations does not produce trustworthy discoveries.
Worked example: secondary outcomes
A team tests a redesigned upgrade flow and predeclares purchase conversion as its primary outcome, with cancellation rate, first-week feature activation, support contact, and revenue per eligible user as four confirmatory secondary outcomes. The primary conversion comparison passes its fixed alpha rule. For the secondary family, the team uses Holm FWER control at 0.05 because each could influence the release decision. Ordered secondary p-values are 0.006, 0.018, 0.043, and 0.31.
Holm compares 0.006 with 0.0125 and rejects. It compares 0.018 with 0.0167 and stops. Only the first secondary claim is confirmed; the next two are not confirmed even though 0.018 is below 0.05 unadjusted. The report gives all effects and intervals: activation improved by 1.1 points, revenue is uncertain, support is flat, and cancellation has a small uncertain increase. The team does not describe every p below 0.05 as validation.
Had the four outcomes been a discovery screen intended to generate follow-up studies, an FDR target might be defensible instead. That different choice should be justified by the cost of false discoveries and the planned confirmation process. The method follows the decision context, not the result the team hopes to publish.
A practical multiple-testing workflow
- List every hypothesis, variant comparison, endpoint, segment, and interim look that could influence a claim or action.
- Classify analyses as primary, confirmatory secondary, guardrail, diagnostic, or exploratory before examining outcomes.
- Define coherent families and choose FWER, FDR, a hierarchy, or another justified method based on false-positive consequences.
- Predeclare valid tests, directionality, alpha allocation, stopping rules, and the exact correction procedure.
- Report the full family, raw estimates, intervals, adjustment method, adjusted results, and excluded analyses honestly.
Risks and common mistakes
- Calling only the chosen winner a test. The family includes alternatives considered before selection.
- Adjusting after results are known. A correction cannot undo unrecorded searches through metrics, segments, and time windows.
- Correcting the wrong collection. Family definition should match claims and decisions, not a convenient spreadsheet tab.
- Using FDR for a no-false-win requirement. FDR permits false discoveries in expectation; use FWER or stronger confirmation where needed.
- Ignoring effect size. A corrected discovery may still be too small, harmful on a guardrail, or operationally infeasible.
- Using adjustment as a data-quality substitute. Invalid individual p-values remain invalid after correction.
Frequently asked questions about multiple testing
Do I need to adjust for every metric in an A/B test?
Not necessarily. Define a protected primary metric and a clear role for other metrics. Adjust coherent families of confirmatory claims; label diagnostic and exploratory results rather than silently treating them as confirmations.
What is the difference between FWER and FDR?
FWER limits the chance of any false positive in a family. FDR limits the expected proportion of false positives among discoveries. FWER is stricter; FDR is often more useful for discovery screens.
Does testing multiple segments require correction?
If segment results are candidates for a claim, targeting, or rollout decision, yes: include them in a planned family or confirm them independently. Post-hoc segment patterns are hypotheses, not proof of heterogeneous effects.
Can I simply lower alpha to 0.01?
A lower unadjusted alpha may be a deliberate policy, but it does not automatically give a stated family-wise guarantee for an arbitrary number of tests. Use a procedure linked to the number and structure of hypotheses.
Does correction reduce power?
Usually, because stricter evidence is required to protect against false positives. Reduce the cost through focused hypotheses, adequate sample size, hierarchical design, and choosing an error criterion that matches the goal.
Summary
Multiple testing is not a software nuisance but a selection problem: every metric, comparison, segment, and look that could generate a claim contributes to false-positive risk. Define the family and decision role before analysis, use an error criterion suited to consequences, validate the underlying experiment, and report effects and uncertainty alongside adjusted decisions.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, Multiple Comparisons.
- Hochberg and Tamhane, Multiple Comparison Procedures.
- Benjamini and Hochberg (1995), Controlling the False Discovery Rate.