Quick definition: The Holm–Bonferroni correction is a step-down multiple-testing procedure that controls the family-wise error rate: the probability of making one or more false-positive rejections within a defined family of hypotheses. It is uniformly at least as powerful as the ordinary Bonferroni correction.
What is the Holm–Bonferroni correction?
When a team evaluates several hypotheses at a conventional per-test alpha of 0.05, chance alone makes an apparently significant result increasingly likely. The Holm–Bonferroni procedure, usually called the Holm correction, modifies the decision thresholds so that the probability of at least one false rejection in the whole planned family is no greater than the chosen alpha, provided the individual p-values are valid. It applies to any set of valid p-values, not only t-tests.
The method is a sequentially rejective improvement on Bonferroni. Bonferroni tests every one of m hypotheses at alpha / m. Holm begins with the smallest p-value at that strict threshold, but relaxes the threshold as earlier hypotheses are rejected. Consequently it makes every rejection that Bonferroni would make and can make additional rejections. Its guarantee is strong family-wise error rate control, meaning it holds even when some null hypotheses are false and without requiring independent test statistics.
How the step-down procedure works
First define the family, choose a family-wise alpha such as 0.05, and calculate the p-values using the analysis methods specified before results are examined. Sort the m p-values from smallest to largest: p(1) ≤ p(2) ≤ ... ≤ p(m). Compare p(1) with alpha / m. If it exceeds that boundary, stop and reject none. If it is below the boundary, reject its hypothesis and compare p(2) with alpha / (m − 1). Continue until the first p-value that exceeds its threshold; that hypothesis and all larger p-values are not rejected.
For six hypotheses at alpha 0.05, the ordered boundaries are 0.00833, 0.010, 0.0125, 0.0167, 0.025, and 0.05. Suppose ordered p-values are 0.002, 0.006, 0.011, 0.020, 0.033, and 0.41. The first three pass their respective limits. The fourth, 0.020, exceeds 0.0167, so the procedure stops: only the first three hypotheses are rejected. It would be incorrect to continue and compare 0.033 with 0.025 after the earlier failure.
Software often reports Holm-adjusted p-values. For the p-value at ordered rank i, form (m − i + 1)p(i), then replace it with the running maximum of this quantity and earlier values, capped at one. A hypothesis is rejected when its adjusted p-value is at most alpha. Adjusted values are useful in reports, but preserve the original family and ordering information so that reviewers can reconstruct the decision.
Assumptions and scope
Holm’s multiplicity guarantee relies on valid marginal p-values. Each test must use the right randomization and analysis unit, a pre-specified or otherwise valid stopping rule, and an appropriate model for the outcome. The correction cannot repair biased assignment, a changing metric definition, missing exposure records, or repeated unplanned looks at accumulating data. A sample ratio mismatch is a data-quality problem to investigate before inferential adjustment.
The procedure makes no independence assumption among p-values, which is an important practical advantage for correlated product metrics. Its family-wise protection is nevertheless conditional on a defensible family. A family is a collection of claims that will be considered together for a decision: perhaps all treatment-versus-control comparisons for one primary metric, or all predeclared claims in an experiment report. It is not “the significant results we decided to show.” Defining a family after inspecting outcomes defeats the intended protection.
Family-wise error rate is a conservative target when a single false launch claim is costly. For broad hypothesis discovery, false discovery rate methods may yield more useful signal, but they answer a different question. Holm also does not decide what effects matter. A corrected p-value below 0.05 may correspond to an effect too small to justify engineering cost, while a non-rejection may leave a commercially meaningful improvement plausible if the interval is wide.
Holm correction in A/B and A/B/n testing
In a multi-variant experiment, Holm is useful for a finite, predeclared collection of comparisons. Consider a control and three treatments with a primary conversion metric. If the team plans three treatment-versus-control contrasts, it can calculate the three valid contrast p-values and apply Holm at family alpha 0.05. That controls the chance of falsely declaring any treatment superior when the relevant null effects are absent. It is more flexible than procedures tailored to a particular model and can also be used for a mixed collection of planned endpoints.
Do not assume the correction is needed for every number on a dashboard. A product team should designate one primary outcome and the decision it governs, then categorize secondary outcomes as confirmatory, guardrail, diagnostic, or exploratory. The framework in primary and guardrail metrics helps make those roles explicit. If several confirmatory primary claims are all eligible to trigger a success decision, they normally belong in a multiplicity strategy. Guardrails may use separate harm-monitoring rules rather than being casually folded into a “win” family.
For conversion, analyze a user-level success indicator when users are randomized. For revenue per user, aggregate to one contribution per assigned user, including zero revenue where that matches the estimand. A correction applied to event-level p-values does not fix pseudoreplication. Likewise, applying Holm after choosing the most favorable of dozens of segments understates the selection problem; either include the full planned segment family or label the result exploratory and confirm it.
Worked workflow: checkout variants
A retailer randomizes 120,000 eligible users equally among control and three checkout designs. Before launch, it declares three confirmatory treatment-versus-control comparisons for seven-day purchase conversion and sets family alpha to 0.05. The estimates are: design A, +0.10 percentage points with p = 0.031; design B, +0.31 points with p = 0.004; and design C, −0.02 points with p = 0.72. The team has checked allocation, exposure logging, the fixed analysis window, and the user-level denominator.
Order the p-values: 0.004, 0.031, and 0.72. Holm compares 0.004 with 0.05/3 = 0.0167 and rejects B’s null. It then compares 0.031 with 0.05/2 = 0.025 and stops, so A and C are not rejected. The correct conclusion is that B has statistically detectable evidence of a conversion difference under the planned family-wise rule; A is not a confirmed improvement, even though its unadjusted p-value is below 0.05. The team reports B’s absolute lift and confidence interval, expected incremental orders, refund and latency guardrails, and rollout constraints.
The adjusted p-values illustrate the same result. For B, 3 × 0.004 = 0.012. For A, the running maximum of 0.012 and 2 × 0.031 = 0.062 is 0.062. For C it is 0.72. At alpha 0.05 only B passes. The effect size remains central: if B’s interval is +0.09 to +0.53 points, the business case may be strong; a barely positive effect can still be rejected for operational reasons.
Risks and common mistakes
- Using a convenient subset of tests. The family must include the comparisons that were candidates for a claim, including unfavorable ones.
- Stopping incorrectly. After the first ordered p-value fails its threshold, later hypotheses are not rejected, even if a later p-value would pass its looser boundary.
- Correcting invalid p-values. Multiplicity adjustment cannot rescue peeking, bad denominators, or broken randomization.
- Calling non-rejection “no effect.” Review the confidence interval and minimum important effect; the study may simply lack precision.
- Ignoring selection of a winner. Picking the largest observed lift from many variants needs a preplanned comparison and cautious effect estimation.
- Equating error control with product safety. Practical impact, guardrails, and implementation review remain necessary.
Frequently asked questions about the Holm–Bonferroni correction
Is Holm–Bonferroni the same as Bonferroni?
No. Both control family-wise error, but Bonferroni uses alpha divided by the total number of tests for every hypothesis. Holm tests ordered p-values sequentially and is never less powerful.
Does Holm require independent tests?
No. Its family-wise error guarantee holds under arbitrary dependence when the individual p-values are valid. Correlation still matters for interpretation and should be documented.
Can Holm adjust confidence intervals?
Yes. Simultaneous confidence intervals can be constructed to match a family-wise procedure, although software support varies. Reporting estimates and appropriately adjusted intervals is clearer than reporting only adjusted p-values.
Should I use Holm for many exploratory metrics?
It can be used, but it may be overly stringent when the purpose is discovery and planned validation follows. Consider false discovery rate control or a hierarchy, while clearly labeling discoveries.
What is the difference between Holm and Hochberg?
Holm is a step-down method with broad dependence validity. Hochberg is a more powerful step-up approach under stronger dependence conditions. Choose based on the required guarantee, not merely the smallest adjusted p-value.
Summary
The Holm–Bonferroni correction controls the chance of one or more false positive claims across a defined family by testing ordered p-values from the smallest upward with progressively less strict thresholds. It is a strong default for a limited set of consequential A/B-test comparisons when the family, valid p-values, estimand, and business decision were defined in advance.
Sources
- Holm, S. (1979), A Simple Sequentially Rejective Multiple Test Procedure.
- NIST/SEMATECH e-Handbook of Statistical Methods, Multiple Comparisons.
- Hochberg and Tamhane, Multiple Comparison Procedures.