Statistics·Glossary term

Bonferroni Correction

Bonferroni Correction A/B testing Reference guide

Bonferroni Correction is a concept used in statistical tests & methods.

Quick definition: The Bonferroni correction controls the chance of at least one false positive across a predefined family of tests by dividing the family-wise significance level by the number of tests.

What is the Bonferroni correction?

When a team tests many hypotheses at a 5% significance level, false positives accumulate. The Bonferroni correction protects the family-wise error rate (FWER): the probability of making one or more false rejections in a defined family. For m tests and family-wise alpha α, test each individual hypothesis at α/m, or equivalently multiply each p-value by m and compare it with α.

αper test = αfamily / m. Reject Hi when pi ≤ α/m.

The guarantee follows from the union bound, so it does not require independent tests. That robustness makes Bonferroni attractive, but it is conservative when tests are correlated. Conservatism reduces false positives at the cost of power and can make useful effects harder to detect.

Define the family before calculating

The hard part is not division; it is deciding what claims form one family. A family may be every treatment-versus-control comparison for a single primary metric, every primary endpoint in a release decision, or all claims in a report. It should match the set of conclusions for which a false positive would be misleading. Do not retroactively define a narrow family around the result that happened to be significant.

Bonferroni adjustment can be applied to p-values or intervals. A two-sided family-wise 95% confidence interval for each of m effects uses individual coverage 1 − 0.05/m. This produces wider intervals than unadjusted 95% intervals, reflecting that they jointly protect against a false claim.

A/B and A/B/n application

Suppose a team tests four new checkout variants against one control on one primary conversion metric. There are four confirmatory comparisons. With family alpha 0.05, Bonferroni uses 0.0125 per comparison. This prevents the team from declaring a winner merely because one of four noisy estimates happened to have p below 0.05. It does not address metrics, segments, or interim looks unless those are included in the chosen family.

For all pairwise comparisons, the number grows rapidly: five arms have ten pairs. If only control comparisons matter, do not waste power on treatment-versus-treatment claims. Plan contrasts to match the product decision and see multiple comparisons in A/B testing for broader governance.

Worked example: three variants

An experiment compares A, B, and C with control on activation. The predeclared family contains three treatment-control tests and has α = 0.05. Their unadjusted p-values are 0.011, 0.019, and 0.041. Bonferroni’s per-test threshold is 0.05/3 = 0.0167. Only A is significant after adjustment. Its adjusted p-value is min(3 × 0.011, 1) = 0.033; B’s is 0.057 and C’s is 0.123.

The correct conclusion is not that B and C have no effect. Their intervals may still include worthwhile lift. The team should report absolute estimates, adjusted intervals, guardrails, and the original decision objective before choosing a rollout.

Interpretation

A Bonferroni-significant result controls the long-run probability of any false rejection in the specified family at the stated alpha, assuming valid underlying tests. It does not quantify effect size, commercial value, or the probability a variant is best. A non-significant adjusted result means the data did not meet the stricter evidence rule; it is not proof of equivalence.

Risks and limitations

  • It can be unnecessarily conservative for correlated outcomes or many tests.
  • It needs a defensible, predeclared family.
  • It does not repair invalid randomization, metric changes, or SRM.
  • It may encourage teams to test too many variants with insufficient sample size.

Common mistakes

  • Dividing alpha by every dashboard metric, including diagnostics that make no decision claim.
  • Adjusting only the p-values that are below 0.05.
  • Calling an unadjusted winner confirmatory after examining many variants.
  • Forgetting that two-sided intervals also need adjustment.
  • Using correction as an excuse to omit a primary metric and analysis plan.

Frequently asked questions about Bonferroni correction

When should I use Bonferroni?

Use it when strong FWER control is required and the family is small or simplicity matters.

Is Bonferroni valid with dependent tests?

Yes, its FWER bound does not require independence, though the method may be conservative.

What is Holm’s method?

Holm is a step-down Bonferroni procedure that controls FWER and is uniformly at least as powerful.

Does it apply to sequential looks?

Not by itself. Repeated interim looks need a sequential method such as an alpha- or beta-spending design.

Can I correct many segments?

You can, but broad segment exploration is often better labeled exploratory or followed by confirmation.

Summary

Bonferroni is a simple, reliable way to limit false positives across a planned family of A/B-test claims. Its price is lower power, so define the decision family carefully and pair it with adequate sample-size planning.

Sources