Quick definition: The family-wise error rate (FWER) is the probability of making one or more false-positive conclusions within a defined family of hypothesis tests. FWER control is used when any false claim in that family would be costly or unacceptable.
What is the family-wise error rate?
A single test at α = 0.05 has a 5% false-positive rate under its null and assumptions. With several tests, the probability of at least one false positive is larger. If five independent null hypotheses are each tested at 0.05, the chance of one or more false rejections is 1 − (1 − 0.05)5 ≈ 22.6%. FWER is this “at least one” error probability for the chosen family.
Controlling FWER at 5% means a procedure is designed so that, over repeated uses under stated conditions, the probability of making any false rejection in the family is no more than 5%. It is a strict criterion. It does not mean every individual reported effect has a 95% probability of being true, and it does not protect against biased data, optional stopping, or ambiguous metrics. Its practical value comes from limiting false launch claims when one error can have a substantial cost.
Methods, formulas, and assumptions
The simplest procedure is Bonferroni correction. For m hypotheses and target FWER α, test each p-value against α/m, or multiply each raw p-value by m and cap at one. With five tests and α = 0.05, each test uses 0.01. Bonferroni works under arbitrary dependence but can be conservative, especially when hypotheses are correlated.
Holm’s step-down method controls FWER while usually offering more power. Sort p-values from smallest to largest. Compare the smallest to α/m, the next to α/(m−1), and continue until a p-value fails; do not reject that hypothesis or any larger p-values. Hochberg’s step-up procedure can be more powerful under suitable dependence conditions. For all pairwise mean comparisons, Tukey-style methods; for many treatments versus one control, Dunnett’s method; and for planned contrasts, resampling or closed-testing approaches may be more efficient than a universal correction.
Every method needs a declared family. It might be all three treatment-versus-control comparisons in an A/B/n experiment, all primary endpoints in a regulated decision, or the primary endpoint assessed at multiple interim looks. It is not necessarily every number a company has ever computed, but it cannot be selected after favorable p-values are seen. The tests also must be valid individually: correction cannot fix dependence created by repeated events, a wrong standard error, or a data-quality issue such as sample ratio mismatch.
FWER in A/B and A/B/n testing
Use FWER control when a team needs a strong confirmatory statement, especially when a release will be expensive, hard to reverse, or associated with user risk. In a four-arm test, testing each of three variants against control at unadjusted 0.05 gives three opportunities for a false winner. Dunnett’s procedure is efficient when only those control comparisons matter. If the team also wants to compare every pair of variants, the family is larger and a method such as Tukey or a planned contrast strategy may fit.
Multiplicity extends beyond variants. Testing many primary-like metrics, running many segment analyses, and checking a fixed-horizon p-value repeatedly all create error families. A good experiment plan names one primary metric, labels guardrails and diagnostics, defines comparisons, and specifies a stopping method. The distinction and governance rationale appear in primary and guardrail metrics; detailed trade-offs are covered by multiple comparisons in A/B testing.
FWER control is not always the best objective. For exploratory analysis across hundreds of metrics, it can yield too few discoveries; false discovery rate control may better match a workflow that confirms selected signals. The right choice follows the cost of a false positive, the number of decisions, and the planned validation path—not a preference for whichever adjustment produces a desired result.
Worked example: three variants versus control
A retailer tests three checkout variants against control with one primary conversion metric. The predeclared family consists of exactly the three treatment-control contrasts, and the team wants strong control at α = 0.05. Raw p-values are 0.012, 0.028, and 0.041. Under simple Bonferroni, each must be below 0.05/3 = 0.0167; only the first contrast is significant. Adjusted p-values are 0.036, 0.084, and 0.123.
With Holm, sort 0.012, 0.028, 0.041. The first is below 0.0167, but the second exceeds 0.025, so reject only the first and stop. A Dunnett adjustment may produce a different, often less conservative result because all comparisons share control and are correlated; use the calculated output from the prespecified procedure rather than applying an unrelated threshold. The team then examines absolute lift, intervals, payment failures, and latency before selecting a rollout. “Adjusted significant” is evidence for the contrast, not a substitute for a decision threshold.
Interpretation and workflow
- Define the confirmatory question, complete hypothesis family, and cost of one false claim.
- Choose the adjustment matched to the contrast structure and dependence assumptions before results are viewed.
- Validate assignment, exposure, denominators, data maturity, and individual-test assumptions.
- Report raw p-values, adjusted p-values or simultaneous intervals, effect estimates, and the family definition.
- Make release decisions using practical benefit, guardrails, and reversibility in addition to FWER-controlled evidence.
Simultaneous confidence intervals are often clearer than adjusted p-values. A family of 95% simultaneous intervals is constructed so that all intervals cover their respective parameters together with the stated probability under the procedure. It lets stakeholders see which effects remain compatible with meaningful benefit or harm. A result failing FWER correction is not necessarily unimportant; it may be uncertain or require a confirmatory follow-up.
Limitations and common mistakes
- Correcting a selectively defined family. The guarantee applies to the family actually specified, not a favorable subset.
- Using Bonferroni by reflex. It is valid but may sacrifice power when Dunnett, Holm, or a hierarchical plan fits the decision better.
- Calling an adjusted p-value a probability of truth. It remains a procedure-based quantity, not a posterior probability.
- Ignoring sequential looks. Variant correction does not automatically correct repeated monitoring.
- Adjusting away poor planning. A weak primary metric and many unplanned comparisons remain poor experiment design.
- Equating no rejection with equivalence. Inspect simultaneous intervals and prespecified practical-equivalence criteria.
Frequently asked questions about family-wise error rate
What is the difference between FWER and per-comparison error?
Per-comparison error concerns each test separately. FWER concerns the chance that at least one test in the full family is a false positive.
When is Bonferroni appropriate?
It is a simple valid choice under arbitrary dependence, especially for a small family. Consider more targeted methods when the comparison structure is known.
Does FWER control protect guardrails?
Only if they are included in a specified testing and decision policy. Some guardrails use one-sided harm thresholds, sequential monitoring, or hard operational limits instead.
Can I change the family after results are known?
Not while retaining the original confirmatory guarantee. You may describe a new analysis as exploratory and confirm it independently.
Is FWER always preferable to FDR?
No. FWER is more stringent and suited to high-cost false claims. FDR often better serves broad discovery where validation follows selection.
Summary
Family-wise error rate control limits the chance of any false positive in a defined collection of tests. It gives strong protection for confirmatory A/B and A/B/n decisions when the family, adjustment, stopping behavior, and data quality are planned rather than chosen after outcomes appear.
Sources
- Holm, A Simple Sequentially Rejective Multiple Test Procedure
- Dunnett, A Multiple Comparison Procedure for Comparing Several Treatments with a Control
- NIST/SEMATECH e-Handbook: Multiple Comparisons