Quick definition: The false discovery rate (FDR) is the expected proportion of false positives among all findings declared significant. It is a multiple-testing error criterion designed for settings where identifying promising signals is valuable and some follow-up validation is expected.
What is the false discovery rate?
When many hypotheses are tested, a per-test α of 0.05 does not mean the collection of reported “wins” is 95% reliable. The false discovery rate addresses a different question from family-wise control: among the discoveries a procedure reports over repeated use, what proportion are expected to be false? If V is the number of false rejections and R the total number of rejections, FDR is E[V / max(R,1)].
An FDR target of 5% does not promise that exactly 5% of findings in one dashboard are false, nor that each individual result has a 5% chance of being false. It is an average property of a specified procedure, hypothesis family, dependence assumptions, and data-generating conditions. It is therefore valuable for discovery-oriented analysis but must not be oversold as a per-experiment launch guarantee.
Benjamini–Hochberg procedure and assumptions
The standard Benjamini–Hochberg (BH) procedure controls FDR at level q under independent p-values and under several positive-dependence conditions. For m p-values, sort them: p(1) ≤ … ≤ p(m). Find the largest rank k for which p(k) ≤ (k/m)q. Reject the hypotheses ranked 1 through k. BH becomes less stringent as more strong signals are present, which gives it higher discovery power than procedures that protect against even one false positive.
Under arbitrary dependence, the Benjamini–Yekutieli adjustment is more conservative. Permutation-based approaches, hierarchical testing, and model-based methods may better reflect a platform’s correlated metrics and variants. Do not claim FDR control merely because a dashboard labels p-values “adjusted.” Record the exact method, q level, full set of p-values, hypotheses included, directional choices, and the dependence rationale.
FDR control assumes each p-value is valid for its own null. It cannot cure p-hacking, optional stopping, changing the metric definition, broken randomization, or a p-value calculated with the wrong analysis unit. In experiments, validate assignment and measurement before applying multiplicity adjustment; sample ratio mismatch is an operational warning that precedes formal inference.
FDR in experimentation
FDR is a sensible choice for broad discovery programs: screening many product ideas, analyzing a large set of predeclared mechanism metrics, or generating hypotheses across many markets where selected findings will be confirmed. It is less suitable when one false positive has a serious user, legal, safety, or financial consequence. In that case, family-wise error rate control, a gated hierarchy, or independent replication may be preferable.
Define the family before examining outcomes. A family could be all 40 predeclared secondary engagement metrics for one experiment, all treatment-versus-control comparisons in a release batch, or all hypotheses in a research screen. It should not be narrowed after results appear to include only favorable metrics. Keep the primary metric separate and give it a protected confirmatory rule; the principles of primary and guardrail metrics make this governance explicit.
FDR does not make every adjusted discovery a production winner. Report effect estimates, uncertainty, quality checks, and practical thresholds. An FDR-adjusted secondary metric may reveal a useful mechanism, while the action still depends on magnitude, guardrail behavior, external validity, and whether the feature was selected from a large candidate pool.
Worked example: secondary-metric screen
A team predeclares 10 secondary metrics for an onboarding experiment and chooses BH FDR q = 0.10. Their p-values in ascending order are 0.003, 0.009, 0.018, 0.041, 0.062, 0.18, 0.31, 0.45, 0.67, and 0.88. The BH thresholds are 0.01, 0.02, 0.03, 0.04, 0.05, and so on. The first three p-values satisfy their thresholds; 0.041 exceeds 0.04. The largest qualifying rank is therefore three, and the procedure calls the first three metrics discoveries.
The interpretation is not that all three effects are real. Over repeated comparable families, the procedure targets an average false-discovery proportion no greater than 10% under its conditions. The team presents the three effect estimates and intervals, investigates whether they are coherent with the product mechanism, and reserves a consequential metric claim for a confirmation test. It does not retroactively exclude the seven unfavorable metrics from the family.
Interpretation workflow
- Classify hypotheses as confirmatory, guardrail, diagnostic, or discovery-oriented before analysis.
- Define the complete FDR family and choose q based on the cost of false discoveries and planned validation.
- Ensure every constituent p-value follows a valid, documented design and stopping rule.
- Apply BH or another specified procedure to the full family and retain the adjusted results.
- Use discoveries as evidence for measured follow-up, not as a substitute for practical and causal review.
FDR is particularly helpful when a strict “no false positives anywhere” criterion would identify almost nothing. That trade-off is intentional. If stakeholders hear “10% FDR” as “this one change has only a 10% error chance,” correct the interpretation: it is a property of the pipeline’s reported set, not a posterior probability for a selected result.
Limitations and common mistakes
- Using FDR for a safety-critical release. Expected false findings may be unacceptable when even one error is costly.
- Defining the family after selection. Excluding unflattering tests defeats the guarantee.
- Ignoring dependence assumptions. Use an appropriate method or conservative alternative for complex correlation.
- Reporting adjusted p-values without effects. Multiplicity control says nothing about business magnitude.
- Combining exploratory and confirmatory claims. Label discoveries honestly and confirm high-impact claims.
- Applying BH to invalid p-values. Adjustment cannot fix repeated peeking or data-quality bias.
FDR can also be misunderstood after a discovery set has been selected. A result that survives BH may have an exaggerated observed effect because it was selected partly for looking favorable, particularly when the screen has limited power. Estimate magnitudes with intervals, use shrinkage or replication where appropriate, and avoid ranking a handful of adjusted discoveries as though their point estimates were equally stable. If a product decision depends on one selected metric, move it into a fresh, confirmatory experiment rather than recycling the screening p-value.
Frequently asked questions about false discovery rate
How does FDR differ from FWER?
FDR limits the expected proportion of false discoveries; FWER limits the chance of at least one false positive. FDR is generally more powerful but allows some false discoveries in expectation.
Is a q-value the same as an FDR-adjusted p-value?
Terminology varies. A q-value often represents the minimum FDR level at which a finding is called significant; software may use related adjusted p-value labels. Document the exact method.
Can I use BH for A/B/n variants?
Yes for a defined family of valid comparisons, but decide whether all pairwise comparisons or only treatment-versus-control contrasts belong in the family and account for selection of the winning variant.
Should primary metrics use FDR?
Usually a primary metric has its own confirmatory alpha and decision rule. FDR is more natural for a planned collection of secondary or discovery hypotheses.
Does FDR control practical false wins?
No. It concerns statistical false rejections, not effects too small to matter, implementation defects, or future-population failure.
Summary
False discovery rate control limits the expected share of false positives in a defined set of discoveries. It is useful for high-volume hypothesis generation when the family is fixed, p-values are valid, dependence is considered, and important discoveries receive practical review or confirmation.
Sources
- Benjamini and Hochberg, Controlling the False Discovery Rate
- Benjamini and Yekutieli, FDR Control Under Dependency
- NIST/SEMATECH e-Handbook: Multiple Comparisons