Quick definition: Minimum detectable effect (MDE) is the smallest true effect an experiment is designed to detect with its planned sample size, significance level, and statistical power.
What is minimum detectable effect?
MDE translates a vague wish to “detect a difference” into a planning quantity. An experiment cannot reliably distinguish every tiny change from random variation. Before launch, a team chooses the smallest effect worth detecting, estimates baseline variation and traffic, and calculates the sample needed to achieve a chosen probability of detecting that effect if it is real. Conversely, for a fixed sample, it calculates the MDE the test can resolve.
MDE is not the expected effect, the observed effect, or the minimum business value. The expected effect is a forecast; the observed effect is the estimate after data arrives. The minimum practical effect is the smallest improvement that would justify implementation. A useful MDE is usually at or below that practical threshold, but they answer different questions. Designing for a much smaller MDE than is decision-relevant can consume excessive traffic; designing for a larger one means meaningful improvements may be missed.
MDE also differs from statistical significance. A significant result can be too small to matter, and a non-significant result can be too imprecise to rule out an important benefit. Confidence intervals show the range of effects compatible with the data; this guide explains their interpretation.
MDE formulas and inputs
For an approximately equal-sized two-arm test of a binary rate, a common planning approximation is:
MDE ≈ (z(1−α/2) + z(power)) × √(2p(1−p) / n per arm)
Here p is the baseline conversion rate, n is planned units per arm, α is the false-positive rate, and power is commonly 80% or 90%. The exact calculation should use the selected test method, allocation ratio, variance model, and whether a one- or two-sided hypothesis is justified. For a mean metric, replace binomial variance with an estimated standard deviation; for skewed revenue, use a method appropriate to its distribution.
Relative MDE is absolute MDE / baseline rate × 100. At a 2% baseline, an absolute 0.2-point MDE equals 10% relative lift. Always report both units. Relative language can make a small absolute change sound large, while absolute points show the operational scale.
MDE in A/B testing
Choose MDE before observing results, alongside the primary metric, unit of randomization, baseline, alpha, power, allocation, and duration. Start with the decision: what improvement would repay engineering, operational, and risk costs? Then check whether traffic and time can support detecting it. Our sample-size guide covers the relationship between these choices.
Do not set MDE from a convenient calculator output or a past win alone. A change to a critical checkout flow may warrant detecting a small effect; a low-cost copy test may need a larger effect to matter. Account for expected exclusion, incomplete exposure, clustering, and multiple variants. An A/B/n test divides traffic and normally needs more observations per comparison. Repeated looks without a valid sequential design also change error control; see test-duration guidance.
MDE is a design promise, not a stopping rule. Ending early because the estimate crosses a desired threshold invalidates a fixed-horizon plan. Use the preplanned sample or a valid sequential method. At the end, judge the estimate and interval against practical value and guardrails, not merely whether the observed difference exceeds the original MDE.
Worked scenario
A signup page converts 5.0% of eligible users. The team considers +0.5 percentage points, from 5.0% to 5.5%, the smallest valuable improvement. It chooses a two-sided 5% alpha and 80% power. A calculator using its planned two-proportion method estimates roughly 31,000 eligible users per arm; exact requirements vary with the method and continuity assumptions.
The team has 4,500 eligible users per day and expects a 50/50 allocation. It therefore plans about 14 days for raw traffic, then adds time to cover weekly patterns and conversion maturation. If it can only run for five days, it must accept a larger MDE, change the design, or postpone the test. It should not run an underpowered study and interpret an inconclusive result as evidence that no meaningful effect exists.
Data-quality limitations
MDE calculations are only as sound as their inputs. An inflated baseline, outdated variance estimate, or wrong eligibility count produces an unrealistic duration. Estimate baseline from recent comparable traffic, document exclusions, and allow for bots, consent loss, outages, and expected attrition. Confirm that the numerator and denominator use the same unit and that the success event is stable.
Check assignment and exposure during the test. A sample-ratio mismatch can reduce effective power or indicate a more serious implementation fault; use SRM diagnostics. Clustered assignment, repeat measures, and correlated users reduce independent information, so treating every event as an independent user can make the claimed MDE falsely optimistic.
Common MDE mistakes
- Confusing MDE with expected lift: it describes detectable precision, not a forecast.
- Choosing a relative MDE only: always inspect absolute percentage points or currency.
- Ignoring the decision threshold: a detectable effect can still be commercially trivial.
- Using total sessions when users are randomized: power follows the analysis unit.
- Stopping because results look promising: fixed-horizon calculations assume the planned endpoint.
- Reading non-significance as no effect: wide intervals may still include important effects.
Frequently asked questions
Is a smaller MDE always better?
No. It requires more sample and may target effects too small to justify action. Choose the smallest decision-relevant effect.
Can MDE change after a test starts?
The original design should remain the basis for confirmatory inference. Material operational changes should be documented and may require replanning rather than quietly changing the target.
How does power affect MDE?
Higher power means a lower chance of missing the target effect, but it requires more observations for the same MDE.
What MDE should a revenue test use?
Use a currency amount per assigned unit or a profit-based threshold tied to implementation cost, risk, and expected scale.
Summary
MDE is the planned sensitivity of an experiment. Set it before launch from the smallest meaningful change, then calculate sample size using a credible baseline, variance, alpha, power, and analysis unit. Afterward, use intervals and guardrails—not MDE alone—to make the decision.
Sources
- NIST/SEMATECH: Sample sizes for tests
- NIST: Confidence intervals
- CDC: Statistical power and sample size