Quick definition: Effect size quantifies the magnitude of a difference, association, or treatment effect in units that make practical interpretation possible.
What is effect size?
Effect size answers “how much?” rather than merely “is there evidence of a difference?” In an A/B test it can be the raw change in conversion rate, the revenue-per-user difference, a relative risk, an odds ratio, or a standardized mean difference. The appropriate measure depends on the outcome and decision. A tiny effect can be statistically detectable with enough traffic, while a meaningful estimate can remain uncertain in a small test.
Effect size is not a p-value. A p-value describes how surprising data would be under a null model; it does not report the size or business value of treatment’s estimated impact. It is also not synonymous with relative lift. A 1-point conversion change and a 10% relative lift may describe the same result, but they answer different communication needs.
Effect-size definitions and denominators
For a binary outcome, a useful absolute effect is the risk difference:
risk difference = p
For relative comparison, use risk ratio = p. For a continuous outcome, a raw mean difference is mean. A standardized mean difference such as Cohen’s d divides that gap by a chosen standard deviation, making scales comparable but often making operational interpretation less direct.
Use the same outcome, unit, eligibility rule, and window in each arm. For example, paid orders per assigned visitor estimates a different effect from orders per purchaser. In an experiment, the former is generally better aligned with the causal decision because it includes all eligible randomized users.
Effect size in A/B testing
Choose the effect measure and the minimum practically important effect before launch. A checkout team may require at least +0.3 percentage points in paid conversion, provided refunds do not rise. A performance team may require a meaningful reduction in a high percentile of load time. Sample-size planning uses the baseline variability and smallest worth-detecting effect; see how to calculate sample size.
Always report uncertainty. A +0.5-point point estimate with an interval spanning −0.2 to +1.2 points does not establish a precise benefit. Confidence intervals help communicate both plausible direction and magnitude. Judge the estimate against a decision threshold, implementation cost, and guardrails, not against a significance label alone.
Worked effect-size example
A treatment produces 560 paid conversions from 10,000 eligible users; control produces 500 from 10,000. The absolute effect is 5.6% − 5.0% = +0.6 percentage points. The relative risk is 5.6% / 5.0% = 1.12, or a 12% relative lift. At 100,000 comparable users, the point estimate corresponds to 600 additional paid conversions. The team then evaluates its interval, payment failures, refund rate, and contribution margin before acting.
Data-quality caveats
An effect size is only as valid as the measured outcome. Confirm random assignment, exposure, event completeness, identity rules, and matching denominators. Investigate unexpected allocation before analysis using sample-ratio mismatch diagnostics. For revenue, a handful of extreme values can dominate the mean; pre-specify robust or transformed analyses rather than changing methods after results.
Practical interpretation
Prefer an effect size in units stakeholders can use: percentage points, dollars per assigned user, milliseconds, or prevented failures per thousand requests. Relative measures provide scale but require the baseline. Standardized measures aid comparison across variables, yet may obscure the actual customer consequence.
Start with the decision, not the statistical label. A product owner deciding whether to maintain a more complex checkout needs an estimate of incremental paid orders, their expected value, the implementation cost, and the plausible downside. A service owner deciding whether to deploy a slower personalization model needs the expected relevance benefit alongside the latency cost. An effect-size report should make that trade-off inspectable.
For ratio metrics, inspect the components as well as the result. A change in revenue per user might arise from conversion, order value, discounting, refunds, or a shift in user mix. A change in average time may conceal a worse tail experience. Predefine relevant decompositions as diagnostics so they explain an estimate rather than becoming a search for a favorable narrative.
Effect sizes also depend on the population to which they apply. An estimate among new mobile visitors can be valuable without implying the same result for returning desktop customers. State the target population and avoid generalizing beyond the eligibility criteria. If a segment effect would change the product decision, name it before launch and ensure adequate sample rather than relying on an unstable post-hoc comparison.
When outcomes mature slowly, distinguish observed effects from forecasts. An early subscription conversion estimate is not a measured lifetime-value effect. Keep a holdout or follow cohorts long enough for churn, renewal, and realized margin to appear. A leading effect can justify continued measurement, but it does not prove downstream value.
Common effect-size mistakes
- Reporting only a p-value: magnitude remains unknown.
- Confusing points with percent: the scale is misstated.
- Using post-treatment denominators: the estimand changes.
- Ignoring intervals: precision is overstated.
- Optimizing one effect alone: harms can offset benefit.
Finally, do not compare standardized effects mechanically across unrelated populations or metric definitions. A historical benchmark calculated with another variance estimate, window, or inclusion rule may not be comparable. Preserve raw counts and raw-unit effects so future readers can reconstruct what changed.
For launch decisions, record the estimated effect and interval at the time of decision, then compare it with later realized outcomes. This creates an evidence loop: teams learn whether their practical thresholds were sensible, whether early effects persist, and which metrics are reliable leading signals. It also discourages retrospective claims that a noisy estimate promised more certainty than it did.
Record any deviations from the planned analysis, including data exclusions and late metric changes. Transparent reporting lets reviewers separate a genuine treatment estimate from an analysis decision made after the outcome was known.
Frequently asked questions
Is effect size the same as significance?
No. Effect size is magnitude; significance concerns evidence under a chosen statistical procedure.
Which effect size should conversion tests use?
Usually report absolute percentage-point difference and relative change, with counts and an interval.
Can a small effect be important?
Yes, at high volume or where the outcome is valuable; practical importance depends on context.
Should I use Cohen’s d for every metric?
No. It suits certain continuous-outcome comparisons; raw effects are often clearer for product decisions.
Summary
Effect size quantifies the magnitude of an estimated difference. Define the estimand, units, denominator, and practical threshold before testing; report uncertainty and guardrails with the estimate.
Sources
- NIST/SEMATECH: Comparing process means
- CDC: Measures of effect
- American Statistical Association: P-values statement