Confidence Intervals in A/B Testing — Why They Beat a Naked p-Value

Dashboard says \( p = 0.03 \). The room hears “we won.” Nobody asks: won by how much? That missing question is why confidence intervals matter more than a naked p-value for product decisions.

Netflix has been explicit about this shift in how they think about experimentation at scale. In their return-aware experimentation writing, they frame A/B tests less as pure scientific hypothesis tests and more as tools for decisions that lift business metrics over time. Scientists obsess over keeping false discoveries out of the literature; businesses may tolerate more false greens if the portfolio of launches still compounds returns. Either way, you cannot optimize returns if all you know is “different from zero.” You need a range for the lift.

This guide explains what a confidence interval (CI) is, how to read one on an A/B result, why it dominates “\( p < 0.05 \)” alone — and how that fits a return-aware mindset, including the winner’s curse Netflix warns about when you pick winners by significance and then trust their point estimates.

What a confidence interval is

A \( (1-\alpha) \) confidence interval is a range of plausible values for the true treatment effect, constructed so that — in repeated experiments under the same design — about \( (1-\alpha)\times 100\% \) of such intervals cover the true effect.

For proportions, the same idea applies with the usual SE for a difference of rates. For log-transformed revenue, build the CI on the log scale and convert endpoints with \( e^{L}-1 \) and \( e^{U}-1 \) if you communicate percent lifts (see our log-transform guide).

How to read a CI on an experiment

Look at three things, in order:

• Does it include zero? If yes, the data are compatible with “no effect” at your confidence level — same information as a non-significant two-sided test at that \( \alpha \).
• Where does the interval sit relative to what you care about? A CI of \( [+0.1\%,\; +0.3\%] \) on conversion may be “significant” and still too small to bother shipping. A CI of \( [-0.2\%,\; +4\%] \) includes zero and a huge upside — underpowered, not “proof of nothing.”
• How wide is it? Width is uncertainty. Wide intervals mean you cannot pin down returns yet — exactly the resource question Netflix discusses when trading many weak tests vs fewer precise ones.

A simple decision map

• CI entirely above your minimum useful lift → strong case to ship (guardrails permitting).
• CI entirely below zero or below “worth it” → kill or redesign.
• CI covers both junk and meaningful upside → need more sample, a better design, or a clearer primary metric — not a vibe-based ship.
• Point estimate looks great, CI barely clears zero → fragile win; treat magnitude as uncertain.

Why this beats a naked p-value

A p-value answers one narrow question: how surprising is this data (or more extreme) if the true effect were exactly zero? It does not say how large the effect is, whether it clears your MDE, or what return to expect if you launch.

Two experiments can share \( p = 0.04 \) and tell opposite product stories:

• Lift \( +0.2\% \) with CI \( [+0.01\%,\; +0.39\%] \) — significant, tiny.
• Lift \( +3\% \) with CI \( [+0.1\%,\; +5.9\%] \) — significant, maybe material.

Same “win” flag. Different returns. If your mental model of experimentation is return-aware — cumulative lift across many noisy tests — the interval (and honest evaluation of launched arms) is the object you actually need. The p-value is a side check that zero is an awkward explanation, not a substitute for magnitude.

Return-aware angle: the winner’s curse

Netflix’s return-aware work highlights a second reason magnitude estimates matter — and why “pick the arm with the largest significant effect and add up those lifts” overstates portfolio returns. When you select winners because they looked good on the same data you use to measure them, those point estimates are optimistically biased. That is the winner’s curse, and it bites hardest when effects are small and noisy — the usual digital case.

Practically:

• Do not treat the winning arm’s raw \( \widehat{\Delta} \) as the expected lift after launch without skepticism.
• Prefer reading the full CI, not just the point that cleared \( p < 0.05 \).
• For serious return accounting across many tests, Netflix’s antidote is to split selection data from evaluation data (or otherwise debias). Even without that machinery, knowing the CI is wide should lower your confidence in “we’ll get the full +4%.”

CI, α, and “significance”

For the usual two-sided test, these are two views of the same information:

So “95% CI excludes zero” ↔ “significant at 5%.” The interval still wins for communication because it shows which non-zero values remain plausible. Netflix’s portfolio view also notes that when implementation cost is low, organizations may deliberately run more low-powered tests and relax thresholds — again, a conversation about expected returns and uncertainty width, not a sacred \( p < 0.05 \) tattoo.

Practical tips

• Always show CI next to the lift in reviews. Hide the p if you must; never hide the interval.
• Pre-define “useful” (MDE or minimum absolute lift) so you can compare CI endpoints to a line, not to vibes.
• On logs / ratios, build intervals on the modeled scale, then transform for humans.
• Segment CIs without a plan reintroduce multiple comparisons — treat them as diagnostics unless pre-registered.
• If the CI is huge, the decision is often “keep running / redesign,” not “insignificant = no effect.”

Summary

A confidence interval is the practical answer to “how big might this change be?” A p-value only asks whether zero is awkward. Product experimentation — especially in a return-aware framing like Netflix’s — cares about cumulative lift under noise, resource limits, and selection bias. Read whether the CI clears your useful threshold, how wide it is, and whether your “win” is a fragile point estimate. Ship on magnitude and coherence, not on a lonely green p.