Multiple Comparisons in A/B Tests — Without Overcorrecting

You ship a checkout change. The dashboard lights up: 3 of 18 metrics are green at \( p < 0.05 \). Someone wants to ship. Someone else wants Bonferroni. Both instincts are incomplete.

Multiple comparisons are real — if you test enough null effects, some will look “significant” by chance. Classical fixes (Bonferroni, FDR) exist for a reason. But product A/B testing is not a Phase III drug trial. The cost of a false green on a secondary metric is usually a tweak you can roll back — not irreversible clinical harm. Netflix made this point explicitly in their experimentation series: statistics inform decisions; they don’t replace judgment across a bundle of evidence.

This guide covers the math you should know, then the product-sensible way to apply it: correct inside a small family of decision metrics, treat the rest as diagnostic, and never ship on a single lonely green.

The multiple comparisons problem

Run one two-sided test at \( \alpha = 0.05 \) under a true null. Your false positive rate for that test is about 5%. Run \( m \) independent null tests at the same \( \alpha \), and the chance of at least one false positive climbs fast:

So if you stare at twenty uncorrected metrics and celebrate every \( p < 0.05 \), you are fishing. Netflix’s jelly-bean cartoon of this idea is the right mental model: test enough colors and one will “cause acne.”

What Bonferroni and FDR actually do

Bonferroni — control the family-wise error rate

For \( m \) tests in a family, Bonferroni uses a stricter threshold so that the probability of any false positive in the family stays near \( \alpha \):

It is simple and conservative. Power drops hard as \( m \) grows. Great when a single false claim is expensive. Painful when you apply it to every secondary chart on the page.

FDR — control the false discovery rate

Procedures like Benjamini–Hochberg control the expected share of false discoveries among the rejections you make — usually less harsh than Bonferroni when many effects are real:

FDR is a better default than Bonferroni for large exploratory screens. It is still a family procedure: you have to decide which set of tests belongs in the family.

IT products are not clinical trials

In medicine, a false positive can put patients on a useless or harmful treatment. Error control is rightly extreme. In product experimentation you are usually choosing between UI variants, ranking tweaks, pricing copy, or onboarding flows. Mistakes are reversible. Learning speed compounds. That changes the optimal trade-off between Type I errors and velocity.

Netflix’s public writing on decision-making pushes the same direction: fix a nominal false-positive rate for the tests you treat as confirmatory, report non-significant results too, and interpret metric movements as a story — do they line up with the hypothesized mechanism? — not as a scoreboard of greens. That framing fits most product orgs better than treating every dashboard cell like a clinical endpoint.

A practical policy: families, not global correction

You do not need one correction across every number the platform can compute. You need clear families:

• Decision family (small). Primary metric + a few pre-registered guardrails (e.g. conversion, revenue, refund rate). Apply Bonferroni or FDR inside this family if you will ship or kill based on them.
• Diagnostic metrics (larger). Funnel steps, engagement, segment cuts. Treat each as supportive context. Uncorrected \( p \)-values here are exploratory signals, not ship criteria.
• Debug / quality. Latency, errors, SRM. These are often go/no-go ops checks, not “wins.”

This is not the most academic stance. It matches how product decisions actually get made: one or two claims you are willing to defend, plus a coherent picture around them.

Example

Checkout experiment. Pre-register:

• Primary: purchase conversion
• Guardrails: revenue per visitor, payment-error rate

Correct those three as a family if you want a formal FWER/FDR story. Then look at “add-to-cart,” “time on page,” and “mobile vs desktop” without pretending each green is an independent product claim. If conversion is flat, revenue is flat, and one random segment is green — you do not ship the segment story.

Decide on the overall picture

A single green secondary metric is weak evidence. Stronger evidence looks like:

• The primary moves in the hypothesized direction with a meaningful effect size and a CI that excludes junk.
• Guardrails are not quietly on fire.
• Related diagnostics move in a way that matches the mechanism (e.g. fewer form fields → higher completion, not just “sessions up”).
• You are not discovering the win by scanning 30 cuts after the fact.

False positives still happen. That is fine if your process expects them: ship carefully, monitor, reverse when the story falls apart. The alternative — Bonferroni over 40 metrics — often yields perpetual “nothing is significant,” which is just another way to make bad decisions (never learning).

What not to do

• Bonferroni the whole dashboard. You will underpower the few metrics that matter.
• Ignore multiplicity entirely on the decision family. Three correlated “primaries” are still a family.
• Ship on one green with contradictory guardrails. The picture failed.
• Re-label metrics after peeking. That is the multiple comparisons problem wearing a product hat.
• Confuse “not significant after correction” with “no effect.” Underpowered tests stay silent; see also our notes on p-values and effect sizes.

Pre-launch checklist

• Write down the decision family (1 primary + ≤ few guardrails).
• Choose \( \alpha \) and whether that family uses Bonferroni, FDR, or a single primary with descriptive secondaries.
• List diagnostics you will look at — and state they are non-confirmatory.
• Define what “ship / iterate / kill” looks like as a pattern of results, not a single cell.
• Log the plan before traffic starts.

Summary

Testing twenty metrics without a plan will manufacture false wins. Bonferroni and FDR are the right tools when you need tight control inside a defined family. For most product work, that family should be small. Correct there. Read the rest as context. Ship when the overall evidence — primary, guardrails, and mechanism — agrees, not when one exploratory chart turns green.

That is less pure than a textbook chapter on multiplicity. It is closer to how high-velocity product orgs actually use experiments: statistics as decision support, not a substitute for judgment about reversible product changes.