Quick definition: Bayes’ theorem states how to update the probability of a hypothesis after observing evidence by combining the evidence’s likelihood with the hypothesis’s prior probability.
What is Bayes’ theorem?
Bayes’ theorem is the rule underlying Bayesian inference. It reverses conditional probabilities correctly: evidence that is common when a hypothesis is true is useful only after considering how common the hypothesis was before observing evidence and how often the evidence occurs otherwise. In experimentation, it transforms a prior distribution for an unknown conversion rate or treatment effect into a posterior distribution after outcomes arrive.
P(H | E) = P(E | H)P(H) / P(E). H is a hypothesis and E is evidence. For continuous parameters, probability masses become densities: p(θ | y) = p(y | θ)p(θ)/p(y).
The denominator, P(E), is the evidence’s overall probability across hypotheses. It normalizes the result so posterior probabilities sum to one. The theorem is an identity, but its application depends on defensible inputs. A wrong likelihood, a hidden population change, or an irrelevant prior gives a precise-looking but unhelpful posterior.
Prior odds, likelihood ratios, and posterior odds
posterior odds = prior odds × likelihood ratio. The likelihood ratio is P(E | H1) / P(E | H0).
This form shows why evidence and prior belief play separate roles. A likelihood ratio of 10 multiplies prior odds by 10; it does not mean a hypothesis has 90% probability. For estimation, the same update is performed across every plausible parameter value, creating a posterior curve rather than choosing only “true” or “false.”
Assumptions and careful use
Bayes’ theorem itself has no special A/B-test assumption, but the model supplying its terms does. A binomial conversion model assumes a defined binary outcome, correct denominator, and suitable independence at the analysis unit. Randomized users should generally be the rows; do not treat page views as independent evidence when assignment was by user. Cluster experiments require cluster-aware modeling.
Priors need a documented origin. Historical results are informative only when audience, product, outcome window, and measurement are comparable. Test sensitivity to a skeptical prior, a weak prior, and a plausible historical prior. The posterior should not flip from ship to do-not-ship merely because an unexamined prior was selected.
Bayes’ theorem in A/B testing
Use the theorem to update uncertainty about arm-level outcomes, then calculate quantities that match a decision: probability of positive lift, probability of clearing a minimum useful effect, probability of exceeding a harm threshold, and expected value. This is more useful than asking only which arm has the larger observed rate. Keep one primary metric and use guardrails as defined in the experiment plan; primary and guardrail metrics should not be swapped after seeing data.
Worked example: interpreting a diagnostic signal
Suppose a monitoring alert has 90% sensitivity for a genuine tracking break and a 5% false-alert rate. Before an alert, the team believes a break is present on 1% of releases. After an alert, Bayes’ theorem gives P(break | alert) = (0.90 × 0.01) / [(0.90 × 0.01) + (0.05 × 0.99)] = 0.154, or about 15%.
The alert is valuable—it raises concern from 1% to 15%—but it does not establish a break. The right action is investigation, not automatically discarding an experiment. This base-rate calculation is also why a low p-value, a model score, or a favorable A/B dashboard should not be translated directly into the probability that a product hypothesis is true.
Interpretation
State probabilities conditionally: “Under the stated prior and model, the posterior probability that lift exceeds 0.2 points is 94%.” Do not say “the data prove a 94% chance the feature will work.” The latter mixes the estimated experimental effect with implementation, traffic, novelty, and rollout uncertainty.
Risks and limitations
- Base rates are easily omitted.
- Priors can dominate sparse evidence.
- Likelihoods can hide dependence or measurement error.
- Posterior results are conditional on assumptions, not assumption-free facts.
- Many selected variants require a selection-aware model or confirmation.
Common mistakes
- Confusing P(E|H) with P(H|E).
- Treating a p-value as P(null hypothesis | data).
- Using an undocumented prior.
- Updating on immature or duplicated outcomes.
- Ignoring sample ratio mismatch and calling the posterior causal.
Frequently asked questions about Bayes’ theorem
Does Bayes’ theorem require subjective beliefs?
It requires a prior distribution. That can be weakly informative or evidence-based, but it should be visible and sensitivity-tested.
Is it only for binary hypotheses?
No. Its common modern use updates continuous parameters and full probability models.
Does it replace randomization?
No. Randomization is what supports causal comparison; Bayes’ theorem quantifies uncertainty under a model.
Can it be used with sequential monitoring?
Yes, but decision actions, outcome maturity, and optional adaptation should be planned and simulated.
What does a posterior probability mean?
It is probability conditional on the stated prior, likelihood, data, and model—not a guarantee about all future contexts.
Summary
Bayes’ theorem converts prior uncertainty and observed evidence into posterior uncertainty. It supports transparent A/B-test decisions when the data-generating model, prior, and business thresholds are explicit and validated.
Sources
- Stanford Encyclopedia of Philosophy: Bayes’ Theorem
- Gelman et al., Bayesian Data Analysis
- NIST: Bayesian Statistics