Quick definition: Bayesian statistics combines a prior distribution with observed data through a probability model to produce a posterior distribution: a quantified description of uncertainty about unknown parameters after seeing the data.
What is Bayesian statistics?
Bayesian statistics treats an unknown quantity, such as an A/B test’s conversion-rate difference, as uncertain and represents that uncertainty with probability. Before data, the prior distribution encodes information or deliberately weak regularization. The likelihood describes how probable the observed data are under each possible parameter value. Bayes’ theorem updates the prior with the likelihood to produce the posterior. Decisions can then use posterior probabilities, credible intervals, expected loss, and predictive distributions.
This differs from frequentist inference, where parameters are fixed and probability describes repeated samples. Neither framing removes the need for a valid experiment. Randomization, correct exposure logging, a stable metric definition, and mature outcomes are prerequisites; a posterior cannot turn biased observational data into a causal answer.
Model, prior, likelihood, posterior
p(θ | data) = p(data | θ)p(θ) / p(data). Here θ is the unknown parameter, p(θ) the prior, p(data | θ) the likelihood, and p(θ | data) the posterior. The denominator normalizes the distribution.
For a binary conversion metric, a Beta prior is convenient. If θ is a rate and the prior is Beta(α, β), then observing x conversions in n eligible users produces Beta(α + x, β + n − x). α and β behave like prior pseudo-counts, but their influence and meaning must be explained in the metric’s context. For continuous outcomes, a normal or hierarchical model may be appropriate; for revenue, use a model that respects zeros and skew or a predeclared robust estimator.
A prior can be informative, weakly informative, skeptical, or based on historical experiments. “Noninformative” is not neutral in every parameterization. A weak prior still has consequences when data are sparse. Show sensitivity: rerun the decision under plausible alternative priors and report whether the action changes.
Assumptions and model checks
Posterior probabilities are conditional on the stated model and prior. A binomial model assumes each included unit has a binary outcome and, conditional on its rate, the observations behave independently. Users with multiple correlated sessions, cluster randomization, time trends, or interference violate a naive model. Model the dependence, aggregate at the randomization unit, or use cluster-aware methods.
Posterior predictive checks are essential: simulate outcomes from the fitted model and compare their distributions with observed rates, variance, tails, and important segments. A good-looking posterior from a misspecified likelihood is false precision. Check assignment balance and sample ratio mismatch before modeling treatment effects.
Bayesian A/B-testing application
A useful Bayesian decision begins with the business action, not a probability chosen for a dashboard. Define the estimand, minimum practical lift, maximum tolerable loss, outcome window, traffic cap, and loss function. For example, approve a rollout only when the posterior probability that lift exceeds 0.2 percentage points is at least 95%, the probability of a loss worse than 0.1 points is below 2%, and guardrails are acceptable.
Bayesian monitoring permits updating the posterior as data arrive, but that does not make every operational response harmless. Repeatedly shipping, stopping, switching metrics, or selecting segments after inspection changes the decision problem. Predefine monitoring cadence, actions, and practical thresholds; account for concurrent experiments and data maturity. A posterior probability is not a blanket guarantee of long-run false-positive frequency.
Worked example: conversion-rate decision
Control has 510 conversions from 10,000 users and treatment has 545 from 10,000. With independent Beta(1,1) priors, the posteriors are Beta(511,9491) and Beta(546,9456). Monte Carlo draws take one draw from each posterior and subtract control from treatment. Suppose 94% of draws are positive, 82% exceed a practical lift of 0.2 percentage points, and 4% show a loss exceeding 0.1 points.
The observed lift is 0.35 percentage points, but the stated rollout rule is not met because the probability of exceeding the practical threshold is only 82%, below 95%. “Treatment is probably better” and “we have enough evidence to take this action” are different claims. The team can continue to its traffic or time limit, decide that the expected value justifies a limited rollout, or stop according to a predeclared futility rule—while reporting which rule was used.
Interpreting posterior results
Given the model and prior, a 95% credible interval contains 95% posterior probability for θ. That direct wording is valid in Bayesian inference; it should not be used for a frequentist confidence interval. Report posterior median or mean, an interval, probabilities relative to business thresholds, arm-level metrics, the prior, likelihood, computation, and sensitivity analysis. Avoid presenting “chance to win” without defining win and loss.
Posterior predictive quantities are often more operational than parameter probabilities. A retailer can simulate future seven-day revenue under each arm, including uncertainty in its estimated rates and value distribution, then compare the probability that rollout value exceeds engineering and operational costs. Such forecasts must explicitly state what is held constant. They do not automatically include novelty decay, market changes, or the effects of expanding from the experimental population to all users.
Hierarchical Bayesian models can partially pool related variants, countries, or segments. Partial pooling reduces noisy extreme estimates toward a shared distribution when evidence is weak, while allowing well-supported differences to remain. This can be valuable for sparse segments, but it changes the estimand and requires checking whether the groups are plausibly exchangeable. Never present a pooled model as though it were a separate independent A/B test for every segment.
Risks and limitations
- Prior sensitivity: small samples can be materially driven by prior choices.
- Model misspecification: a convenient likelihood can overlook zeros, overdispersion, clusters, or temporal drift.
- Threshold theater: arbitrary posterior cutoffs can hide a decision’s economic trade-off.
- Selection: reporting only the best variant or segment makes its posterior effect optimistic.
- Computation: complex hierarchical models require convergence diagnostics and reproducible code.
Common mistakes
- Calling a posterior probability the probability that a product will succeed after rollout.
- Using a historical prior without checking population and metric comparability.
- Analyzing purchasers only when treatment changes purchase probability.
- Ignoring multiple variants and repeatedly changing the decision rule.
- Comparing Bayesian and frequentist intervals as though they answer identical probability questions.
Frequently asked questions about Bayesian statistics
Does Bayesian analysis require an informative prior?
No. Weakly informative priors are common, but they are still choices and should be disclosed.
Can a Bayesian test be stopped anytime?
The posterior can be calculated anytime. Whether stopping is sensible depends on the predeclared action rule, outcome maturity, and cost of errors.
Is a 95% credible interval the same as a confidence interval?
No. They can look similar numerically but have different probability interpretations and depend on different assumptions.
How does this handle multiple variants?
Use a joint model and a decision policy that accounts for selection, or confirm the selected variant. Simply ranking many posterior draws is not enough.
Should Bayesian methods replace power calculations?
No. Simulate the Bayesian decision rule before launch to estimate expected sample size, false actions, and probability of useful decisions.
Summary
Bayesian statistics updates prior uncertainty with a data-generating model to make transparent probability and decision statements. In A/B testing it is useful when priors, practical thresholds, monitoring rules, and model checks are explicit—not as permission to ignore experimental design.
Sources
- Gelman et al., Bayesian Data Analysis
- Gelman et al., Bayesian Data Analysis resources
- Kruschke and Liddell, The Bayesian New Statistics