Quick definition: Bayesian inference is the process of updating uncertainty about an unknown quantity after observing data, using Bayes’ theorem, a prior distribution, and a probability model.
What is Bayesian inference?
Bayesian inference estimates unknown parameters as probability distributions rather than returning only a point estimate or a p-value. In an experiment those parameters might be a treatment’s conversion rate, the difference in mean revenue per eligible user, or the probability that a guardrail is harmed. The posterior distribution combines what was known or assumed before the experiment with the evidence in the collected data.
Its practical attraction is direct decision language. Subject to the model and prior, an analyst can state the posterior probability that lift exceeds a product threshold. That statement is conditional, not magical: it depends on correct eligibility, randomization, outcome maturity, and the model’s ability to represent the data.
The updating process
posterior ∝ likelihood × prior. More fully, p(θ | y) = p(y | θ)p(θ) / p(y), where θ is the unknown effect and y is the observed data.
Start by writing the estimand. For a conversion test, it may be the treatment rate minus control rate among randomized eligible users over seven days. Specify a likelihood, such as binomial outcomes per user, and a prior for each rate. Compute the posterior analytically for conjugate cases or with simulation for more realistic models. Draw from the posterior to estimate intervals, threshold probabilities, expected losses, and predictive outcomes.
Inference is more than calculation. Check whether the simulated data from the fitted model resemble observed data. Inspect residuals, tails, time periods, and cluster behavior. A posterior that is numerically narrow can be badly misleading if the likelihood ignores overdispersion, repeated users, or a mid-test instrumentation change.
Assumptions and boundaries
Bayesian methods assume a prior, likelihood, and relevant conditioning information. The prior should be justified by previous comparable experiments, domain constraints, or weak regularization; perform sensitivity analysis under reasonable alternatives. Independence is commonly assumed at the user level. If users affect each other, or stores rather than users are randomized, use an appropriate hierarchical or cluster model.
Inference does not cure biased exposure. Confirm that assigned users were eligible, actually exposed under the protocol, and measured consistently. Investigate sample ratio mismatch before interpreting a treatment posterior. Missing outcomes and late conversions require a stated treatment; quietly excluding them changes the target population.
Application to A/B testing
Bayesian inference works best when tied to a decision policy. Define the minimum useful lift, a loss threshold, a maximum traffic budget, and required guardrails. A team might roll out if there is at least 97% posterior probability that conversion lift exceeds 0.15 percentage points and less than 1% probability that latency worsens beyond its tolerance. This is clearer than declaring any positive probability a win.
Continuous updates are allowed mathematically, but operational peeking still needs discipline. Repeatedly changing thresholds, features, or target segments after dashboards update produces a different decision process than the one evaluated. Predefine review cadence and simulate the complete policy under no effect, useful benefit, and harm.
Worked example: a decision with a loss threshold
A signup test has 400/8,000 conversions in control and 432/8,000 in treatment. With Beta(1,1) priors, the rate posteriors are Beta(401,7601) and Beta(433,7569). One million paired draws give a posterior lift distribution. Assume 96.5% of draws exceed zero, 90% exceed 0.2 points, and 1.8% are below −0.1 points.
If the rule requires 95% probability of at least 0.2 points, do not claim a successful rollout despite the strong probability of a positive effect. If a limited rollout is considered, calculate expected value and monitor the guardrail separately. Report the observed rates, prior, threshold probabilities, and result under skeptical and historical priors.
Interpretation
A 95% credible interval means that, given the model and prior, the interval contains 95% posterior probability for the parameter. It does not mean 95% of future users will improve, nor does it predict the full rollout effect if traffic or implementation changes. Separate statistical uncertainty from product risk and external validity.
Risks and limitations
- Small samples may be sensitive to prior choices.
- Misspecified models produce overconfident results.
- Posterior probability does not replace an economic decision rule.
- Selected winners among many variants have optimistic apparent effects.
- Complex sampling needs convergence checks and reproducible computation.
Common mistakes
- Using historical priors from a noncomparable audience.
- Ignoring post-treatment conditioning, such as analyzing buyers only.
- Calling a credible interval a frequentist confidence interval.
- Stopping on a favorable chart without a planned business action.
- Reporting a “probability to win” without defining the threshold.
Frequently asked questions about Bayesian inference
What is the difference from Bayesian statistics?
Bayesian statistics is the broader framework; Bayesian inference is its updating and estimation process.
Must the prior be subjective?
It is a modeling choice, but it can be grounded in comparable data. Transparency and sensitivity analysis matter more than pretending no choice exists.
Can it analyze revenue?
Yes, but use a model or robust estimand that reflects zero inflation and long tails, and aggregate at the randomized user.
Does it solve multiple comparisons?
No. Selection among variants, metrics, and segments still needs a joint model or confirmation strategy.
Can I monitor daily?
You can update daily, but use mature outcomes and a pretested decision policy rather than ad hoc operational reactions.
Summary
Bayesian inference updates a prior through a likelihood to provide posterior uncertainty and decision-relevant probabilities. Its value in A/B testing comes from explicit assumptions and thresholds, alongside the same rigorous design and data-quality requirements as any experiment.
Sources
- Gelman et al., Bayesian Data Analysis
- Gelman et al., Bayesian Data Analysis
- Kruschke, Bayesian estimation supersedes the t test