Quick definition: A credible interval is a Bayesian interval containing a stated posterior probability for an unknown parameter, conditional on the chosen prior, likelihood, and observed data.
What is a credible interval?
A credible interval summarizes uncertainty after Bayesian updating. If a 95% credible interval for treatment conversion lift is −0.10 to 0.60 percentage points, then, given the model, prior, and data, there is 95% posterior probability that the parameter lies in that interval. This direct probability statement is its defining feature.
The parameter must be specified before calculating an interval. In an A/B test it might be the difference in seven-day conversion among all randomized eligible users, a revenue ratio, or the treatment effect for a defined population. It is not automatically a statement about next month’s revenue, every segment, or the effect after a changed implementation. A narrow interval around the wrong estimand is not a useful result.
Credible intervals belong to Bayesian inference, which combines a prior distribution with a likelihood for observed data. They should not be described as frequentist confidence intervals. A 95% confidence-interval procedure has repeated-sampling coverage properties; it does not, without a Bayesian model, assign 95% probability to the fixed parameter being inside a realized interval. Both approaches can be useful, but their language and assumptions differ.
How credible intervals are calculated
Bayes’ theorem gives p(θ | y) = p(y | θ)p(θ) / p(y), where θ is the parameter and y is the observed data. First state a likelihood: binary user conversion can use a binomial model, while a continuous outcome may require a distribution or a robust model suited to its shape. Then choose and document a prior. The posterior is the distribution used to obtain an interval.
For a conversion rate with a Beta(a, b) prior and x conversions among n independent users, the posterior is Beta(a + x, b + n − x). For two variants, draw rates from each posterior, subtract treatment minus control for every draw, and use the 2.5th and 97.5th percentiles of those differences as an equal-tailed 95% credible interval. Simulation also estimates the probability that lift exceeds zero or a practical threshold.
An equal-tailed interval leaves equal posterior mass in both tails. A highest-density interval instead includes the most probable parameter values and is often shorter for asymmetric distributions. Report which definition was used. With complex hierarchical, regression, or revenue models, posterior draws may come from Markov-chain Monte Carlo. Convergence diagnostics, posterior predictive checks, and reproducible code then matter as much as percentile calculations.
Assumptions and prior choices
A credible interval is only as defensible as its model. The likelihood must reflect the randomization unit, dependence, outcome distribution, and missing-data process. Treating multiple sessions from one user as independent creates an overconfident interval. If stores are randomized, use a cluster-aware model. If revenue has many zeros and a long right tail, a simple normal model may hide meaningful uncertainty.
The prior must be defensible for the current decision. A weakly informative prior can prevent implausible estimates in small samples; a historical prior can add information when earlier experiments involved comparable users, exposure, metrics, and implementation. A prior borrowed from a different country, channel, or product version may bias the result. Show sensitivity to reasonable skeptical, neutral, and historical priors, especially when the sample is small.
Randomization and measurement assumptions remain non-negotiable. Verify eligibility, assignment, exposure, and outcome maturity. A posterior interval cannot repair a sample ratio mismatch or a treatment-specific tracking loss. Missing outcomes need a documented treatment rather than silent deletion. Bayesian probability does not convert biased data into causal evidence.
Credible intervals in A/B testing
Use the interval alongside a predeclared business rule. A team might require a 95% posterior probability that conversion lift exceeds 0.20 points, plus less than 2% probability that latency harm exceeds 100 milliseconds. This is more actionable than a rule requiring only that an interval exclude zero. The threshold should reflect value, implementation cost, reversibility, and risk.
Bayesian updating permits a posterior at any time, but it does not make an improvised launch policy valid. Repeated dashboard checks accompanied by changing thresholds, segments, or stop decisions create a policy whose error and business properties are unknown. Predefine the review cadence, minimum outcome maturity, traffic cap, decision thresholds, and actions; simulate the policy under benefit, no effect, and harm.
Report the interval with observed rates and absolute lift. A result such as “0.35 points, 95% credible interval [0.05, 0.65]” says more than “the treatment probably wins,” but still needs primary versus guardrail governance. A financially promising conversion interval can coexist with unacceptable refunds or performance effects. See primary and guardrail metrics for that decision structure.
Worked scenario: signup completion
Control has 400 conversions out of 8,000 eligible users and treatment has 432 out of 8,000. With independent uniform Beta(1,1) priors, the rate posteriors are Beta(401,7601) and Beta(433,7569). Drawing paired samples produces a posterior distribution for the treatment-minus-control difference. Suppose its median is 0.40 percentage points and its 95% credible interval is −0.04 to 0.85 points.
The posterior probability of positive lift might be 96%, yet the probability of exceeding a commercially useful 0.25-point lift might be 75%. A rule demanding 95% probability of useful lift would not authorize full rollout. The interval crossing zero does not say the change is worthless; it says a small harm and a useful benefit are both still plausible under the stated model. The practical next action might be more data, a limited reversible release, or a different variant.
Interpretation and communication
Always condition the statement: “Under the documented prior and model, the posterior probability that lift lies between A and B is 95%.” Include the prior, outcome window, analysis population, interval type, and sensitivity result. Avoid calling it a probability that users will convert, a probability that the experiment is correct, or a guarantee of future rollout performance.
Intervals should drive trade-offs. If the whole interval is above the minimum useful effect and guardrails are acceptable, the decision is robust. If it spans important loss and gain, uncertainty is decision-relevant. If it is narrow around a trivial positive effect, implementation may still not pay for itself. This emphasis on effect scale parallels the role of confidence intervals in A/B testing.
Limitations and common mistakes
- Using an unexamined prior: small samples can inherit its assumptions.
- Ignoring dependence: posterior precision can be falsely high for clustered or repeated data.
- Confusing credible and confidence intervals: their probability interpretations differ.
- Stopping on any favorable probability: use an explicit, evaluated decision policy.
- Selecting among many variants or segments: winner selection remains optimistic without a joint plan.
- Skipping model checks: a precise posterior from a poor likelihood is misleading.
Frequently asked questions about credible intervals
Does a 95% credible interval mean a 95% chance the effect is inside it?
Yes, conditional on the specified Bayesian model and prior. That conditioning is essential.
Can I use a credible interval for revenue?
Yes, with a model that accommodates zeros, skew, tails, and the randomized user-level outcome.
Must I use a uniform prior?
No. Use a justified prior and show how reasonable alternatives affect the decision.
Does an interval excluding zero guarantee rollout success?
No. It does not address practical magnitude, guardrails, implementation changes, or external validity.
Can Bayesian intervals solve multiple comparisons?
No. Comparing many variants and reporting the apparent best still creates selection bias unless the analysis and decision rule account for it.
Summary
A credible interval expresses posterior uncertainty for a defined parameter. In A/B testing it supports direct, threshold-based decisions when prior choice, likelihood, randomization unit, data quality, and model checks are transparent. It complements rather than replaces disciplined experiment design.
Sources
- Gelman et al., Bayesian Data Analysis
- Gelman et al., Bayesian Data Analysis
- Kruschke, Bayesian estimation