Statistics·Glossary term

Binomial Distribution

Binomial Distribution A/B testing Reference guide

Binomial Distribution is a concept used in statistical inference.

Quick definition: The binomial distribution models the number of successes in a fixed number of independent trials when every trial has the same probability of success. It is a foundational model for conversion counts in A/B testing.

What is the binomial distribution?

A binomial random variable counts successes rather than describing a single trial. If each of n eligible users either converts or does not convert, and conversion probability is p for each user, then the total number of conversions X can be modeled as Binomial(n, p). Its possible values are 0 through n.

“Success” is a label for the event coded as one; it need not be good. A payment failure, support ticket, or policy violation can be the modeled event. The important properties are a fixed number of trials, two mutually exclusive outcomes per trial, a common event probability, and suitable independence. In practical data, these are assumptions to assess, not facts supplied by software.

Probability, mean, and variance

The probability of exactly k successes is:

P(X = k) = C(n, k) pk(1 − p)n − k

Here C(n, k) = n! / (k!(n − k)!) counts the ways k successes can occur among n trials. The distribution has mean E[X] = np and variance Var(X) = np(1 − p). The observed conversion rate X/n has expected value p and variance p(1 − p)/n under the model.

These formulas explain why larger samples make a rate estimate more precise: its variance shrinks as n grows. They also show why rare and near-certain events behave differently from a 50% event. A normal approximation can work when expected successes and failures are sufficiently large, but exact or score-based methods are safer for sparse conversion data.

Assumptions and boundaries

The number of trials must be defined before analysis. For a user-level purchase metric, every eligible assigned user is normally one trial, including non-purchasers. Counting repeated sessions as separate independent trials changes the estimand and often violates independence because one person’s sessions are correlated.

Equal probability is an approximation. Visitors from countries, devices, campaigns, or returning-user cohorts may have different inherent conversion probabilities. Randomization makes the mix comparable between arms in expectation, which supports a comparison of average rates; it does not make every person identical. Extra heterogeneity can produce overdispersion, where observed count variability exceeds binomial variance.

Independence can fail through social sharing, inventory constraints, household devices, marketplace supply, or cluster-based assignment. If one treatment user’s behavior changes another user’s opportunity to convert, the simple binomial model may understate uncertainty. Cluster-aware analysis or a different experiment design may be needed.

A binomial model does not repair a bad denominator. Excluding users who did not see a page, after a variant has affected whether they reach it, can create selection bias even if the remaining count is mathematically binomial.

Binomial distribution in A/B testing

Many A/B-test outcomes are binary at the chosen unit: purchased within seven days or not, activated or not, clicked or not, retained at day 30 or not. Control conversions can be represented as XC ~ Binomial(nC, pC) and treatment conversions as XT ~ Binomial(nT, pT). The causal estimand is often the difference pT − pC.

Analysts estimate that difference with observed rates and report a confidence interval. Depending on sample size and convention, they may use a score interval, a regression model, a randomization test, or another suitable method. A p-value from a two-proportion test addresses compatibility with a no-difference model; it is not the probability that the variant is better. Confidence intervals show the range of effects compatible with the sample and method.

The binomial model also informs sample size. A lower baseline rate produces fewer observed successes per visitor and can require more traffic to detect a small absolute lift with the same power. Read the sample-size guide before treating a conversion count as sufficient evidence.

Worked example: purchase conversion

A store assigns 12,000 eligible visitors to control and 12,000 to a checkout variant. Control produces 480 orders, so its observed conversion rate is 4.00%. Treatment produces 540 orders, or 4.50%. The estimated absolute effect is 4.50% − 4.00% = +0.50 percentage points.

Under a simple binomial model, the expected number of control conversions at p = 0.04 is 12,000 × 0.04 = 480, and the variance of that count is 12,000 × 0.04 × 0.96 = 460.8. The expected count is not a promise that exactly 480 orders will occur; ordinary sampling variation produces a range of counts. The treatment-control comparison quantifies whether the observed difference is large relative to that variation.

The team reports the two counts, denominators, rate difference, interval, and predeclared guardrails such as refunds and payment errors. It verifies sample-ratio mismatch is absent and that purchase attribution is equally complete in both arms. A significant rate difference without reliable exposure or order data is not decision-ready.

Interpretation and alternatives

A binomial count answers how many units experience an event, not how much value each event carries. For purchases with varying order values, conversion is often paired with revenue per eligible user. A treatment can increase conversion by discounting heavily while decreasing profit. Use the binary rate for its intended question and include business guardrails.

When a unit can have several events, select a binary definition deliberately. “At least one purchase within 14 days” is binomial at the user level; “number of purchases” is a count metric and may be better modeled by Poisson, negative binomial, or user-level mean methods. Do not force a multi-event outcome into a binary model merely because a dashboard expects a conversion rate.

Common mistakes

  • Calling all event counts binomial: a binomial variable has a fixed number of binary trials.
  • Assuming identical users: randomization balances heterogeneous users; it does not erase heterogeneity.
  • Using sessions as independent users: repeated sessions can make standard errors too small.
  • Ignoring clustering or interference: related units may need cluster-aware inference.
  • Using normal approximations for very sparse data without checks: use methods suited to small expected counts.

Frequently asked questions

Is conversion rate binomial?

The number of converters can be binomial when each eligible unit has one binary conversion opportunity under the model assumptions. The conversion rate is that count divided by the number of units.

What is the difference between Bernoulli and binomial?

A Bernoulli distribution models one binary trial. A binomial distribution models the sum of n independent Bernoulli trials with the same probability.

Can a binomial model handle unequal traffic allocation?

Yes. Each arm has its own n and estimated probability. Unequal allocation usually affects precision and should be considered in sample planning.

When is the binomial assumption poor?

It can be poor with strong dependence, variable probabilities that create substantial overdispersion, changing eligibility, or a non-binary outcome.

Does a binomial model prove causality?

No. Causal interpretation comes from valid random assignment, consistent treatment delivery, and reliable measurement; the distribution supplies an outcome model and uncertainty calculation.

Summary

The binomial distribution models conversion counts as successes across a fixed number of independent, equally probable binary trials. It underlies rate estimates, confidence intervals, and sample-size calculations in many A/B tests. Define the unit and denominator carefully, test the practical assumptions, and pair conversion results with relevant business outcomes.

Sources