Quick definition: A Bernoulli distribution models one trial with exactly two outcomes, conventionally coded 1 and 0. The probability of outcome 1 is p, and the probability of outcome 0 is 1 − p.
What is the Bernoulli distribution?
The Bernoulli distribution is the simplest probability distribution for a binary outcome. A random variable X has a Bernoulli distribution with parameter p when X can equal 1 or 0, P(X = 1) = p, and P(X = 0) = 1 − p. The labels “success” and “failure” are conventional; success can mean a purchase, activation, click, error, churn event, or any event selected as the one outcome.
At the right analysis unit, many product outcomes are Bernoulli. A visitor either places at least one order in a seven-day window or does not. A new account either completes activation within 24 hours or does not. A user either encounters a checkout error or does not. Defining the unit and time window is essential, because the same raw event can become a different variable when measured per session, user, account, or day.
Properties and notation
The probability mass function can be written compactly as:
P(X = x) = px(1 − p)1 − x, for x ∈ {0, 1}
Its expected value is E[X] = p. Therefore, the average of many correctly defined 0/1 observations estimates the event probability. Its variance is Var(X) = p(1 − p), largest at p = 0.5 and smaller near zero or one. The standard deviation is √(p(1 − p)).
For example, let X indicate whether an eligible visitor purchases within seven days. If p = 0.04, then E[X] = 0.04: over many comparable visitors, four percent are expected to purchase. A specific visitor is not “0.04 purchased”; their observed X is either 0 or 1. The probability describes uncertainty across comparable trials, not a fractional individual outcome.
Assumptions and careful definitions
A Bernoulli variable needs one unambiguous binary rule. “User converted” must specify the qualifying event, attribution window, identity resolution, and treatment of duplicates, refunds, and canceled orders. If a user purchases three times, a user-level “at least one purchase” variable remains 1; number of purchases is no longer Bernoulli.
Individual Bernoulli observations do not need to have identical probabilities merely to be coded 0/1. However, the familiar binomial model for their sum assumes a common p and independence. Real users differ in intent and may influence each other. Random assignment helps compare the average probability across arms, but clustered households, teams, markets, or social networks can require cluster-aware inference.
Do not define the binary outcome using information affected by treatment unless that is the intended endpoint. “Converted among users who reached the payment page” can be biased when the variant changes who reaches that page. Prefer a pre-assignment eligible denominator for total treatment effect.
Bernoulli outcomes in A/B testing
Binary metrics make A/B-test results concrete. Assign each eligible unit a value of 1 when it satisfies the predeclared conversion rule and 0 otherwise. The average in each arm is the conversion rate. The usual estimand is the difference in mean Bernoulli outcomes: pT − pC, reported in percentage points along with a confidence interval.
For a signup experiment, the team may define success as “confirmed account created within one day of eligibility.” It assigns users 50/50, logs assignment before rendering either form, and retains zeros for every eligible user who does not create an account. This supports an intent-to-treat interpretation: what is the effect of offering form B rather than form A?
Binary metrics are often easier to explain than continuous metrics, but they can hide mechanism. A checkout change may improve purchase conversion while reducing revenue per purchaser, raising refunds, or harming latency. Combine the primary Bernoulli outcome with guardrail metrics and relevant value measures. For uncertainty, use methods appropriate to proportions rather than relying on a dashboard color alone.
Worked example: error-rate experiment
An engineering team compares a new address-validation service with the current service. The unit is an eligible checkout session. X = 1 means the customer receives an address-validation error that blocks checkout; X = 0 means they do not. Control has 360 errors among 18,000 sessions, a 2.00% error rate. Treatment has 252 among 18,000, a 1.40% rate.
The estimated change is 1.40% − 2.00% = −0.60 percentage points. Because lower is better for this event, the negative difference is an improvement. The team also evaluates completed orders, validation accuracy, response latency, and support contacts. A lower error rate is not a win if invalid addresses increase later delivery failures.
It checks that each session appears once and that error logging uses the same server-side rule in both arms. If treatment suppresses an error event without preventing the underlying failure, the measured Bernoulli outcome no longer represents the intended customer outcome.
Interpretation and edge cases
The mean of a Bernoulli variable is a probability, so a rate difference is an absolute probability difference. Relative change can complement it: a fall from 2.0% to 1.4% is a 30% relative reduction, but the absolute change of 0.6 points better indicates the number of affected sessions at scale.
Rare outcomes deserve special care. A rate of 0.02% may have only a few observed events, producing wide or asymmetric uncertainty. Extend the observation window if the business decision allows, use appropriate proportion methods, and avoid declaring “zero risk” just because a small sample observed zero events. Conversely, near-certain events can have limited headroom for improvement.
Repeated outcomes require an explicit choice. A user-level binary metric answers whether any error occurred; an event-level rate can answer how frequently errors occur. Both may be useful, but neither should be relabeled as the other. The unit of analysis must match the decision and randomization plan.
Common mistakes
- Calling a count outcome Bernoulli: one variable must have only two possible values.
- Changing the time window between variants: it changes the outcome definition.
- Dropping non-converters: zeros are fundamental to a conversion probability.
- Assuming sessions are independent users: repeat behavior can invalidate simple standard errors.
- Optimizing a coded event rather than the real outcome: validate that the event represents customer or business value.
Frequently asked questions
Is a conversion rate a Bernoulli distribution?
Each correctly defined unit-level conversion indicator can be Bernoulli. The conversion rate is the mean of those indicators, while the total number of conversions can be binomial under additional assumptions.
Can the success outcome be bad?
Yes. “Success” is only notation. A fraud flag or error can be coded 1 when that makes the analysis clear.
What is the difference between Bernoulli and binary metric?
Bernoulli is a probability distribution for a binary random variable. A binary metric is a measurement definition that may be modeled with a Bernoulli distribution at the selected unit.
Are Bernoulli observations always independent?
No. Independence is an additional modeling assumption and often fails with repeated users, shared environments, or network effects.
Why include a confidence interval for a binary metric?
The observed rate is subject to sampling variation. An interval communicates plausible effect sizes rather than only a threshold decision.
Summary
A Bernoulli distribution represents one binary outcome with event probability p. In A/B testing, well-defined user-, account-, or session-level indicators turn conversions, errors, and retention outcomes into comparable rates. Define the denominator and window before assignment, retain zero outcomes, assess dependence, and interpret rate movement with uncertainty and downstream guardrails.
Sources
- NIST/SEMATECH e-Handbook: Binomial distribution
- OpenIntro Statistics
- NIST/SEMATECH e-Handbook: Hypothesis testing