Quick definition: Beta (β) is the probability of a Type II error: failing to reject a specified null hypothesis when a real effect of a stated size exists. Statistical power is its complement, 1 − β.
What is beta in hypothesis testing?
In a fixed-horizon A/B test, beta describes a particular kind of wrong conclusion. A team may run a test, observe a result that does not cross its significance threshold, and conclude that it lacks evidence to reject the no-effect hypothesis. If the treatment truly has the effect the team planned to detect, that conclusion is a Type II error. Beta is the probability of making that error under the stated design.
Beta is not a property of a metric, product, or dataset by itself. It depends on the test’s significance level, sample size, outcome variability, allocation, one- or two-sided alternative, and the true effect size under consideration. A test might have 20% beta for a 5% relative conversion lift but much higher beta for a 1% lift. Saying only “beta is 0.2” leaves out the most important condition: what effect was meant to be detectable.
Power is more commonly used in planning. A design with β = 0.20 has 80% power: if the assumed effect is real and all design assumptions hold, repeated tests would detect it according to the prespecified decision rule about eight times out of ten. It does not mean there is an 80% probability that the treatment works in one completed test.
Beta, power, and the decision rule
For a test statistic with rejection region R, beta at a specific alternative parameter value θ1 is:
β(θ1) = P(not reject H0 | θ = θ1)
Power at that same value is:
power(θ1) = P(reject H0 | θ = θ1) = 1 − β(θ1)
For a two-sample proportion test, exact power calculations use the assumed control conversion, treatment conversion, sample sizes, and critical values. Planning tools usually solve the inverse problem: given alpha, desired power, baseline rate, and minimum detectable effect, how many units per arm are required? The normal approximation is useful for intuition but production planning should use a validated calculator or analysis library, especially for sparse outcomes.
larger sample or larger true effect → lower beta → higher power. Raising alpha also raises power, but it increases Type I error risk and should not be used merely to rescue an underpowered design.
The trade-off with alpha
Alpha (α) controls the chance of a Type I error: rejecting the null when the null is true. With the same data and decision threshold, demanding a lower alpha makes rejection harder. That generally increases beta at a fixed sample size. The usual response is not to relax rigor after seeing results; it is to size the study before launch for both error rates.
Teams often choose α = 0.05 and power of 80% or 90%, but these are conventions, not laws. A risky irreversible change may justify stronger evidence and higher power. A low-risk exploratory prototype may support a different, explicitly documented standard. The choice should reflect the cost of false launches, missed opportunities, traffic availability, and the smallest effect worth acting on.
Multiple variants, many primary comparisons, interim looks, clustering, and uneven allocation can all alter the required information. A naïve calculation for a 50/50 two-arm fixed-horizon test does not automatically protect an A/B/n test or a test stopped at a convenient moment. See how to calculate A/B-test sample size and multiple-comparisons guidance when those conditions apply.
Beta in A/B-test planning
Begin with the decision, not a desired traffic number. Suppose a team is considering a checkout redesign. It decides that an absolute conversion gain below 0.3 percentage points would not justify added maintenance, while a loss of that magnitude would be harmful. Its baseline conversion is 4.0%, it plans a two-sided 5% test, and it wants 90% power to detect a 0.3-point difference. Those inputs define the sample requirement; traffic then determines the likely duration.
The assumed baseline and effect should be plausible and stable. An outdated baseline can underpower or overpower the study. A minimum detectable effect is not a forecast that the treatment will achieve exactly that lift; it is a planning target connected to a practical decision. If the team would only ship a 1-point gain, designing for 0.1 points spends precision on a difference it would ignore. If it cares about 0.1 points but sizes for 1 point, a non-significant result will be uninformative.
Power calculations also assume the metric is measured as planned. Exposure logging failures, bot traffic, treatment noncompliance, dilution from crossover, and variance larger than expected reduce effective power. Valid randomization does not compensate for an analysis population that no longer represents the planned denominator. Use an A/A test or launch checks to validate the pipeline before relying on a large experiment.
Worked example: detecting a conversion change
An ecommerce team has a baseline purchase rate of 4.0%. It wants to detect an increase to 4.4%, an absolute difference of 0.4 percentage points, with two-sided α = 0.05 and 80% power. A standard two-sample proportion calculation indicates that it needs roughly 17,000 eligible users in each arm; the exact value varies with the method and continuity adjustment used.
If the true treatment rate really is 4.4% and the operational assumptions hold, the test has about an 80% chance of producing a statistically significant result in the planned direction or, for a two-sided test, a rejection of equality. It also has about a 20% probability of not rejecting the null: β ≈ 0.20 at that effect. If the true rate is 4.2%, the beta is higher because a smaller difference is harder to distinguish from sampling variation. If the true rate is 5.0%, beta is lower.
After the planned sample arrives, the result is 4.39% in treatment and 4.02% in control, with a confidence interval of +0.05 to +0.69 percentage points. The team should not say it “proved a 0.4-point lift.” The interval and business threshold guide the decision. It should also review refund rate, page latency, and revenue per visitor as guardrails.
Interpretation after a test
Post-hoc power calculated from the observed effect is usually not helpful. It is largely a transformation of the p-value and can make a noisy estimate look like a precise design diagnosis. Instead, inspect the confidence interval against the smallest practically important benefit and harm. A wide interval spanning both can mean more information is needed. A narrow interval excluding the worthwhile benefit can support stopping work on that version even if the p-value is not significant.
“Non-significant” does not mean “no effect,” and it does not establish that beta in the completed test was a particular number. The original beta referred to a hypothesized effect under a design. Once data arrive, uncertainty about effect size is more clearly communicated by an interval, together with sample size and data-quality evidence.
Do not change beta, alpha, or the target effect after looking at the result to declare a winner. Amendments can be valid when operational facts change, but they need prospective documentation and an analysis that preserves the intended error control.
Common mistakes
- Calling beta the probability that the null is true: it is a long-run error probability conditional on a specified alternative.
- Omitting the effect size: beta changes across possible treatment effects.
- Equating 80% power with an 80% chance a result is correct: predictive probability depends on more assumptions than power alone.
- Using post-hoc observed power to interpret a null result: use confidence intervals and practical thresholds instead.
- Ignoring design effects: repeat visits, clustering, attrition, peeking, and multiplicity can reduce actual power or invalidate nominal guarantees.
Frequently asked questions
What beta should an A/B test use?
A common planning choice is β = 0.20, or 80% power. Choose it based on the consequences of missing a valuable or harmful effect, not because it is a universal requirement.
Is beta the same as a p-value?
No. Beta is a predesign error probability at a specified alternative. A p-value is calculated from observed data under a null model.
Can I lower beta without more traffic?
You may reduce outcome variance, improve allocation, use pre-experiment covariates, or target a larger minimum effect. Increasing alpha also increases power but changes false-positive risk.
Does 80% power guarantee significance?
No. Even when the assumed effect is real, a design with 80% power misses it about one time in five over repeated similar studies.
Why is power lower for rare conversions?
Rare events have less information per visitor. Detecting a small absolute difference generally requires more eligible units and a reliable attribution window.
Summary
Beta is the Type II error probability for a specified alternative and design; power is 1 − β. In A/B testing it turns a business-relevant minimum effect into a sample-size requirement. Plan it before launch, preserve the assumptions through measurement and stopping rules, and interpret completed results with confidence intervals rather than post-hoc power claims.
Sources
- NIST/SEMATECH e-Handbook: Type I and Type II errors
- NIST/SEMATECH e-Handbook: Sample-size determination
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments