Quick definition: Alpha (α) is the error rate a frequentist testing procedure is designed to allow for false positive conclusions when its null hypothesis and assumptions are true. It is a rule chosen before results are evaluated, not a probability that one observed result is wrong.
What is alpha in statistics?
Alpha is the predeclared threshold used to decide whether evidence is sufficiently unusual under a null hypothesis to reject that null. In a conventional two-sided test, α = 0.05 means the procedure rejects a true null no more than 5% of the time in repeated, comparable uses. This event is a Type I error: declaring an effect when the data-generating process has no effect of the kind being tested.
The number 0.05 is a convention, not a scientific law. A team can use 0.10 when missing a promising opportunity is relatively costly and false launches are easy to reverse. It may use 0.01, 0.001, or a stricter decision process for medical, financial, legal, or irreversible product changes. The appropriate threshold reflects the decision’s consequences, the number of opportunities to make a false claim, and what evidence stakeholders need.
Alpha is often described as a “significance level.” A result is statistically significant at the chosen level when its p-value is at or below α. That shorthand is useful, but it should not hide the conditional statement: if the null model were correct and the test were repeated under its assumptions, outcomes this extreme or more extreme would occur with probability no greater than α.
What alpha does and does not mean
Alpha controls a long-run property of a decision rule. It does not say there is a 5% chance that the null hypothesis is true, a 95% chance that the treatment works, or a 5% chance that this particular finding is false. Those statements require prior probabilities and a different inferential framework.
Nor is 1 − α the probability that an experiment will replicate. Replication depends on the underlying effect, sampling variation, implementation quality, analysis choices, and publication or selection effects. A low alpha reduces one route to false discoveries, but it cannot repair biased assignment, invalid tracking, biased exclusions, or a metric that does not represent the decision.
Alpha also differs from power. Alpha governs false positives under the null; statistical power is the chance of rejecting the null when a specified alternative is true. Lowering α while holding traffic constant usually lowers power. To keep power while demanding stronger evidence, an experiment needs more information—often more randomized users, events, or clusters. See how to calculate sample size for an A/B test for the practical connection.
How alpha determines a test threshold
A hypothesis test turns alpha into a rejection region. For a two-sided z-test, the test statistic is compared with a standard normal distribution. At α = 0.05, the 2.5% probability in each tail gives critical values near −1.96 and +1.96. The null is rejected when |z| > 1.96. For a one-sided α = 0.05 test, all error probability is in one tail and the critical value is about 1.645.
For a difference in independent means, a common statistic is:
t = (x̄B − x̄A) / SE(x̄B − x̄A).
The standard error measures expected sampling fluctuation under the model. A p-value is then the probability, assuming the null, of a statistic at least as extreme as the observed one. The decision rule is simple: reject when p ≤ α; otherwise, do not reject. “Do not reject” is intentionally not the same as proving no meaningful difference.
For two-sided confidence intervals, alpha has a direct interpretation. A 95% confidence interval corresponds to α = 0.05 under the same model and method: values outside the interval would be rejected by the matching two-sided test. The interval adds information the binary threshold suppresses, including the plausible range and whether effects large enough to matter remain compatible with the data. Read more about confidence intervals in A/B testing.
Choosing alpha before an experiment
Write alpha into the test plan alongside the primary metric, unit of randomization, effect size of interest, test direction, target power, and stopping rule. This protects the decision from being tuned after a p-value appears. A product team might specify: “For the primary conversion metric, use a two-sided α of 0.05, 80% power, and a fixed sample of 40,000 eligible users per arm.”
Choosing two-sided versus one-sided testing deserves the same discipline. A one-sided test can be appropriate when only one direction would ever support the action and the direction was genuinely fixed in advance. It is not a legitimate way to halve a p-value after an unexpected result. Most launch decisions also need to detect harm, so two-sided inference and explicit guardrail metrics are often safer.
Important: Do not choose α after inspecting results, switch from two-sided to one-sided testing because the estimate points the desired way, or treat a threshold crossing as the only launch criterion. Those practices make the stated error guarantee misleading.
Alpha in A/B testing
In a standard A/B test, alpha is the false-positive budget for a predeclared claim such as “the new checkout changes conversion.” The null usually states that the treatment and control have equal expected conversion rates. If the p-value for their difference is below the specified alpha, the team has evidence against that equality under the test’s assumptions.
Random assignment, a stable definition of eligibility, correctly logged exposure, and independent analysis units are prerequisites for interpreting that result causally. Alpha cannot compensate for sample ratio mismatch, bots, a broken denominator, or users who appear in more than one arm. Before reading a p-value, validate allocation and event data; sample ratio mismatch checks are one practical safeguard.
Experimentation introduces multiplicity. Testing five variants against a control, evaluating twenty metrics, or repeatedly slicing by segment creates many chances to see a small p-value by chance. If each comparison uses α = 0.05 independently, the family’s chance of at least one false positive exceeds 5%. A team can limit the family to planned primary analyses, adjust p-values or thresholds with methods such as Bonferroni or Holm, or use a false-discovery-rate approach when screening. The choice should match the claim being made; see multiple comparisons in A/B testing.
Repeated monitoring causes a related problem. A fixed-horizon 0.05 threshold assumes one final analysis at the planned information size. Checking every day and stopping whenever p falls below 0.05 gives more opportunities for noise to cross the boundary. Use a fixed-duration plan or a valid sequential method with preplanned boundaries; A/B test duration explains why calendar time and information size matter.
Worked example: conversion-rate decision
Suppose a team tests a redesigned pricing page. Its pre-registration names purchase conversion as the sole primary metric, sets a two-sided α = 0.05, and targets a fixed sample of 20,000 users per group. Control records 1,000 purchases (5.00%); treatment records 1,100 (5.50%). The estimated absolute lift is 0.50 percentage points, or 10% relative to control.
Using a two-proportion test, the estimated standard error of the difference is roughly 0.0022. The z-statistic is approximately 0.0050 / 0.0022 = 2.27. Its two-sided p-value is about 0.023, below 0.05, so the team rejects the equal-conversion null according to the plan. A compatible 95% confidence interval is roughly 0.07 to 0.93 percentage points of absolute lift.
That outcome does not automatically mean “ship everywhere.” The team should check that exposure was logged before purchase, allocation was as planned, no revenue or support guardrail deteriorated, and the interval clears the minimum practical lift. It should also distinguish its predeclared conclusion from exploratory breakdowns. If the primary metric had produced p = 0.07, the correct statement would be that the study did not reject the null at α = 0.05—not that the page has no effect.
How to interpret an alpha threshold responsibly
Report the threshold, sidedness, planned analysis count, p-value, effect estimate, and confidence interval together. For example: “With a predeclared two-sided α = 0.05, conversion rose 0.50 percentage points (95% CI: 0.07 to 0.93; p = 0.023).” This makes clear what was estimated and what decision rule was used.
Consider practical significance separately. A statistically significant 0.03-point lift can be unimportant at very high traffic, while an estimated 1-point lift with p = 0.06 may be commercially meaningful but uncertain. Decisions can incorporate expected value, rollout reversibility, evidence from related tests, and risk constraints; alpha remains a guardrail against overclaiming from random variation.
Common alpha mistakes
- Equating α with the probability a result is false. It is a long-run Type I error property conditional on the null and the full procedure.
- Moving the goalposts. Selecting alpha, direction, metric, exclusions, or stopping time after seeing data invalidates the advertised guarantee.
- Ignoring multiplicity. Several variants, metrics, segments, and looks need a stated error-control strategy.
- Calling non-significance “no effect.” Examine the estimate and interval; the study may simply be imprecise.
- Using alpha as a quality check. Significance cannot diagnose biased targeting, contamination, or defective instrumentation.
Frequently asked questions about alpha
Is alpha always 0.05?
No. 0.05 is common because of convention. Set a threshold that fits the cost of false positives, the decision’s reversibility, and the study’s multiplicity, then document it before analysis.
Does p = 0.04 mean the experiment has a 96% chance of winning?
No. It means that, under the null model, data at least this extreme would occur about 4% of the time. It is not a probability assigned to the treatment hypothesis.
Should I use one-sided alpha for an A/B test?
Only when the action and hypothesis genuinely rule out the other direction before data collection. Most product experiments should still monitor for harm through guardrails.
How does alpha relate to a 95% confidence interval?
For matching two-sided procedures, α = 0.05 corresponds to a 95% confidence interval. If zero is outside that interval, the associated two-sided test rejects at 0.05.
Can I look at results before the planned end date?
Yes only under a monitoring plan whose sequential method preserves error control. Ordinary repeated fixed-horizon p-value checks do not.
Summary
Alpha is a precommitted false-positive budget for a complete frequentist decision procedure. In A/B testing, set it with the primary hypothesis, sample-size plan, multiplicity strategy, and stopping rule; then interpret threshold crossings alongside effect size, uncertainty, data quality, and business risk.
Sources
- NIST/SEMATECH e-Handbook: Hypothesis Testing
- American Statistical Association: Statement on Statistical Significance and P-Values
- NIST/SEMATECH e-Handbook: Confidence Intervals