Quick definition: The null hypothesis, H0, is the reference claim used in a statistical test—often that a treatment produces no difference from control on a defined outcome.
What is a null hypothesis?
The null hypothesis is a precise model of no effect or another reference condition that observed data are evaluated against. In a two-arm conversion experiment, a common null is H0: pT = pC, where pT and pC are treatment and control conversion rates. A test statistic and p-value describe how compatible the observed data would be with that null, given the planned analysis assumptions.
“No difference” does not mean that every user experiences exactly the same outcome, nor does it mean a product change has no mechanism. It means the population parameter being tested is equal under the stated model. Random variation produces different observed rates even when the underlying rates are equal. The null is therefore a benchmark for evidence, not a claim that the two implementations are literally identical in every respect.
The null is paired with an alternative hypothesis, which states the pattern that would contradict it. For a two-sided test, Ha may state pT ≠ pC; for a predeclared directional test, it may state pT > pC. The formal pair is distinct from the product decision: whether to ship also depends on effect size, uncertainty, guardrails, implementation cost, and risks.
The null hypothesis in experiment design
Set the null, metric, analysis population, and stopping rule before reading outcome data. The test should answer a decision-ready comparison: among eligible users assigned to a new checkout or existing checkout, is completed-order rate different within a seven-day attribution window? Define the denominator consistently. If treatment logging uses page loads while control logging uses assigned visitors, the analysis is not testing the intended null; it is comparing incompatible measures.
Random assignment and a concurrent control group make the null comparison meaningful. They balance unobserved pre-treatment characteristics on average and expose both arms to the same season, campaign, and operating conditions. Historical baselines are helpful for planning sample size, but they are not a substitute for a live reference. An unexplained allocation imbalance, missing exposure events, or post-assignment exclusions can invalidate the comparison before a p-value is even considered.
Choose a fixed horizon or a sequential method that matches the monitoring plan. Repeatedly checking a conventional fixed-sample test and stopping at the first result below 0.05 makes the nominal null rejection rate misleading. Similarly, testing numerous variants, metrics, or segments creates more chances for a chance result. Predefine the primary claim and apply an appropriate multiplicity or sequential approach for confirmatory decisions.
| Readout | What it supports | What it does not support |
|---|---|---|
| Small p-value under H0 | Evidence against the specified null under the planned model | Proof that treatment is valuable or universally better |
| Large p-value | Data are not strong evidence against the null | Proof that the versions are equal |
| Narrow interval around a tiny effect | Precision about a small effect | Automatic business justification |
| Wide interval crossing harm and benefit | Uncertainty remains | A confident no-effect conclusion |
Practical scenario: a payment-method change
A retailer tests a new wallet button. Its primary metric is completed orders per eligible visitor within seven days. The formal null is equal order rates between treatment and control. The team chooses two-sided inference because a decrease is commercially and ethically important, sets a minimum useful increase, and includes payment error rate, refund rate, and revenue per visitor as guardrails. Assignment is stable by customer where possible, and the test runs over full weekly cycles.
At completion, the estimate is a 0.15 percentage-point increase, but the interval ranges from a 0.10-point decrease to a 0.40-point increase. The p-value does not cross the predefined threshold. This is not evidence that the wallet button has zero effect. It means the study did not provide enough evidence to reject its no-difference model under the chosen procedure. The interval still includes a meaningful benefit and a meaningful loss, so the team may gather more planned information, change the implementation, or decide that uncertainty is too costly.
In another run, the estimate is statistically distinguishable from zero but only 0.03 points, with a narrow interval. The null may be rejected, yet rollout can remain unattractive if integration complexity, fees, or guardrail effects outweigh the incremental gain. Hypothesis testing answers one part of a decision; it does not replace a value calculation.
Decision workflow
- Write the comparison first. Document population, treatment, control, metric, time window, and the null parameter statement.
- Define practical stakes. Set a minimum worthwhile effect and guardrails separately from the statistical threshold.
- Plan information. Use baseline, variance, allocation, alpha, power, and MDE to calculate a sample size; see sample-size planning.
- Validate data. Check assignment, exposure, metric definitions, outcome maturity, and sample allocation before final inference.
- Apply the planned analysis. Do not change sidedness, metric, segment, or stopping point because a dashboard result is appealing.
- Report uncertainty. Include counts, effect estimate, interval, p-value where applicable, and every planned guardrail.
- Choose an action. Assess evidence with value, downside, reversibility, and the quality of the implementation.
Limitations and common mistakes
A null hypothesis is a statistical device, not a complete causal guarantee. Rejecting it can be driven by a tiny, unimportant effect when traffic is large. Failing to reject it can occur because data are too noisy or too limited. It does not validate randomization, ensure a representative population, or establish long-term effects. Those require design review and appropriate follow-up measurement.
- “Accepting” H0 from a large p-value: use an equivalence or non-inferiority design when the decision requires evidence of negligible difference.
- Interpreting p-value as probability H0 is true: it is calculated conditional on the null model and procedure.
- Changing H0 after results: label new analyses exploratory and confirm them later.
- Ignoring multiple tests: many outcomes and slices make chance rejections more likely.
- Using null rejection as a launch rule: practical significance and guardrails still matter.
- Forgetting model assumptions: clustering, interference, missing data, and invalid units can distort standard errors and conclusions.
Frequently asked questions
Does a large p-value prove the null hypothesis?
No. It says the data did not provide strong evidence against that null under the chosen method. The confidence interval shows which effects remain plausible.
Is the null hypothesis always “no effect”?
Often, but not always. It can specify a boundary or reference value, such as no more than an allowed degradation in a non-inferiority design.
Can I use a one-sided null?
Yes, when the directional decision and analysis are justified before data is viewed. Do not select one-sided inference simply to make a positive result easier to claim.
What is the difference between null and baseline?
A baseline is a historical or reference metric level. A null is a formal statistical statement about a parameter comparison; the current control rate is observed during the test.
What should accompany a null-hypothesis result?
Effect size, uncertainty interval, sample counts, quality diagnostics, practical threshold, and guardrail outcomes make the conclusion decision-ready.
Summary
The null hypothesis defines the reference claim a statistical experiment evaluates, commonly equal outcomes in treatment and control. Predefine it with the metric and analysis plan, then interpret any result alongside uncertainty, data quality, practical significance, and guardrails. Its purpose is disciplined comparison, not automatic acceptance or rejection of a product idea.
Sources
- Alternative hypothesis glossary
- AB Labz: P-values in A/B testing
- AB Labz: Confidence intervals in A/B testing