Quick definition: In experimentation, a hypothesis is a specific, testable prediction about how a defined change will affect a defined population, outcome, and decision.
What is a hypothesis?
A hypothesis turns an idea into a claim that an experiment can challenge with evidence. “Make onboarding simpler” is an intention, not a useful hypothesis. “For newly created workspaces, replacing a five-step setup checklist with a template picker will increase seven-day project creation without increasing support contacts” is testable because it names the population, change, outcome, comparison, and possible cost.
In product experimentation, a good hypothesis connects a proposed mechanism to a decision. The mechanism might be that a template reduces blank-page anxiety, a delivery estimate reduces purchase uncertainty, or a reminder helps users return at the right moment. The decision is what the team will do if the evidence is favorable, unfavorable, or uncertain. This distinguishes a hypothesis from a feature request, a success target, or a retrospective explanation for a result.
A product hypothesis is related to, but not identical with, a formal alternative hypothesis. The product statement explains why a change could matter and what action follows. The statistical statement defines a parameter comparison, such as whether treatment conversion differs from control conversion. Both should be written before exposure begins, but neither replaces the other.
Hypotheses in experimental design
Start with a decision that is within the team’s control: launch a new checkout, retain the current ranking model, choose between two messages, or invest in another iteration. Define the eligible population before assignment, then select an experiment unit that can receive one consistent experience. A user-level onboarding change generally needs stable user or workspace assignment; a store policy may require store-level assignment. The unit, eligibility rule, and exposure definition determine whose outcome the hypothesis can address.
Next, specify a primary metric that represents the intended value, a measurement window, and guardrails. A hypothesis about reducing checkout friction should not be judged by button clicks alone. Completed orders per eligible visitor may be primary, while payment failures, refunds, page latency, and customer contacts constrain the decision. Establish the baseline, minimum worthwhile effect, allocation, and sample size before launch. The published guide to calculating sample size explains why a desired lift alone cannot determine run length.
The comparison should be concurrent and randomized whenever feasible. A control group estimates what eligible units would have experienced during the same period without the change. Comparing this week’s release to last month’s dashboard cannot isolate the change from seasonality, campaigns, inventory, or traffic mix. Log assignment, actual exposure, and outcomes with compatible identifiers so the final analysis tests the hypothesis rather than an instrumentation difference.
| Weak wording | Decision-ready hypothesis |
|---|---|
| “Users want faster setup.” | “For new workspaces, a template picker will increase seven-day project creation by at least one percentage point versus the current checklist.” |
| “The new banner is better.” | “For eligible returning visitors, the banner will increase completed upgrades without raising unsubscribe rate beyond the agreed guardrail.” |
| “People dislike shipping costs.” | “Showing delivery cost earlier will reduce checkout abandonment while preserving net revenue per visitor.” |
Practical scenario: testing an activation prompt
A collaboration product observes that many new workspaces invite teammates before creating any work. Research suggests that offering a starter template immediately after workspace creation could help people reach value sooner. The team writes this hypothesis: among new self-serve workspaces in supported languages, a template picker will increase the share creating a first project within seven days by at least 1 percentage point relative to the current invitation screen, without increasing setup errors or support tickets.
Control retains the invitation screen; treatment adds the template picker and keeps invitations available. Workspaces are assigned persistently, because multiple people may use the same workspace. The primary metric uses every eligible assigned workspace, not only those who click the picker. Time to first project and invitation completion are diagnostic metrics. Seven-day project creation is chosen because it is close to the proposed mechanism, while 30-day retained activity is a longer-term follow-up. The team also records the engineering cost and the threshold at which the change is worth maintaining.
At readout, treatment raises first-project creation by 1.4 percentage points, but the confidence interval includes 0.3 points and setup errors rise slightly. The hypothesis is not simply “proven” by a positive point estimate. The decision workflow asks whether the interval excludes a meaningful loss, whether the expected benefit exceeds the practical threshold, whether guardrails are acceptable, and whether data-quality checks passed. The team may continue to the planned sample, revise the template selection flow, or decide the expected value does not justify rollout.
How to use a hypothesis in decisions
- Describe the problem and mechanism. Use research, analytics, support evidence, or prior tests to explain why the change might alter behavior.
- Name the decision. State whether the result will inform launch, rejection, iteration, or further research.
- Scope the claim. Document population, eligibility, treatment, reference experience, randomization unit, and outcome window.
- Predefine measurement. Choose one primary metric, practical threshold, guardrails, exclusions, and analysis method before results are visible.
- Validate delivery. Check stable assignment, sample allocation, exposure logging, and metric definitions; an A/A test can help validate the platform.
- Read evidence in context. Review effect size, uncertainty, quality diagnostics, and the full set of planned outcomes.
- Record the learning. Preserve the original hypothesis, deviations, result, decision, and follow-up question so future teams do not rewrite history.
A hypothesis is especially useful when several stakeholders want different outcomes. It requires a team to make trade-offs visible before a dashboard creates pressure to favor a convenient metric. If a launch is reversible and cheap, a smaller expected benefit may be sufficient. If a policy affects trust, price, or safety, require stronger evidence and explicit stop conditions. Statistical significance is evidence about a comparison under a model; it is not a complete shipping rule.
Limitations and common mistakes
A clear hypothesis cannot make an invalid metric meaningful or repair failed randomization. It also does not guarantee that an observed effect will persist across countries, seasons, devices, or future implementations. A result applies first to the tested population, version, and measurement period. Treat unplanned segment patterns as exploratory unless the study was designed to estimate them.
- Writing it after the result: post-hoc stories can generate ideas, but they are not confirmatory evidence.
- Using vague success language: “engagement” or “better experience” needs an operational metric and denominator.
- Confusing a target with a prediction: a quarterly revenue target is not a causal claim about a change.
- Choosing metrics after seeing winners: this increases the chance that random variation becomes a launch story.
- Ignoring harms: every stated benefit should have relevant guardrails and safety review.
- Claiming universal truth: the hypothesis concerns the specified context, not all users forever.
Frequently asked questions
Is a hypothesis the same as an A/B test?
No. A hypothesis is the claim and decision framework; an A/B test is one method for comparing experiences and collecting evidence about it.
How many hypotheses should one experiment have?
Use one primary hypothesis for the main decision. Secondary questions and diagnostics are useful, but label them clearly and account for multiple comparisons when making confirmatory claims.
Can qualitative research create a hypothesis?
Yes. Interviews, usability studies, and support feedback can reveal mechanisms and friction. They generate a testable idea; randomized measurement assesses its production effect.
What if the result is inconclusive?
It may mean the interval includes both useful benefit and meaningful harm, the sample was insufficient, or the data is unreliable. Do not equate it with proof of no effect.
Must every hypothesis predict an increase?
No. A team may test reduction of errors, latency, cancellations, or cost. The useful direction follows the decision and should be set before data is read.
Summary
A hypothesis makes an experiment decision-ready by specifying a change, population, mechanism, outcome, comparison, and practical action. Predefine it with metrics and guardrails, test it using reliable concurrent evidence, and limit conclusions to the context actually studied.
Sources
- AB Labz: How to write an A/B-test hypothesis
- AB Labz: Primary versus guardrail metrics
- Experiment design glossary