What is an A/B test?
An A/B test is one concrete experiment within the broader discipline of A/B testing. It can compare a landing page, checkout flow, pricing presentation, email subject line, mobile screen, recommendation rule, or backend feature. The control is normally the current production experience; the treatment is the proposed change.
The distinction matters for documentation. “A/B testing” describes the method and operating practice. “An A/B test” describes a single test with a defined population, assignment rule, exposure, metric, analysis plan, and decision.
GrowthBook, Statsig, Optimizely, and academic experimentation guides consistently describe the key mechanism as random assignment followed by comparison against a defined metric [1] [2] [3] [4].
The structure of an A/B test
| Component | What to define | Example |
|---|---|---|
| Decision | What will change after the test? | Ship, iterate, or roll back a shorter signup form |
| Control A | Current reference experience | Existing form with five fields |
| Treatment B | Precisely documented change | Form with three fields |
| Unit | User, account, session, or other assignment unit | Stable user ID |
| Primary metric | Main outcome used for the decision | Activated-account rate |
| Guardrails | Outcomes that must not materially worsen | Support contacts and fraud rate |
| Stopping rule | When and how the result will be evaluated | Planned sample plus full weekly cycle |
Example: a signup-flow A/B test
A SaaS team believes that removing two optional fields will increase the number of new users who complete onboarding. Its hypothesis is:
Version A keeps the five-field form. Version B uses three fields. Eligible new visitors are assigned persistently at user level. The primary metric is activation within seven days—not just form completion. Secondary metrics explain where the funnel changes; guardrails monitor fraud, support requests, and account quality.
Assignment and exposure
Random assignment is what supports the causal interpretation. It balances observed and unobserved user characteristics in expectation, so the groups are not created by choosing “easy” users for B. The assignment should be persistent: a user who first sees B should not see A on the next visit.
The randomization unit should match the way the experience is consumed. For a collaborative B2B product, company-level assignment may be safer than user-level assignment; otherwise colleagues can receive incompatible experiences. For a page component, user- or session-level assignment is usually more coherent than changing the variation on every page view. The choice affects independence, power, and contamination risk [5].
Choosing metrics
Choose one primary metric that reflects the decision. Secondary metrics are useful for mechanism and diagnosis, but declaring a winner when any one of many metrics happens to be significant increases false-positive risk. Statsig explains that the probability of at least one false positive rises as the number of comparisons grows; ten independent tests at a 5% threshold can create roughly a 40% chance of at least one false positive under a simplified model [6].
| Metric layer | Use | Example |
|---|---|---|
| Primary | Make the planned decision | Activated accounts per eligible user |
| Secondary | Understand why it moved | Form completion, onboarding step rate |
| Guardrail | Catch harmful trade-offs | Fraud, cancellations, latency |
| Diagnostic | Validate the experiment itself | Exposure count, traffic allocation, SRM |
Sample size and duration
Calculate sample size before launch. The estimate depends on baseline rate, minimum detectable effect, significance level, desired power, number of arms, and allocation. A smaller MDE requires more traffic; splitting traffic across more variants reduces the sample per arm. Optimizely documents the direct relationship between baseline, MDE, sample size, and duration [7].
Run until the planned sample is reached and normal operating cycles are covered. Do not stop an ordinary fixed-horizon test the moment its dashboard crosses a significance threshold. Repeated interim looks change the error properties; if early decisions are necessary, use a sequential method built for interim analysis [8].
How to interpret results
For a standard A/B test, the result is not simply “B had the larger number.” Use the primary metric, effect size, uncertainty, power, guardrails, and implementation checks together.
| Outcome | Meaning | Recommended action |
|---|---|---|
| Win | B shows a practically worthwhile effect with acceptable uncertainty and no guardrail harm. | Ship gradually and monitor post-launch behavior. |
| Loss | B is credibly worse on the primary metric or creates unacceptable harm. | Do not ship; investigate the hypothesis and failure mechanism. |
| Inconclusive | The interval remains compatible with both a worthwhile effect and no meaningful effect. | Collect more evidence, improve the design, or move on. |
A p-value is evidence evaluated under a null model; it is not the probability that B is better. Statistical significance also does not tell you whether the uplift pays for engineering cost. Report absolute change, relative uplift, interval, sample counts, metric definitions, and quality diagnostics.
Common A/B test mistakes
- No real hypothesis: testing a cosmetic change without a mechanism or decision.
- Changing several unrelated things: a win becomes difficult to attribute.
- Using a feature metric as the business outcome: clicks rise but activation or revenue falls.
- Peeking: stopping at the first favorable result inflates false positives.
- Changing allocation: moving traffic mid-test can complicate inference and assignment.
- Ignoring SRM: an unexpected split can signal a broken experiment path.
- Fishing through segments: post-hoc subgroup wins require confirmation on fresh data.
- Using A/B testing alone: experiments measure what happened; interviews and session research help explain why.
A/B test launch checklist
- Decision and hypothesis are documented before launch.
- Control and treatment differ only in the intended way.
- Randomization unit and persistence are appropriate.
- Eligibility and exposure are clearly defined.
- One primary metric and explicit guardrails are selected.
- Baseline, MDE, power, sample size, and duration are recorded.
- Event logging, denominators, and identity are QA’d.
- Traffic allocation and SRM diagnostics are available.
- Stopping and multiple-comparison rules are written down.
- Rollout, rollback, and post-launch monitoring plans exist.
FAQ
Is an A/B test the same as A/B testing?
Not exactly. A/B testing is the methodology; an A/B test is one experiment using that methodology.
Is an A/B test always 50/50?
No. Equal allocation is common for efficiency, but staged rollouts may use unequal exposure. Keep the planned allocation explicit and stable.
Can an A/B test have more than two variants?
Yes. That is usually called A/B/n testing. More arms require more traffic and a multiple-comparison plan.
What if the test is inconclusive?
Check whether the experiment reached its planned power, inspect the interval and data quality, and distinguish “no evidence of a meaningful effect” from “proof of no effect.”
When should I not run an A/B test?
Do not use it when traffic is too low for a decision-useful design, the change is mandatory, the outcome takes too long to observe, or qualitative research better answers the question.
Summary
An A/B test is a single, documented decision experiment. Its credibility depends less on the visual polish of the variant than on stable assignment, a meaningful primary metric, adequate planning, valid analysis, and disciplined interpretation. A result can be a win, loss, or inconclusive—and all three can produce useful learning.
Sources
- GrowthBook: What is A/B testing?
- Statsig: Methods, metrics, and best practices
- Optimizely: A/B testing
- UBC Statistical Modelling: A/B Testing
- Choosing a randomization unit
- Statsig: Multiple comparisons
- Optimizely: MDE and experiment prioritization
- GrowthBook: Open Guide to A/B Testing