What is an experiment?
An experiment is a structured test of a hypothesis. Unlike an observational analysis, where a team studies what happened naturally, an experiment deliberately changes a condition and measures the consequence. That intervention may be a new interface, recommendation model, pricing rule, onboarding flow, campaign message, or operational policy.
The central question is causal:
That alternative outcome is the counterfactual and cannot be observed for the same unit at the same time. A control group provides a concurrent estimate of it. GrowthBook defines a controlled experiment as a comparison between what happened under treatment and a credible estimate of what would have happened without it [1].
Experiment vs. related methods
| Method | Is treatment assigned? | Concurrent control? | Main use |
|---|---|---|---|
| Randomized controlled experiment | Yes, randomly | Yes | Estimate causal effect |
| A/B test | Yes, usually randomly | Yes | Digital controlled experiment |
| Before-and-after | Sometimes | No | Describe change over time |
| Observational study | No | Not necessarily | Study naturally occurring differences |
| Quasi-experiment | Not fully random | Often | Estimate effects when randomization is infeasible |
| Usability study | May show a prototype | Not necessarily | Understand interaction problems and “why” |
“Experiment” is the broad concept. “A/B test” is one common online implementation, not a completely separate scientific method.
The components of a trustworthy experiment
1. A decision or research question
Start with the decision, not the tool. Examples include: should the new checkout ship, does the ranking model improve completed sessions, or does the pricing page reduce qualified-lead conversion?
2. A minimal viable hypothesis
A useful hypothesis names the intervention, expected outcome, mechanism, and population:
then completed purchases will increase,
because uncertainty about fulfilment will fall for first-time shoppers.
A hypothesis should be falsifiable. “Make the experience better” is an aspiration, not an experiment hypothesis.
3. Experimental units and population
Define who or what can receive the intervention: users, accounts, sessions, requests, stores, regions, listings, or organizations. Choose a randomization unit that matches how the treatment spreads and how outcomes are shared. User-level assignment usually provides a consistent cross-session experience; account or cluster assignment may be necessary when members interact.
4. Control and treatment
The control is the baseline condition. The treatment is the intervention. Document the difference precisely: code version, content, eligibility, exposure event, rollout, dependencies, and expected mechanism.
5. Outcomes and evaluation criteria
Choose one primary outcome for the main decision. Add secondary metrics for explanation, guardrails for potential harm, and diagnostic metrics for experiment trustworthiness. Mature organizations may define an Overall Evaluation Criterion (OEC) that expresses the business objective across multiple outcomes, but it must be designed before launch rather than assembled from whichever metric moved.
6. Assignment and instrumentation
Randomization balances confounding influences in expectation. Instrumentation records assignment, exposure, user actions, outcomes, errors, latency, and relevant timestamps. Without reliable exposure and outcome logs, the statistical analysis is only as good as the missing or asymmetric data allows.
7. An analysis and decision rule
Predefine baseline, MDE, power, alpha or Bayesian threshold, duration, primary comparison, stopping rules, treatment of missing data, and rollout criteria. The statistical result is the final link in a chain; it cannot repair weak experimental design.
The experiment lifecycle
Research on the online controlled experiment lifecycle groups the work into three broad phases: ideation, design and execution, and analysis and learning [2]. A useful operational version is:
| Phase | Key work | Output |
|---|---|---|
| Ideation | Problem discovery, evidence review, hypothesis, expected mechanism | Decision brief and minimal viable hypothesis |
| Design | Population, unit, variants, metrics, power, duration, risks | Pre-registered experiment plan |
| Build and QA | Implement assignment, treatment, logging, dashboards, alerts | Validated experiment ready to launch |
| Ramp and execute | Expose traffic gradually, monitor guardrails and quality | Clean observations under planned conditions |
| Analyze | Check SRM, data quality, primary metric, uncertainty, segments | Scorecard and decision |
| Learn | Record result, mechanism, limitations, follow-up, rollout | Institutional memory and next hypothesis |
The final phase is often neglected. A result that is not documented cannot improve future prioritization, metric definitions, or experiment design.
How to design an experiment
- Frame the problem: identify the user or business friction and supporting evidence.
- Define the decision: specify what will happen after positive, negative, or inconclusive evidence.
- Write the hypothesis: state change, expected effect, mechanism, and population.
- Select the unit: use user, account, request, region, or cluster according to treatment spillover.
- Choose metrics: primary, secondary, guardrail, and diagnostic.
- Plan power: estimate baseline, MDE, sample, allocation, and duration.
- Implement and QA: validate assignment, identity, exposure, events, rendering, and performance.
- Run and monitor: ramp risk-sensitive changes and stop only under the planned rules or critical guardrails.
- Analyze: inspect validity before interpreting impact.
- Ship and learn: roll out, roll back, or design a follow-up experiment.
Randomization and causal inference
Randomization gives each eligible unit a known chance of receiving each condition. It tends to balance observed and unobserved characteristics across arms within the limits of sample size. The observed difference then estimates an average treatment effect under the assumptions of the design.
That estimate is only meaningful if treatment assignment is credible, users remain in their assigned experience, outcomes are measured consistently, and treatment does not contaminate control. If users influence one another, share inventory, or compete for a fixed resource, interference can violate the simple individual-level model. In such cases, consider cluster, geo, time-based, or other specialized designs. The controlled-experiments literature explicitly identifies leakage and interference as threats to validity [3].
Metrics and the Overall Evaluation Criterion
| Metric class | Question it answers | Example |
|---|---|---|
| Success / primary | Did the intervention achieve its main objective? | Activated accounts per eligible user |
| Driver / secondary | What mechanism moved? | Form completion, feature adoption |
| Guardrail | Did we harm something we refuse to trade away? | Errors, latency, cancellations |
| Trust / diagnostic | Can we trust the comparison? | SRM, exposure rate, missing events |
Metrics should be normalized to the experimental unit where appropriate. For example, revenue per user is generally more interpretable than total revenue when arm sizes differ. Do not turn every metric into a winner criterion; multiple unplanned tests increase false discoveries and make decisions ambiguous.
Threats to experiment validity
| Threat | What it looks like | Mitigation |
|---|---|---|
| Sample Ratio Mismatch | Observed allocation differs from plan | Check assignment counts; investigate before reading impact |
| Peeking | Stopping when a p-value turns favorable | Fix horizon or use valid sequential inference |
| Carryover | Previous treatment affects later outcomes | Stable user assignment, washout, or specialized design |
| Interference | Treatment users affect control users | Cluster, geo, resource, or time-based randomization |
| Novelty | Short-lived reaction to a new experience | Run through adoption cycles; use long-term holdouts |
| Instrumentation asymmetry | Events or denominators differ by arm | QA raw logs and invariant metrics |
| Selection after treatment | Analyzing only users behaviorally triggered by treatment | Prefer intent-to-treat or pre-treatment eligibility |
Experiment vs. analytics
Analytics describes what users did and where patterns occur. An experiment deliberately changes a condition to estimate what the change caused. Both are necessary:
- Analytics can identify a funnel drop, segment, or anomaly worth investigating.
- Qualitative research can explain user intent, confusion, and unmet needs.
- An experiment can estimate whether a proposed intervention changes behavior.
- Post-experiment analysis can reveal mechanisms and new hypotheses—but subgroup discoveries should be treated as exploratory unless designed in advance.
An experiment does not answer every product question. It measures outcomes, not the full reason behind them; GrowthBook recommends combining experiments with interviews, surveys, session research, and usability work [4].
Expert experiment checklist
- The decision and problem are explicit.
- The hypothesis is falsifiable and mechanism-based.
- Population, eligibility, and randomization unit are defined.
- Control and treatment are documented precisely.
- Primary, secondary, guardrail, and diagnostic metrics are separated.
- Baseline, MDE, power, sample, and duration are planned.
- Assignment is random, persistent, and auditable.
- Exposure and outcomes are logged symmetrically.
- SRM, latency, errors, and missing events are monitored.
- Interference, carryover, novelty, and seasonality are considered.
- Stopping, multiple-testing, and missing-data rules are written before launch.
- Result, limitations, and follow-up are recorded in a searchable repository.
FAQ
What is an experiment in product analytics?
It is a deliberate intervention assigned to a defined population and evaluated against a measurable outcome, usually with a concurrent control. An A/B test is the most common online form.
Does every experiment need a control group?
For a strong causal comparison, a concurrent control is usually the default. Some designs use historical controls, switchbacks, geo controls, or quasi-experimental methods when a conventional control is impossible—but each carries additional assumptions.
What is the difference between an experiment and an A/B test?
Experiment is the broad methodological term. A/B test is a common digital experiment with two variants, typically control A and treatment B.
Can experiments prove causation?
Randomization supports causal inference by reducing confounding, but only when assignment, exposure, measurement, interference, and analysis are valid.
What makes an experiment trustworthy?
A credible control, appropriate randomization unit, clear hypothesis, reliable instrumentation, adequate sample, predefined outcomes and stopping rules, quality diagnostics, and a documented decision.
Summary
An experiment is not simply “trying a new version.” It is a disciplined learning system: define a counterfactual, intervene deliberately, randomize appropriately, measure outcomes, test trustworthiness, analyze uncertainty, and institutionalize what was learned. The best experimentation programs optimize not only pages or features, but the quality and speed of organizational learning.
Sources
- GrowthBook: What is a controlled experiment?
- The Online Controlled Experiment Lifecycle
- Kohavi, Tang & Xu: Trustworthy Online Controlled Experiments
- Google Research: Overlapping Experiment Infrastructure
- Analytics ToolKit: Online Controlled Experiment
- Practical Guide to Controlled Experiments on the Web
- Randomized Experiments and Causal Inference