What is a controlled experiment?
A controlled experiment tests whether changing an independent variable causes a change in a dependent variable. The researcher—or an experimentation system—assigns units to conditions rather than merely observing which condition they happen to receive.
The control group provides a baseline for the question: what would have happened to the treated units without the intervention? That counterfactual cannot be observed for the same unit simultaneously. A comparable control group approximates it at the population level.
In digital products, an A/B test is a common controlled experiment. In medicine, it may be a randomized controlled trial. In agriculture, it may compare fertilizer treatments. In education, it may evaluate teaching methods. The setting changes; the logic remains comparison under assigned conditions.
Core components
| Component | Meaning | Practical question |
|---|---|---|
| Independent variable | Factor deliberately changed | What exactly is being manipulated? |
| Dependent variable | Outcome measured | What will indicate an effect? |
| Control condition | Baseline, standard, placebo, or no treatment | What is the credible counterfactual? |
| Treatment condition | Intervention being evaluated | What is different from control? |
| Experimental unit | Entity receiving a condition | Person, account, plant, store, request, or region? |
| Random assignment | Procedure allocating units to conditions | Could assignment be influenced by outcome risk? |
| Control variables | Conditions held stable or measured | What else could explain the outcome? |
| Decision rule | Predefined evidence threshold and action | What will we do with positive, negative, or null results? |
A controlled experiment design
- State a testable hypothesis.
- Define the target population and experimental unit.
- Specify control and treatment conditions.
- Choose a randomization method and allocation.
- Define the dependent variable and measurement window.
- Control, block, stratify, or record relevant conditions.
- Plan sample size, power, and stopping rule.
- Run the experiment concurrently and consistently.
- Check data quality before interpreting the outcome.
- Estimate effect, uncertainty, limitations, and practical value.
Why random assignment matters
Random assignment gives each unit a known chance of entering each condition. It does not make groups identical in every finite sample. It makes systematic pre-existing differences less likely, including differences that are difficult or impossible to observe.
LibreTexts describes random assignment as the key mechanism for preventing treatment groups from differing systematically in lurking variables [1]. GrowthBook makes the same point for digital products: randomization is the practical way to balance factors that cannot be physically held constant [2].
Randomized estimate ≈ mean(Y | Treatment) − mean(Y | Control)
Here, Y(1) is the outcome under treatment and Y(0) is the outcome under control. The same unit has only one observed outcome, so the control group is essential to estimate the missing counterfactual.
Example 1: product experiment
A SaaS team wants to test whether a guided onboarding checklist increases activation.
| Design element | Specification |
|---|---|
| Hypothesis | If new users receive a checklist, more will complete the activation event because the next action is clearer. |
| Unit | New account, because all members should see the same onboarding. |
| Control | Current onboarding flow. |
| Treatment | Checklist with three recommended actions. |
| Primary outcome | Activated accounts within seven days. |
| Guardrails | Support contacts, setup errors, time to first value. |
| Analysis | Intent-to-treat at account level. |
If control activation is 32% and treatment activation is 35%, the absolute effect is 3 percentage points. The relative uplift is:
The uplift is only decision-ready after checking its interval, power, implementation, and guardrails. A higher activation rate paired with more abandoned accounts or support burden may not justify shipping.
Example 2: physical controlled experiment
A researcher asks whether fertilizer A increases plant growth relative to no fertilizer.
- Units: comparable plants.
- Control: no fertilizer or standard treatment.
- Treatment: a defined dose of fertilizer A.
- Randomization: plants assigned to conditions randomly, ideally within blocks of similar size or location.
- Controls: soil, water, light, container, observation period, and measurement procedure.
- Outcome: change in height or biomass.
Physical controls can hold many conditions constant. Randomization helps balance the conditions that cannot be perfectly controlled. The same design logic applies to a webpage, though the uncontrolled environment is more complex.
What does “control” mean?
The word control has several related meanings:
| Meaning | Explanation |
|---|---|
| Control group | Units receiving the baseline, placebo, standard, or no active treatment. |
| Control condition | The reference experience used to estimate the counterfactual. |
| Control variable | A factor held constant, blocked, stratified, or included in analysis. |
| Controlled study | A design where the researcher assigns conditions and manages alternatives. |
A control group is not the same as a control variable. The group provides a comparison. The variable is a condition managed to reduce alternative explanations.
Randomization, blocking, and blinding
Randomization
Simple randomization assigns units independently. Restricted or blocked randomization can improve balance in small studies or across known important factors. In digital experiments, persistent user-ID hashing is common; in a marketplace or region-level intervention, cluster or geo randomization may be more appropriate.
Blocking and stratification
Block or stratify when a known factor strongly affects the outcome and the sample is limited. For example, randomize separately within mobile and desktop strata. Do not over-segment: each added cell needs enough units and a plan for analysis.
Blinding
Blinding hides assignment from participants, investigators, assessors, or analysts. It reduces expectation and measurement bias. In many interface experiments, users can see the treatment and blinding is impossible; randomization, objective metrics, and automated logging remain essential.
Threats to validity
| Threat | How it threatens inference | Mitigation |
|---|---|---|
| Selection bias | Groups differ before treatment | Randomize; conceal allocation where relevant |
| History or seasonality | External event affects one period or group | Run concurrently; cover full cycles |
| Maturation | Units change naturally over time | Use concurrent control and consistent window |
| Measurement bias | Outcome is measured differently by arm | Blind assessors; QA instrumentation |
| Attrition / missingness | Dropout differs by condition | Track missingness; preserve intent-to-treat |
| Interference | One unit’s treatment affects another unit | Cluster, geo, or resource-level design |
| Noncompliance | Assigned units do not receive their condition | Log assignment and exposure separately |
Controlled experiment vs. observational study
In an observational study, the researcher observes exposure and outcome without controlling who receives the exposure. Groups may differ for reasons related to both exposure and outcome. Statistical adjustment can help, but it depends on assumptions about measured confounders.
In a controlled experiment, the researcher assigns the intervention. Randomization makes causal interpretation more defensible, but it does not guarantee perfect implementation, generalizability, or ethical suitability.
Limitations and when randomization is not possible
- Legal, ethical, or safety constraints may prohibit withholding an intervention.
- Policy changes or infrastructure events may not be assignable at individual level.
- Network, marketplace, or household spillovers can violate independence.
- Very low traffic may make a randomized test uninformative.
- Long-lag outcomes may require holdouts, survival methods, or a different design.
- Artificial conditions can reduce external validity.
When individual randomization is impossible, consider cluster randomized trials, geo experiments, switchbacks, difference-in-differences, synthetic controls, or regression discontinuity. These are not interchangeable; each requires explicit assumptions.
Quality checklist
- The hypothesis identifies a manipulable cause and measurable outcome.
- Control and treatment conditions are precisely documented.
- Experimental, randomization, and analysis units are compatible.
- Assignment is random, persistent, and auditable.
- Sample size, MDE, power, alpha, and duration are planned.
- Primary, secondary, guardrail, and diagnostic metrics are separated.
- Measurement is symmetric and missingness is monitored.
- Interference, seasonality, carryover, and novelty are considered.
- Blinding or objective automated measurement is used where possible.
- Results include effect size, uncertainty, practical significance, and limitations.
FAQ
What is the purpose of a controlled experiment?
To isolate the effect of an intervention by comparing assigned conditions while reducing alternative explanations.
Does a controlled experiment always require randomization?
A fully randomized controlled experiment does. Some controlled designs use blocking or deliberate balancing, but if assignment is not random, the causal assumptions and terminology should be stated clearly.
What is the difference between a control group and a treatment group?
The control receives the baseline or comparison condition. The treatment receives the intervention being evaluated.
What is the difference between an independent and dependent variable?
The independent variable is manipulated; the dependent variable is measured as the outcome.
Can a controlled experiment prove causation?
It can provide strong evidence for causation when randomization, implementation, measurement, and analysis are valid. No single experiment removes every uncertainty.
Summary
A controlled experiment turns an interesting difference into a defensible causal question. Its strength comes from the whole design: an explicit intervention, credible control, appropriate units, random assignment, consistent measurement, and a decision rule. If any link is weak, the result should be interpreted as limited evidence rather than certainty.
Sources
- GrowthBook: What is a controlled experiment?
- LibreTexts: Causation and Experiments
- Simply Psychology: Controlled Experiment
- Scribbr: Control Groups and Treatment Groups
- Urban Institute: Experiments and RCTs
- Causal Wizard: Experiment
- Analytics ToolKit: Online Controlled Experiment