What is a control group?
A control group is the reference condition in an experiment. It should be exposed to the same general environment as the treatment group, except for the intervention being tested. In a digital experiment, that usually means the current production experience. In a clinical trial, it might be standard care or a placebo. In an education study, it might be the existing teaching method.
Analytics ToolKit describes the online control group as randomly assigned users, sessions, or other units that are not exposed to the experimental treatment [1]. Britannica defines it as the standard for comparison against which experimental outcomes are evaluated [2].
The control group does not need to have exactly the same individual users as the treatment group. It needs to be comparable in aggregate. Random assignment is the usual way to make systematic differences between the groups unlikely.
Why the control group matters
Imagine a product team releases a new onboarding flow and activation rises from 30% to 34% over the next two weeks. That sounds positive, but several other things may have changed: an ad campaign brought higher-intent users, a seasonal promotion was launched, the app became faster, or a competitor had an outage.
A concurrent control group experiences those same calendar conditions without the new onboarding flow. If the control remains at 30% and treatment reaches 34%, the 4 percentage-point difference is evidence about the intervention rather than the entire time period.
Relative uplift = (Treatment − Control) / Control × 100%
A control group helps with:
- seasonality and day-of-week effects;
- traffic-source and audience changes;
- product releases happening elsewhere;
- natural improvement or regression to the mean;
- measurement of incremental impact;
- statistical comparison and uncertainty.
How a control group works in an A/B test
Types of control groups
| Type | What it receives | When it is useful |
|---|---|---|
| Current-experience control | Existing product or process | Most website, app, and product A/B tests |
| No-treatment control | No active intervention | When withholding the change is ethical and realistic |
| Placebo control | Inactive experience that resembles treatment | Separating treatment effect from expectation |
| Active control | Established treatment or standard of care | Comparing a new option with an accepted alternative |
| Holdout control | Excluded from a program or set of treatments | Measuring cumulative or incremental program impact |
| Positive control | Known-effective intervention | Checking that the experimental setup can detect an expected effect |
| Historical control | Past data from a comparable period | When a concurrent control is infeasible; weaker causal design |
In ordinary online experimentation, “control” usually means the current experience, not an empty or inactive page. Removing a useful feature from the control can create an unfair comparison.
Example: an A/B test for a pricing page
A SaaS team tests a simplified pricing page. Control users see the current five-plan layout. Treatment users see a three-plan layout with clearer plan recommendations.
| Metric | Control | Treatment |
|---|---|---|
| Eligible users | 20,000 | 20,000 |
| Trial starts | 1,000 | 1,080 |
| Trial conversion | 5.0% | 5.4% |
| Absolute difference | 0.4 percentage points | |
| Relative uplift | 8.0% | |
The treatment has a higher trial-start rate, but the team still checks paid conversion, average revenue per account, support questions, refund requests, and retention. The control makes it possible to estimate the incremental effect of the simplified layout during the same period.
How to choose a control
A good control answers the counterfactual question for the decision at hand. Use these checks:
- Same population: control and treatment are eligible under the same rules.
- Same time: both run concurrently wherever possible.
- Same measurement: events, denominators, attribution, and windows are identical.
- Single intended difference: unrelated releases do not affect only one arm.
- Stable assignment: a user does not drift between conditions.
- Realistic baseline: the control represents what the business would actually keep.
- Enough units: both arms can support the planned effect and power.
Randomization and control-group quality
Randomization assigns each eligible unit to a condition without using its likely outcome. It helps distribute both visible and hidden confounders between the groups. It does not guarantee identical groups in a small sample, and it cannot repair assignment, exposure, or logging problems.
Check the control and treatment for:
| Quality check | What to look for |
|---|---|
| Sample ratio | Observed allocation matches planned allocation; no SRM |
| Identity | Users remain in their assigned condition |
| Exposure | Users assigned to control actually receive the control experience |
| Baseline balance | Important pre-treatment characteristics are not systematically imbalanced |
| Instrumentation | Events and denominators are logged symmetrically |
| Interference | Treatment users do not materially change control users’ outcomes |
If a sample-ratio check fails, stop interpreting the impact result until the cause is understood. A large sample and a small-looking allocation error can still be evidence that the experiment is not comparing the intended populations.
Control group vs. holdout group
These terms are related but not identical.
| Control group | Holdout group | |
|---|---|---|
| Scope | One experiment | Usually a program or portfolio of treatments |
| Duration | Test window | Longer period, often months |
| What it avoids | The treatment in that experiment | All or a defined set of winning treatments |
| Question | What did this change cause? | What did the overall program cause? |
| Example | Current checkout vs new checkout | Users excluded from all lifecycle campaigns |
Do not use a program-level holdout as a substitute for the control in an individual experiment. The holdout answers a different question.
Control group vs. control variable
A control group is a set of experimental units. A control variable is a condition kept constant, balanced, stratified, or measured across groups.
Common control-group mistakes
- Using a before/after baseline as the only control: time and seasonality are mixed with treatment.
- Changing the control during the test: the reference condition is no longer stable.
- Giving control users a different measurement path: comparison becomes a logging test.
- Using a non-random “control”: the groups differ before treatment.
- Reassigning returning users: users experience both conditions.
- Ignoring external campaigns: marketing or product changes affect one arm differently.
- Choosing a control that is not the real baseline: the result does not answer the shipping decision.
- Optimizing only the primary metric: revenue, quality, latency, or retention can deteriorate.
Control-group checklist
- Control condition is named and versioned.
- Control represents the experience that would remain without the treatment.
- Eligibility rules are identical across arms.
- Assignment is random and persistent.
- Control and treatment run concurrently.
- Exposure and outcome events use the same definitions.
- Sample-ratio, identity, and instrumentation checks are active.
- Primary, guardrail, and downstream metrics are documented.
- Interference and concurrent experiments are considered.
- Any ramp or allocation change is planned and logged.
- The analysis reports effect size and uncertainty, not only a winner label.
FAQ
What does a control group see in an A/B test?
Usually the current or original experience. It should not receive the experimental change, but it should remain eligible for the same surrounding product, marketing, and measurement conditions.
Does a control group receive nothing?
Not necessarily. It may receive the current experience, standard treatment, placebo, or no active treatment depending on the question.
Why not compare results before and after launch?
A before/after comparison cannot separate the treatment from seasonality, traffic mix, other releases, or natural change. A concurrent randomized control is usually stronger.
How large should a control group be?
For a standard two-arm test, equal allocation is often statistically efficient. A risk-sensitive rollout may use unequal allocation, but sample size and power must be recalculated.
Can there be more than one control group?
Yes. A study may compare a new treatment with no treatment, standard care, and a placebo, or use multiple baselines. Each comparison should be planned before analysis.
Sources
- Analytics ToolKit: Control Group
- Britannica: Control Group
- Editage: Control Group
- Scribbr: Control Groups and Treatment Groups
- Mida: Control Group in A/B Experimentation
- GrowthBook: Controlled Experiment
- Control Group Do’s and Don’ts