What is a treatment group?
A treatment group is a group of users, accounts, requests, regions, patients, plants, or other experimental units that receives one defined condition in an experiment. The “treatment” is not necessarily medical; it means the intervention or level of the variable the team wants to evaluate.
In a digital product, treatment might be a redesigned checkout, a new recommendation model, a different pricing message, an onboarding checklist, or a feature enabled behind a flag. In a study with several variants, each variant can have its own treatment group.
Mida defines the treatment group in A/B experimentation as the users exposed to the new design, copy, pricing, or other change being tested [1]. Statsig emphasizes that treatment and control must be assigned randomly and measured consistently to support causal interpretation [2].
What the treatment group does in an experiment
| Role | What it means in practice |
|---|---|
| Receives the intervention | The group sees the new condition the hypothesis is about. |
| Provides an outcome estimate | Its behavior is measured using predefined metrics. |
| Creates a comparison | Its outcome is compared with the control group. |
| Tests a mechanism | Driver metrics show whether the expected behavior changed. |
| Reveals trade-offs | Guardrails show whether the change harmed quality, speed, or retention. |
The treatment group does not “prove” an idea by itself. A treatment conversion rate of 5.4% has no causal meaning without knowing what a comparable control group achieved during the same period.
Treatment vs. control in an A/B test
The treatment group should differ from control in the intended way—not because it received different traffic, tracking, timing, eligibility, or unrelated releases. This is why assignment, exposure, and measurement need to be checked separately.
Example: testing a new product-page layout
An ecommerce team believes that moving reviews closer to the product summary will reduce uncertainty and increase purchases.
| Design element | Treatment-group specification |
|---|---|
| Population | Visitors who view an eligible product page |
| Assignment | Persistent user-level randomization |
| Control | Current product-page layout |
| Treatment | Same page with reviews moved beside the product summary |
| Primary metric | Purchase rate per eligible user |
| Driver metric | Add-to-cart rate |
| Guardrails | AOV, refunds, page latency, support contacts |
After the planned run, control has 10,000 users and 240 purchases; treatment has 10,000 users and 270 purchases.
Treatment rate = 270 / 10,000 = 2.7%
Absolute effect = 2.7% − 2.4% = 0.3 percentage points
Relative uplift = 0.3% / 2.4% × 100% = 12.5%
The 12.5% relative uplift is an estimate. The team still needs an interval, a statistical decision rule, SRM and instrumentation checks, and a review of AOV and refunds before shipping.
How treatment groups are formed
Random assignment is the standard method. Each eligible unit receives a known chance of entering the treatment or control group. The goal is not to create identical individuals; it is to avoid systematic differences that would explain the outcome independently of the intervention.
| Assignment unit | Use case | Treatment-group concern |
|---|---|---|
| User | Personalized web or app experience | Stable identity across sessions and devices |
| Account | Shared B2B workspace | All members need the same treatment |
| Request | Independent API recommendation | Repeated requests may not be independent for user outcomes |
| Region | Geo experiment with spillover | Fewer units and cluster-aware analysis |
| Session | Short-lived experience | Returning users may see both conditions |
The treatment group is not formed by picking the most engaged users, volunteers, or users who already tried the feature. That would create selection bias. Assignment to treatment is a method; treatment group is the resulting set of units.
Assigned treatment vs. received treatment
In real products, assignment does not guarantee exposure. A user can be assigned to a new feature but never visit the relevant screen. A treatment can fail to render. A user can be assigned to B but encounter an error and fall back to A.
| State | Meaning | Why it matters |
|---|---|---|
| Assigned to treatment | Experiment system selected the treatment | Defines intent-to-treat population |
| Exposed to treatment | User encountered the changed experience | Useful for diagnostics; filtering can introduce bias |
| Used treatment feature | User actively interacted with it | Behavioral compliance, not random assignment |
| Converted | User completed the outcome | Measured response variable |
Multiple treatment groups
An experiment can have several treatment groups. For example, a pricing test may compare the current page with annual-first, monthly-first, and usage-based layouts. Each treatment group receives one complete condition.
| Arm | Experience | Comparison |
|---|---|---|
| A | Current pricing page | Control baseline |
| B | Annual plan emphasized | B vs A |
| C | Usage-based pricing explained | C vs A |
| D | Three-plan recommendation | D vs A |
Adding treatment groups spreads traffic and increases multiple-comparison risk. Predefine the comparisons, sample size per arm, and correction method before looking at results. The arm with the highest observed metric is not automatically a winner.
Metrics for a treatment group
| Metric type | Question | Example |
|---|---|---|
| Primary | Did the intervention achieve its main objective? | Purchase conversion |
| Driver | Did the expected mechanism move? | Add-to-cart rate |
| Guardrail | Did the change create unacceptable harm? | Refund rate, latency, churn |
| Quality | Can we trust the treatment comparison? | Exposure, event completeness, SRM |
| Segment | Does the effect differ by relevant population? | Mobile vs desktop, new vs returning |
Use the same definitions and windows for control and treatment. A new event that exists only in the treatment group cannot be used as a neutral comparison without careful design.
Noncompliance and treatment effect
Some users assigned to treatment do not receive or use the treatment; some control users may find an alternative route to it. This is called noncompliance.
With full compliance:
Observed treatment effect ≈ effect of assignment when assignment is implemented correctly
When compliance is imperfect, the intent-to-treat effect answers the operational question: what happens when we assign eligible users to this rollout? A per-protocol or “treated users only” estimate answers a different question and can be biased because actual use is not random. For advanced cases, encouragement designs and instrumental-variable methods can estimate effects among compliers; Spotify discusses this distinction in the context of online experiments [3].
Common treatment-group mistakes
- Non-random selection: assigning treatment to users who look most likely to benefit.
- Unstable assignment: the same user moves between treatment and control.
- Different eligibility: treatment and control do not come from the same population.
- Mixing treatment changes: several unrelated interventions make attribution impossible.
- Ignoring delivery failures: treatment is assigned but does not render.
- Filtering on treatment usage: post-treatment behavior is mistaken for a neutral subgroup.
- Using the wrong unit: user treatment is analyzed as independent page views.
- Ignoring guardrails: conversion rises while revenue quality or retention falls.
- Calling every treatment a winner: multiple variants need a planned comparison and correction.
Treatment-group checklist
- Intervention is specified precisely and versioned.
- Population and eligibility are the same as control.
- Randomization unit matches how the treatment spreads.
- Assignment is random, persistent, and logged.
- Exposure is logged separately from assignment.
- Primary, driver, guardrail, and quality metrics are defined.
- Sample size, MDE, power, duration, and stopping rules are planned.
- SRM, identity, rendering, errors, and latency are monitored.
- Intent-to-treat and any triggered analysis are clearly separated.
- Multiple treatment groups have planned comparisons.
- Rollout and rollback criteria are documented.
FAQ
What is a treatment group in A/B testing?
It is the group assigned to receive the new variation or experimental condition. The control group receives the current or baseline experience for comparison.
Is the treatment group the same as the experimental group?
Usually. “Experimental group” is a broader term; “treatment group” emphasizes that the group receives a defined intervention.
Can there be several treatment groups?
Yes. A/B/n tests have one control and multiple treatment groups. Each additional group requires enough traffic and a multiple-comparison plan.
Does everyone in the treatment group need to use the feature?
No. Assignment and actual use are different. The primary intent-to-treat analysis generally keeps everyone assigned in the analysis population.
How large should the treatment group be?
For a simple low-risk A/B test, a 50/50 split is often efficient. The correct size depends on baseline, MDE, power, alpha, allocation, number of arms, and risk.
What should I do if treatment is assigned but not displayed?
Log the failure, compare assigned and exposed populations, investigate the delivery path, and avoid silently treating only successful exposures as the treatment group.
Sources
- Mida: Treatment Group
- Statsig: Treatment vs Control
- Spotify Engineering: Encouragement Designs
- Fiveable: Treatment Group
- Statistics How To: Experimental Group
- PlainSignal: Treatment Group in Analytics
- Analytics ToolKit: Control Group