Quick definition: An experiment unit is the entity assigned to a variant and counted in the analysis—such as a user, account, device, session, store, or market. It must match how treatment is experienced and how outcomes can be measured.
What is an experiment unit?
The experiment unit, also called the unit of randomization, is the entity that receives one experimental assignment. In a web A/B test it might be a visitor; in a collaborative product it may be an account or workspace; in an email test it may be a subscriber; in a delivery-policy test it can be a city or warehouse. The choice is a design decision, not an implementation detail, because it determines what causal effect the test estimates.
It is easy to confuse the experiment unit with the analysis unit, observation unit, or metric denominator. They can be the same, but not always. A workspace may be randomized to an onboarding flow while outcomes are recorded for its individual members. A store may be randomized to a promotion while transactions are observed per customer. A user can generate many page views, but treating each page view as independent when assignment is at user level overstates the amount of information.
The right unit is usually the smallest entity that can receive a stable, independent version without leaking it to another unit. It should be assigned before treatment, remain in the same arm across relevant visits, and support a defensible outcome denominator. When units influence one another through sharing, inventory, auctions, teams, or social networks, independence may fail and a larger unit or different design is needed.
How the experiment unit shapes an A/B test
Start with the intervention. A new button label visible only in one person’s interface can often be randomized by user or stable anonymous visitor. A billing setting shared by a company should be randomized by account, otherwise colleagues may see conflicting invoices. A delivery fee change that shifts demand across a region may require market-level assignment because customers compete for the same capacity. Randomizing too low in the hierarchy can cause contamination; randomizing too high can reduce the number of independent units and increase required runtime.
Assignment must be persistent. A deterministic hash of a stable unit identifier is common: the same eligible user or account returns to the same arm. Session-based assignment is appropriate only when the treatment is inherently session-scoped and carryover is not material. Switching a returning user between variants can create a mixed treatment, alter behavior, and make outcome attribution ambiguous.
The analysis needs to respect the assignment. If users are randomized, aggregate repeated events to a user-level metric or use a method that accounts for within-user correlation. If stores are randomized, the effective sample size is the number of stores, not the number of receipts. A test with millions of customer transactions but ten assigned markets may have little independent information. This is a frequent source of false confidence in clustered experiments.
Identity resolution matters too. A visitor can browse anonymously, sign in on another device, and belong to a shared account. Decide whether the experiment is about people, devices, or accounts; map identities consistently; and document how merges, logouts, and duplicate records are handled. Do not change identity rules after seeing a favorable result.
Scenario: testing a shared workspace setting
A B2B application considers enabling a new default notification digest for newly created workspaces. The feature affects everyone in a workspace: one member’s notification setting can change what collaborators expect and how administrators configure the account. Randomizing individual members would allow the same workspace to receive conflicting defaults. The team therefore randomizes workspaces, using the workspace ID as the stable key.
The primary metric is the share of eligible workspaces with at least two active members after 28 days. Supporting metrics include notification open rate, administrator setting changes, member activity, unsubscribe requests, and support contacts. The outcome is calculated once per workspace for the primary analysis even though the product logs many member-level events. This respects the randomization and answers a workspace-level decision: should the default be enabled for new workspaces?
| Possible unit | When it fits | Common risk |
|---|---|---|
| User | A personal experience follows one identified person. | Cross-device identity gaps or sharing treatment with colleagues. |
| Account or workspace | Settings, billing, permissions, or collaboration are shared. | Fewer independent units and unequal account sizes. |
| Session | The intervention is temporary and no carryover is expected. | Returning users see both variants or repeated observations inflate confidence. |
| Store, city, or market | Operations, supply, local ads, or policy have shared effects. | Too few clusters and strong between-cluster variation. |
If treatment improves member activity but the workspace-level primary metric is uncertain, the team should not substitute the member result as proof of success. It is a diagnostic clue. The primary result remains the evidence for the stated deployment decision, and a follow-up may be needed to test the mechanism or a narrower population.
How to choose an experiment unit
- State the causal question: specify whose outcome and whose experience the decision concerns.
- Map treatment boundaries: identify who can see, share, copy, or be affected by the assigned version.
- Choose the lowest safe level: use the smallest unit that avoids meaningful spillovers and can receive persistent assignment.
- Align eligibility and metrics: define both around the same unit or explain the aggregation and correlation method when they differ.
- Check identifier quality: ensure the assignment key is stable, available before exposure, privacy-appropriate, and consistently joined to outcomes.
- Estimate independent sample size: account for clustering, unequal unit size, baseline variation, and the number of available units—not just event volume.
- Test for contamination: audit whether a unit or its close neighbors can receive both arms, then adjust the design if needed.
The choice often involves a trade-off. User-level assignment has more units and can be statistically efficient, but it may be invalid for shared experiences. Cluster-level assignment reduces leakage but generally needs more clusters because people inside a cluster behave similarly. Teams should not choose the smallest unit solely because it makes dashboards look powerful. A precise estimate of the wrong contrast is not a useful result.
Limitations and common errors
No unit choice eliminates every problem. Users may share screenshots, referrals, or inventory even when they have separate assignments. Accounts can vary dramatically in size, which makes a simple average account effect different from a member-weighted effect. Markets can be exposed to local events that no randomization rule can balance perfectly in a small number of clusters. These limitations should shape the claim and the monitoring plan.
- Randomizing page views for a persistent change: the same person can see both arms, weakening the intended contrast and creating carryover.
- Analyzing events as independent units: repeated sessions or transactions from one assigned user create pseudo-replication.
- Ignoring cluster correlation: customer volume does not replace the number of independently assigned stores, teams, or markets.
- Using an unstable identifier: cookies can reset, devices can be shared, and anonymous IDs can merge after login.
- Mixing weighted and unweighted estimands: an average account outcome and an average user outcome can legitimately differ; choose one intentionally.
- Changing assignment boundaries mid-test: rekeying, migration, or altered eligibility can change who belongs in each arm and must be documented.
When individual randomization is impossible, alternatives include cluster randomized experiments, geographic tests, switchbacks over time, and persistent holdouts. Each makes additional assumptions about interference, time trends, or representativeness. The right response is not to pretend the design is a standard individual A/B test, but to choose analysis and caution appropriate to the unit.
Frequently asked questions
Can the experiment unit and metric denominator differ?
Yes, but the relationship must be explicit. For example, randomize workspaces and calculate a workspace-level primary metric from member activity. If analysis uses lower-level observations, it must account for correlation within the assigned workspace.
Should I randomize users or sessions?
Use users when the experience persists across visits or learning and consistency matter. Sessions can fit genuinely ephemeral changes, but returning-user carryover and repeated-session correlation need attention.
What is cluster randomization?
Cluster randomization assigns groups such as stores, schools, teams, or cities rather than individuals. It is useful when members share treatment or can affect one another, but it usually needs more independent clusters.
Why does stable assignment matter?
It keeps each unit’s treatment coherent. Without it, return visits can receive different variants, obscuring what caused later behavior and enabling contamination between arms.
How do large accounts affect the result?
They can dominate user-level totals while an account-level analysis weights every account equally. Select the weighting that matches the decision, report it clearly, and consider sensitivity analyses when size is highly skewed.
Summary
The experiment unit is the entity assigned to treatment and is central to causal validity. Choose it from treatment boundaries, persistence, interference, and the deployment decision; then align identifiers, metrics, and analysis with that choice. More observations do not compensate for a mismatched or contaminated unit.
Sources
- Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments
- U.S. National Library of Medicine: cluster randomized trials
- AB-Labz: Sample ratio mismatch in A/B testing
- AB-Labz: How to calculate sample size