What randomization means in an experiment
Randomization is not the same as “doing something arbitrarily.” It is a defined assignment mechanism. If a test is 50/50, each eligible unit should have a 50% chance of entering either condition. If it is 80/20, the probabilities should match that plan.
Analytics ToolKit describes randomization as assigning subjects to treatment groups using chance methods, usually with equal probabilities when group sizes are intended to be equal [1]. The point is to prevent assignment from depending on location, device, browser, previous behavior, or another factor related to the outcome.
Statsig calls randomization a cornerstone of A/B testing: if the samples are not random, even a statistically significant result cannot confidently be attributed to the change being tested [2].
Why randomization supports causality
Suppose a team gives a new checkout only to returning users. Returning users may convert more often anyway. The higher rate cannot be separated from the effect of the checkout.
Random assignment makes this kind of systematic selection less likely. It does not make groups identical person by person. It makes the assignment independent of the potential outcome in expectation, so differences in outcomes can be interpreted as treatment effects under the design assumptions.
Randomization also supports the statistical model used after the experiment. Many tests assume that assignment was not chosen by the subject, analyst, or product team in response to outcome risk.
Four requirements for a reliable assignment mechanism
| Requirement | What it means | Failure example |
|---|---|---|
| Uniformity | Each unit has the planned probability of each arm | Hash range maps 60% of IDs to treatment |
| Consistency | The same unit keeps its assignment | User sees control today and treatment tomorrow |
| Independence | Assignment in one experiment does not predict another | Same IDs repeatedly land in control across tests |
| Monotonic ramp-up | Increasing exposure does not reassign existing users | Moving from 10% to 50% reshuffles treatment users |
These are platform properties, not just analyst preferences. They need automated tests and monitoring before a team trusts a result.
Choose the randomization unit first
Randomization unit is the entity assigned to a condition. It may be a user, account, device, session, request, store, region, household, or organization.
| Unit | Good fit | Watch out for |
|---|---|---|
| User | Multi-session product and user-level outcomes | Anonymous identity, cookie resets, cross-device users |
| Account | Shared B2B workspaces and team features | Fewer independent units |
| Session | Short, self-contained interaction | Returning users may experience both arms |
| Request | Independent recommendations or API responses | User-level outcomes may be correlated |
| Region or cluster | Marketplace, network, or geo spillover | Cluster-aware inference and lower power |
For a checkout test, user-level assignment is usually safer than randomizing every page view. A customer who sees a different checkout on every visit is not experiencing a clean treatment, and repeated sessions are not independent observations.
Ways to randomize users
Simple randomization
Each arriving unit is assigned using a random number generator. It is straightforward and works well for large samples when no special balance is needed. Small samples can temporarily look uneven, which is normal.
Block randomization
Assignments are shuffled within blocks—for example, five control and five treatment slots in every block of ten. This keeps group sizes closer during a short experiment, but the block construction must not leak assignment or create patterns.
Stratified randomization
Users are divided into meaningful strata, such as mobile/desktop or paying/non-paying, then randomized within each stratum. It can improve balance when a known factor strongly affects the outcome, but too many strata create sparse cells.
Seeded or rerandomized assignment
Several candidate random splits are generated and historical covariate balance is checked before launch. The team selects a seed that meets predefined balance rules. This requires historical data and must avoid selecting a seed based on the outcome being tested.
Hashing and stable assignment
Many online systems use a stable identifier plus experiment-specific salt:
if bucket < 5,000 → Control
otherwise → Treatment
This is deterministic but random-like: the same ID and salt produce the same assignment, while a new experiment salt creates a new map. GrowthBook documents consistent hashing as a way to keep assignment stable across pages and applications without storing state [3].
Do not use a language’s default hash function without testing it. Some functions are not stable across processes or versions, and some can produce correlations between experiments. Microsoft’s controlled-experiment research found that uniformity and independence need to be tested, not assumed [4].
Example: why time-based splitting fails
A team runs a two-week test. During the first week, all visitors see control. During the second week, all visitors see treatment. Treatment conversion is higher.
The difference could be due to:
- a weekend or weekday mix;
- a marketing campaign;
- an outage at a competitor;
- seasonality;
- a product release;
- different traffic quality.
The team has run a before/after comparison, not a randomized concurrent experiment. Assigning users throughout both weeks would expose control and treatment to the same external conditions.
Worked example: stratified assignment
A mobile app tests an onboarding change. Paying users are rare but have much higher revenue. A simple split happens to place more paying users in treatment.
| Stratum | Control | Treatment | Reason |
|---|---|---|---|
| Paying users | 50% | 50% | Balance high-value users |
| Non-paying users | 50% | 50% | Balance the larger population |
Randomizing inside both strata makes the mix easier to compare. The analysis still needs to respect the design and report the overall effect according to the prespecified population. Stratification is not a license to create dozens of post-hoc slices.
Independence across concurrent experiments
A user’s assignment in one experiment should not make them more likely to receive treatment or control in another, unless the experiments are deliberately layered or factorial.
Reusing a fixed user-ID split, a poor hash, or overlapping bucket ranges can create correlations. A treatment in Experiment 1 may then appear to improve Experiment 2 simply because its group contains more high-value users.
- Use an experiment-specific salt.
- Use layers or mutual-exclusion groups when experiences interact.
- Log all active assignments.
- Test pairwise assignment independence on simulated or historical IDs.
- Do not silently reuse the same audience partition for unrelated decisions.
How to check randomization
| Diagnostic | Question | Typical method |
|---|---|---|
| Allocation | Do observed counts match the intended split? | SRM or chi-square goodness-of-fit |
| Covariate balance | Are pre-treatment characteristics comparable? | Balance tables and predefined checks |
| Persistence | Does one unit stay in one arm? | Assignment history by user/account |
| Uniformity | Does the assignment algorithm distribute IDs evenly? | Simulation over large ID sets |
| Independence | Are assignments independent across experiments? | Cross-tabulation and interaction tests |
| Exposure | Does assignment lead to actual experience? | Assignment-to-exposure funnel |
An A/A test is useful here: both arms receive the same experience, so any persistent metric difference or allocation problem points to the platform, data path, or analysis rather than the treatment.
Randomization is not random sampling
These concepts are often confused:
| Concept | What is randomized? | What it supports |
|---|---|---|
| Random sampling | Who enters the study from a wider population | Generalizing to that population |
| Random assignment | Which condition a selected unit receives | Causal inference between conditions |
You can randomly assign users from a non-random website audience. That supports a causal comparison for the tested audience, but not necessarily generalization to all internet users. Likewise, a random sample without random assignment does not make treatment and control causal.
Common randomization mistakes
- Assigning by time: Monday users get A and Tuesday users get B.
- Assigning by geography or device intentionally: the groups inherit different behavior.
- Rolling a new random number on every request: users switch experiences.
- Using unstable IDs: cookies, anonymous IDs, and login IDs do not reconcile.
- Reusing the same seed: concurrent assignments become correlated.
- Changing allocation and rebucketing: previously assigned users move arms.
- Ignoring assignment failure: fallback behavior is not logged.
- Checking only aggregate counts: a mobile-only imbalance remains hidden.
- Over-stratifying: too many cells reduce power and complicate analysis.
Randomization checklist
- Randomization unit matches treatment spread and analysis unit.
- Assignment probabilities are documented.
- Identifier is stable and privacy-appropriate.
- Experiment-specific salt or seed is used.
- Assignment remains persistent.
- Exposure is logged separately from assignment.
- Allocation and SRM checks are automated.
- Uniformity and cross-experiment independence are tested.
- Stratification or blocking is used only for a defined reason.
- Ramp-up preserves existing assignments.
- Concurrent tests, exclusions, and dependencies are documented.
FAQ
Why is randomization important in A/B testing?
It reduces systematic differences between groups, making it more credible that an observed outcome difference came from the treatment rather than who received it.
Is randomization the same as random sampling?
No. Random sampling selects units from a population; random assignment places selected units into experiment conditions. Random assignment supports causality; random sampling supports generalization.
Should I randomize users or sessions?
Randomize at the level that keeps the experience coherent and matches the outcome. User-level assignment is usually safer for multi-session behavior; session-level assignment can be appropriate for isolated interactions.
Is hashing randomization?
A well-designed salted hash creates deterministic, approximately uniform assignments. It is not random in the sense of changing on every request; that stability is the point.
What if the groups are not exactly 50/50?
Small deviations are expected. Use an SRM check to determine whether the difference is larger than random variation can reasonably explain. Investigate significant mismatch before interpreting impact.
When should I use stratification?
Use it when a known pre-treatment factor strongly affects the outcome or the sample is small enough that imbalance would matter. Keep the number of strata manageable and predefine the analysis.
Sources
- Analytics ToolKit: Randomization
- Statsig: Randomization in A/B Testing
- GrowthBook: Experiment Assignment
- Kohavi et al.: Controlled Experiments on the Web
- Growth-onomics: Randomization Algorithms
- Choosing a Randomization Unit
- Lyssna: Randomization in Research
- Thumbtack Engineering: Sample Balance