Quick definition: Experiment contamination occurs when a control unit receives, is affected by, or can infer treatment—or when treatment is mixed across units—so the intended contrast between experimental arms is weakened or distorted.
What is experiment contamination?
Experiment contamination is a threat to a clean A/B-test comparison. It happens when treatment effects cross assignment boundaries. A control user may see a screenshot of a treatment offer, a returning visitor may be reassigned to a different variant, colleagues in the same account may receive inconsistent settings, or one arm may change shared inventory available to both arms. In each case, the observed difference no longer represents the intended contrast between a pure control and a pure treatment.
Contamination is related to interference but the terms are not identical. Contamination usually emphasizes unintended mixing or crossover of treatment exposure. Interference is broader: one unit’s assignment can affect another unit’s outcome even if the second unit never directly sees the variant. A referral feature, marketplace ranking, ad auction, and shared support queue can all create interference. Both violate the simple assumption that each unit’s outcome depends only on its own assignment.
It should also be distinguished from ordinary exposure variation. Some assigned users will not visit a tested page; that does not automatically contaminate an assignment-based experiment. Contamination arises when the experience or its effect leaks across arms in a way that changes the comparison. The practical consequence can be dilution toward zero, an exaggerated effect, or an effect that answers a different question from the one the team intended.
How contamination affects A/B-test mechanics
In a standard A/B test, a stable identifier assigns each eligible unit to one version. The causal estimate compares average outcomes in treatment and control. If control users also receive part of treatment, their outcome may move toward treatment, reducing the measured difference even when the treatment is genuinely valuable. Conversely, if a treatment causes scarcity, social proof, or operational load that harms control users, the difference can grow for reasons beyond the direct experience.
Cross-device identity is a common source. A visitor assigned by browser cookie can later sign in on another device and receive a different version. Shared accounts create a similar problem when individual users receive a setting that all team members can observe. Session-level randomization can contaminate a persistent workflow because repeat sessions alternate versions. These failures are often preventable through a stable user or account key, but the correct key must be available before treatment.
Shared resources are harder. A promotion shown to treatment customers can deplete inventory for control customers. A new delivery promise can increase fulfillment load and delay both arms. A social recommendation can alter what friends in control see. In these cases, merely fixing identifiers is insufficient. The design may need randomization by store, market, team, or time period, a persistent holdout, explicit saturation measurement, or a different causal question.
Contamination does not always make a test useless, but it changes interpretation. The team should estimate its likely direction and magnitude, report it, and avoid claiming a pure individual-level treatment effect when the data represent an ecosystem-level effect under partial mixing.
Scenario: referral incentives in a social product
A social app tests a referral invitation flow. Individual users are randomized: treatment sees a new invitation message with a reward; control sees the existing flow. The primary metric is successful referrals per assigned user. Soon after launch, analysts notice that many referred friends land in the product and interact with inviters regardless of the inviter’s assignment. Some control users are also reached by treatment users’ invitations and may increase activity because their network becomes more active.
This is not just a tracking issue. The treatment changes the environment in which controls operate. A simple user-level comparison could underestimate or misstate the total value of the invitation flow. The team might redesign the test by randomizing communities or geographic clusters, measuring network saturation, or using an explicit holdout where no members receive the new program. The choice depends on whether the business decision concerns the direct effect on inviters, the total network effect, or the effect of a realistic partial rollout.
| Contamination source | Potential bias | Mitigation |
|---|---|---|
| Variant switching on return visits | Mixed treatment dilutes or obscures behavior. | Use persistent assignment with a stable key. |
| Shared accounts or devices | One person’s treatment becomes another’s control exposure. | Randomize at the shared account or household boundary. |
| Inventory or capacity constraints | One arm changes availability or service quality for the other. | Use clusters, markets, capacity controls, or measure saturation. |
| Social sharing and referrals | Network effects cross individual assignments. | Use cluster designs or describe the estimand as partial-rollout impact. |
The referral test should not be “fixed” by deleting affected users after the fact if treatment caused the exposure. That is post-treatment filtering and can introduce a new bias. Better options are a redesigned experiment, a preplanned analysis of network exposure, or a qualified conclusion that acknowledges the tested rollout environment.
How to prevent and respond to contamination
- Map treatment pathways: list direct recipients, shared settings, referrals, inventory, support processes, and other routes through which the change could cross arms.
- Choose the assignment boundary: randomize the smallest unit that can receive a coherent treatment without material leakage, such as account rather than user when collaborators share the feature.
- Make assignment persistent: use a stable, privacy-appropriate key and define behavior for login, logout, device changes, account merges, and anonymous traffic.
- Instrument possible mixing: log variant impressions, referral relationships, shared-resource state, fallback behavior, and cross-device identity where appropriate.
- Review design before launch: include product, engineering, data, and operations stakeholders who understand shared systems and customer workflows.
- Quantify and classify incidents: determine whether leakage is rare and symmetric, treatment-related, or large enough to invalidate the intended contrast.
- Adapt the claim or design: use cluster randomization, a holdout, saturation analysis, longer-term monitoring, or a new experiment rather than silently treating contaminated data as standard A/B data.
Prevention is usually cheaper than correction. A short design review that asks “can a control unit experience the treatment indirectly?” often reveals the critical boundary. It should include operational mechanisms, not only UI screens. An experiment can contaminate through a call center script, stock allocation, recommendation cache, delivery schedule, or notification policy even when the product interface is perfectly bucketed.
Limitations and common errors
Some contamination is unavoidable because real products are social and constrained. Eliminating it entirely may require assigning large clusters, which lowers the number of independent units and can make a test slow or expensive. A design must balance causal purity with feasibility. The correct response may be to estimate the impact of the actual rollout policy rather than an artificial isolated effect, as long as the estimand is stated honestly.
- Assuming a feature flag prevents all leakage: flags control code paths but do not stop screenshots, sharing, common accounts, or shared resource effects.
- Using session assignment for a persistent intervention: repeated visits can mix variants for the same user and alter learning behavior.
- Ignoring treatment density: a 5% rollout and a 50% rollout can create different spillovers, especially in networks and marketplaces.
- Dropping contaminated controls post hoc: this can select on treatment-caused behavior and worsen bias.
- Counting every transaction as independent: shared markets and account members produce correlated outcomes.
- Overgeneralizing a partial-rollout effect: a result at one treatment saturation may not predict a full launch.
Contamination can sometimes be diagnosed with auxiliary data: repeat-variant histories, account membership, proximity between clusters, inventory availability, referral graphs, or exposure to treated peers. Such diagnostics are rarely a complete repair. They are most valuable for learning which mechanism caused mixing and for designing the next valid comparison.
Frequently asked questions
Does contamination always bias results toward zero?
No. Direct crossover often dilutes a difference, but shared resources or network effects can make control outcomes better or worse. The bias direction depends on how treatment crosses assignment boundaries.
Is a user seeing both variants always contamination?
For a persistent user-level A/B test, usually yes. For a deliberately designed within-subject or sequential experience, it may be part of the protocol, but the analysis must account for order, carryover, and learning.
How does cluster randomization help?
It assigns whole groups—such as accounts, stores, or cities—to one arm, keeping members who share treatment or influence one another together. It reduces leakage but requires analysis and sample planning for clusters.
Can I exclude users exposed to a treated friend?
Not safely by default. If treatment created that friendship exposure, exclusion conditions on a post-treatment event. Define network exposure and the estimand before launch, or redesign the assignment unit.
What should a readout say when contamination is suspected?
Describe the evidence, likely pathway, affected scope, and how it limits interpretation. Report the result as applicable to the observed rollout environment, or state that it is not reliable enough for the intended decision.
Summary
Experiment contamination mixes treatment across intended A/B-test boundaries or lets one arm change the other’s outcomes. Prevent it with stable assignment, the right unit, and design review of shared systems. When it remains, measure it, adjust the causal claim or design, and never hide it behind a simple winner label.
Sources
- Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments
- U.S. National Library of Medicine: cluster randomized trials
- AB-Labz: Sample ratio mismatch in A/B testing
- AB-Labz: A/B versus A/B/C testing