Quick definition: A carryover effect occurs when exposure to one experimental condition changes a unit’s later behavior or outcomes under another condition, so the later measurement reflects both experiences rather than the assigned one alone.
What is a carryover effect?
Carryover is memory in an experiment. A customer who first learns a new navigation pattern may retain that knowledge after being reassigned to the old navigation. A user who receives a discount may buy later even after the discount disappears. A delivery delay caused by one policy can affect later retention measured under another policy. In each case, the prior condition persists into the next measurement period.
Carryover is especially important in crossover, switchback, repeated-measures, and time-based experiments, where the same unit, location, or system may experience several conditions. It can also affect an ordinary A/B test when assignment is not persistent across devices, sessions, or releases. The boundary is useful: a short-lived immediate response is not carryover merely because it is delayed; carryover requires prior exposure to alter a later condition’s observed outcome.
Why carryover threatens A/B-test mechanics
A conventional A/B test relies on stable assignment: a user is consistently control or treatment for the relevant outcome window. Stable assignment prevents the person from comparing versions, learning from one and acting under the other, or receiving conflicting incentives. Random session-level assignment can violate this for a product flow that users revisit. The measured difference may then be diluted toward zero or become hard to interpret.
In a crossover design, participants receive A then B or B then A. Randomizing order balances some period effects, but it does not remove persistent learning, habit, inventory, or biological effects. A washout period can help only when there is credible reason to believe its duration clears the prior effect. It cannot undo durable education, a redeemed offer, or a permanently changed account state.
Observed later outcome: Y = current-condition effect + prior-condition carryover + time effects + noise. If prior exposure differs systematically by current condition, the current-condition estimate is biased.
Do not assume a washout works because time passed. Define the mechanism and evidence for recovery. A user who learned a workflow or accepted a promotion may not return to an untreated state.
Practical scenario: support-routing switchback
A support organization wants to compare two routing policies. It considers switching the whole queue between policies every day. The outcome is time to resolution and repeat contact rate. However, unresolved tickets opened on a treatment day continue into the following control day; agents also learn the new routing rules. The next day’s outcomes therefore contain both current policy and previous-policy effects.
A safer option is to randomize new tickets persistently by ticket or customer, subject to agent-capacity constraints, and measure full resolution windows. If a switchback is unavoidable because a shared queue must use one policy, the team should use longer blocks, preplanned exclusions around transitions, inspect backlog and staffing, and interpret results as a system-level policy comparison with possible carryover. It should not claim that each daily contrast is independent.
| Potential carryover | Practical safeguard |
|---|---|
| Users learn a new workflow | Use persistent user or account assignment. |
| Discount changes later purchase behavior | Measure a full outcome window; avoid crossover for the same customer. |
| Queue backlog crosses time blocks | Use longer blocks, model transition periods, or randomize individual new work. |
| Shared inventory is depleted | Randomize at market level or measure equilibrium effects explicitly. |
Design decisions for carryover risk
Ask before launch whether a unit can encounter multiple conditions and whether the first can change later behavior, state, capacity, or availability. If yes, use the most persistent feasible randomization unit and define the primary outcome window from first exposure. Link identities across sessions and devices where consent and product architecture allow; otherwise quantify the expected cross-exposure risk rather than ignoring it.
Use crossover only when effects are plausibly short-lived, sequence can be randomized, periods can be separated from transitions, and the analysis accounts for order and time. It is often appropriate for low-stakes, rapidly reversible interface preferences; it is poor for subscriptions, pricing, learning, credit decisions, promotions, and outcomes with long latency. A parallel-group A/B test may need more sample but supplies a cleaner causal contrast.
At readout, inspect first-exposure versus repeat-exposure outcomes, transitions, order effects, and time trends according to the prespecified plan. A treatment that appears weak only after users previously saw control can signal carryover, not an absence of value. Treat an unexpected pattern as evidence for a redesigned test, not as permission to select the most favorable slice.
Limitations and common failures
- Session randomization for a repeat journey: returning users see both versions and bring learned behavior across them.
- Too-short washout: the elapsed time has no demonstrated relationship to decay of the relevant effect.
- Ignoring transition inventory or backlog: a shared system carries work between time blocks.
- Unlogged prior exposures: analysis cannot diagnose who saw which experience first.
- Confusing seasonality with carryover: both can produce time patterns; include time and sequence diagnostics.
- Claiming independent observations: repeated outcomes from the same unit require appropriate analysis.
Some carryover is the product effect teams want to create: learning, loyalty, and habit may be durable benefits. The design challenge is not to erase it from reality, but to estimate it without mixing conditions. A long-run holdout or a persistent cohort can be more informative than rapid rotation when durability matters.
Document carryover assumptions in the analysis plan. State the expected direction and duration, how prior exposure will be identified, which transition observations are excluded if any, and what diagnostic pattern would challenge the design. A simple exposure-history table—first assignment, subsequent assignment, time since prior exposure, and outcome maturity—often reveals a problem that an aggregate dashboard conceals. If reassignment is already widespread and cannot be reconstructed, report the comparison as contaminated rather than assigning false precision to the estimate.
Carryover can affect operational costs as well as user behavior. A treated cohort may create follow-up work, inventory demand, or support cases that persist into periods labeled control. Include queues, capacity, and deferred outcomes in guardrails. When decisions change a shared system, the relevant causal question may be the sustained policy effect at equilibrium, which favors longer persistent cohorts or cluster-level experiments over rapid switching.
Frequently asked questions
Is carryover always bad?
No. Durable learning or retention can be valuable. It is problematic when it prevents a design from isolating the condition currently being measured.
How long should a washout period be?
There is no universal duration. Base it on the causal mechanism, prior evidence, and outcome latency; if durable effects are plausible, avoid relying on washout.
Can persistent assignment eliminate all carryover?
It prevents a unit’s own cross-condition exposure, but shared systems, social interactions, and inventory can still create spillovers between arms.
What is the difference between carryover and novelty?
Novelty is a time-varying response to a new experience. Carryover is an earlier condition influencing a later measurement under another condition. They can occur together.
Summary
Carryover effects arise when earlier experimental exposure affects later outcomes, threatening comparisons that rotate conditions. Use stable assignment for repeat journeys, choose crossover only for reversible effects, justify washout periods, and log exposure sequence. When persistence is itself important, measure it with a design that preserves a clear counterfactual.
Sources
- NIST/SEMATECH e-Handbook: crossover designs.
- Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments.
- AB Labz: How long should an A/B test run?.