Quick definition: Concurrent experiments are tests that run over the same calendar period and may share users, surfaces, metrics, or operational systems. They require explicit coordination so one experiment does not distort another’s assignment, exposure, or interpretation.
What are concurrent experiments?
Experimentation programs rarely run one test at a time. A visitor might simultaneously see a new homepage message, pricing experiment, recommendation model, and checkout change. Concurrent experiments are those active at overlapping times; they can be independent, mutually exclusive, or intentionally factorial. Their existence is not a data-quality failure. Unmanaged overlap is the risk.
Two experiments are safely independent only when their assignment, delivery, outcomes, and decision rules do not interact in a meaningful way. Shared users are not automatically a problem if assignments are independent and effects are additive enough for the stated estimand. But a homepage change that alters traffic to a checkout test, or a price experiment that changes conversion, can alter interpretation even when each allocation system is working correctly.
Mechanics and required controls
Maintain an experiment registry with owner, hypothesis, audience, randomization unit, allocation, start and end times, surfaces, primary and guardrail metrics, and known dependencies. The assignment service should log every active experiment, variant, eligibility decision, exposure timestamp, and version for each analysis unit. Without this record, an analyst cannot reliably reconstruct which conditions a user experienced.
Choose an overlap policy before launch. A mutual-exclusion layer prevents users from entering incompatible tests. Independent namespaces permit orthogonal experiments. A factorial design deliberately assigns combinations and estimates main effects and interactions. The right policy depends on the product path, intervention mechanism, and decision question—not simply on how many experiments are already running.
Monitor assignment and exposure by experiment and by important overlap cells. Overall 50/50 traffic can hide a failure where one treatment disproportionately receives another test’s treatment. Investigate allocation anomalies with sample ratio mismatch checks, but also inspect eligibility changes, holdouts, and treatment delivery.
Assumptions and interpretation
A simple A/B estimate commonly targets the average effect of one treatment in the environment that actually occurred, including the program’s normal background of other experiences. That can be useful for rollout. It is not necessarily the isolated effect under a hypothetical world with no other changes. State which interpretation matters.
If effects interact, the treatment effect differs by the other experiment’s variant. Ignoring a material interaction can make an average misleading, especially if the concurrent allocation changes during the test. Prespecify high-risk interactions, preserve concurrent allocation, and use factorial or regression analyses with interaction terms when sample size supports them. Do not mine every overlap cell after results arrive; that creates severe multiple-comparison risk.
Concurrent experimentation in A/B programs
Before launch, ask whether tests touch the same user journey, eligibility predicate, price, ranking system, page performance budget, or metric denominator. A checkout test and a footer-color test may coexist safely; a checkout test and payment-method experiment probably need coordinated design. Shared guardrails such as latency, error rate, refunds, and support volume also require ownership rules so neither team declares success while harming the combined experience.
Use pre-treatment assignment and stable identifiers. Do not exclude users from one analysis because they converted in another concurrent test unless that exclusion is part of the planned estimand. Outcome maturity should be aligned: a seven-day conversion test can be contaminated by a concurrent change launched halfway through the window if exposure history is not retained.
Worked scenario: landing page and checkout
A retailer runs a landing-page messaging test and a checkout simplification test concurrently. Both assign eligible users independently at 50/50. The landing test changes the number of visitors reaching checkout, while the checkout test changes completion among eligible visitors. For checkout’s primary intention-to-treat metric, the team includes all users assigned to checkout before any downstream behavior, logs landing assignment, and estimates a prespecified landing-by-checkout interaction.
If the interaction is negligible, the checkout effect can be reported as an average across the normal landing mix. If the new landing page makes checkout simplification much more effective, the team should report that dependence and plan rollout as a combined experience rather than claiming one universal checkout lift.
Interpretation and reporting
Report active concurrent tests, overlap policy, realized overlap rates, known dependencies, interaction analyses, and any exclusions. Explain whether the effect is conditional on the production experiment environment or estimated as a factorial main effect. Include effect sizes and intervals, not only a winner label; confidence intervals make interaction uncertainty visible.
Limitations and common mistakes
- Assuming independent bucketing guarantees independent outcomes.
- Allowing incompatible tests to share a surface without exclusion or factorial design.
- Changing eligibility or allocation mid-test without preserving history.
- Analyzing post-treatment overlap groups as if they were baseline segments.
- Ignoring shared guardrails and operational capacity.
- Testing every possible interaction without power or multiplicity control.
Frequently asked questions about concurrent experiments
Can users be in more than one experiment?
Yes, when the overlap policy and logging support it and the interventions are compatible.
Should every concurrent test be mutually exclusive?
No. That wastes traffic when tests are safely orthogonal. Use exclusion for conflicting or high-risk interventions.
What is an interaction effect?
It occurs when the effect of one variant depends on the other experiment’s variant.
Do concurrent tests invalidate A/B results?
Not automatically. They invalidate claims when overlap, interference, or assignment is unmanaged or misinterpreted.
Summary
Concurrent experiments are normal in mature testing programs. Reliable results require an explicit overlap policy, complete assignment and exposure logs, coordinated metric ownership, concurrent timing, and planned treatment of important interactions.
Sources
- Microsoft experimentation platform research
- NIST/SEMATECH Statistical Handbook
- Kohavi et al., Trustworthy Online Controlled Experiments