Quick definition: An interaction effect occurs when the effect of one treatment factor depends on the level of another factor, a baseline characteristic, or a context. In a factorial A/B test, it measures whether two changes combine additively or modify each other’s impact.
What is an interaction effect?
An interaction effect answers a conditional question: does change A work the same way when change B is present? If a clearer pricing page improves conversion by one point without a discount and by four points with a discount, the effect of the page depends on the discount. The difference between those two treatment effects is the interaction. It can be positive, meaning the combination is more effective than expected from separate effects, or negative, meaning one change weakens the other.
Interactions occur between assigned factors in a factorial experiment and between a treatment and a pre-treatment characteristic. The first is usually a design question about combinations; the second is a subgroup or heterogeneous-treatment-effect question. Both need a clearly stated effect scale. A model can show no interaction on an additive probability scale yet show one on a relative-risk or odds-ratio scale.
An interaction is not simply the best observed cell. Four variants will naturally have different sample means. The causal interaction compares differences: it asks whether the A-versus-control contrast changes across B’s assigned levels. It requires randomization to the relevant combinations and an analysis planned to estimate that contrast.
Methodology and formula
In a two-by-two factorial experiment, let Y00 be the mean outcome with neither new factor, Y10 with A only, Y01 with B only, and Y11 with both. The additive interaction is (Y11 − Y01) − (Y10 − Y00). It is a difference in differences. Zero means A has the same absolute effect whether B is off or on.
A regression provides the same estimate. With A and B coded 0/1, Y = β0 + βAA + βBB + βABAB + error. βAB is the interaction on the model’s scale. For a binary outcome, linear probability models make absolute differences directly readable; logistic models estimate an interaction in log-odds, which should be translated into predicted probabilities for product decisions. Include covariates only when they were available before assignment and specified in the analysis plan.
| Observed pattern | Interaction | Possible decision |
|---|---|---|
| A helps equally with B off and on. | Near zero on the chosen scale. | Consider independent rollout. |
| The bundle exceeds additive expectation. | Positive interaction. | Test or ship the bundle if guardrails support it. |
| A helps alone but hurts with B. | Negative interaction. | Avoid simultaneous deployment. |
| Effects differ only in noisy small cells. | Imprecise interaction. | Do not infer compatibility from point estimates. |
Design, assumptions, and sample size
To identify an interaction between factors, randomly assign every feasible combination and maintain stable assignment through the outcome window. Factors must be independently deliverable, compatible, and observable. If B only appears after a click caused by A, then users are not randomly assigned to a fully crossed A-by-B design; analyzing the subset who reached B can introduce post-treatment selection bias.
Interaction estimates are generally less precise than main effects. A two-by-two design divides equal traffic into four cells, and an interaction is a contrast of four cell means. Plan the sample for the smallest decision-relevant interaction, not merely for the main effects. If the intended rollout is only the combined experience, power the direct bundle-versus-current comparison as well.
Keep the primary metric, outcome window, and multiplicity policy explicit. Main effects, interactions, four cell comparisons, guardrails, and segments create a large claim family. An unplanned interaction found after dozens of charts is exploratory. For a practical overview of the error problem, see multiple comparisons in A/B testing.
Interaction effects in A/B testing
Interaction analysis is essential when teams plan overlapping releases. Two independently positive A/B tests do not prove the two features are jointly positive. A recommendation widget may increase engagement alone, while a notification redesign changes which users reach it. A price message can improve purchase intent but work differently when a financing offer is present. When a combination is likely to ship, test the combination or explicitly assess the interaction.
Not every overlap needs a factorial test. If factors target disjoint users or cannot coexist, their interaction is not an operational question. Conversely, if traffic cannot support every combination, serial tests with a stable baseline may be safer than an underpowered factorial design. The right choice follows the product decision, not a preference for complex analysis.
Worked scenario: checkout trust and delivery messaging
An online retailer tests A, a security reassurance module, and B, an estimated-delivery message. Visitors are randomized equally to current checkout, A only, B only, or both. The primary metric is paid checkout completion; refund requests and page latency are guardrails. The protocol says the team will only ship factors independently if a practically important negative interaction can be excluded.
Completion is 61.0% in control, 62.4% with A, 62.0% with B, and 62.7% with both. The A effect is 1.4 points when B is absent but 0.7 points when B is present. The interaction is −0.7 points, with an interval from −1.5 to 0.1. Although all treatment cells are above control, the data cannot exclude a negative interaction large enough to matter. The team ships the delivery message, which has low latency cost, and runs a follow-up on the trust module’s placement instead of announcing a winning bundle.
This is a better interpretation than declaring “both features win.” The decision reflects the estimated conditional effects, uncertainty, compatibility threshold, and guardrails—not just the largest cell mean.
Practical workflow
- List factors that may coexist and write the combined-product decisions the test must support.
- Define valid cells, assignment unit, exposure rules, primary contrast, interaction scale, and guardrails.
- Power main effects, the interaction, and any necessary cell-versus-control comparison.
- Validate allocation and exposure in every cell; an aggregate allocation check is insufficient.
- Analyze locked mature outcomes with the prespecified factorial model and show all cell means.
- Interpret intervals against a practical compatibility threshold, not a p-value alone.
- Roll out independent factors only when the estimand matches the intended environment; otherwise ship a tested combination or confirm.
How to interpret an interaction
A statistically significant interaction supports a difference in effects on the stated scale. It does not automatically make the combination desirable: the combined effect can be positive but economically small, or a guardrail can fail. A non-significant interaction means the experiment did not establish a difference at its precision; it does not prove additivity. Report the interaction estimate and interval, the four cell outcomes, and the scale used.
Main effects average over the other factor’s assigned levels. That average can be useful when the future product will preserve the same mix and the interaction is negligible. If the business will always deploy A with B, the bundle effect—not an averaged A main effect—is normally the decision-relevant quantity.
Limitations and common mistakes
- Calling overlap factorial: independently launched experiments do not create randomized combinations automatically.
- Underpowered cells: a study built for main effects may not answer the interaction question.
- Wrong scale: odds-ratio interactions are often misread as absolute conversion interactions.
- Post-treatment eligibility: conditioning on a treatment-affected screen or action breaks the clean factorial comparison.
- Ignoring multiplicity: scanning cells and segments after launch inflates false discoveries.
- Shipping incompatible changes: separate wins do not guarantee a winning bundle.
Frequently asked questions
Is an interaction always bad?
No. It can reveal useful complementarity. The question is whether the conditional effect is reliable and worth the added complexity.
Can we test an interaction with two separate A/B tests?
Only if their assignment systems intentionally form and log all joint cells with known probabilities. Otherwise overlap is observational.
What does no significant interaction mean?
It means the data do not establish a difference of effects under the planned analysis. Use the confidence interval to assess whether important differences remain plausible.
How is it related to heterogeneous treatment effects?
A treatment-by-baseline-characteristic interaction is a common way to estimate an HTE. Factor-by-factor interactions concern combinations of assigned treatments.
Summary
An interaction effect measures whether one change’s causal effect depends on another factor or context. Factorial A/B tests estimate it through a difference in differences across randomized cells. Sound use requires compatible combinations, sufficient cell-level sample size, a declared scale and multiplicity policy, and decisions based on conditional effects rather than the largest observed variant.
Sources
- NIST/SEMATECH e-Handbook, “Factorial Designs”
- Stark, Randomized Experiments lecture notes
- Knol and VanderWeele, “Recommendations for Presenting Analyses of Effect Modification”