Factorial Design

What is a factorial design?

A factorial design is an experiment in which multiple controllable changes are varied together. A factor is a component being studied, such as a checkout headline, a shipping-message style, or an onboarding reminder. Each factor has levels, often a control and a treatment. In a two-by-two design, factor A has two levels and factor B has two levels, producing four combinations: A control/B control, A treatment/B control, A control/B treatment, and A treatment/B treatment.

The design answers more than “which page is better?” It estimates the average effect of A across B’s levels, the average effect of B across A’s levels, and the interaction: whether A’s effect depends on B. A factorial experiment is therefore useful when a product decision involves components that can coexist. It is not merely placing several unrelated A/B tests on the same traffic. All factors, combinations, and analysis rules must be designed together.

For a 2 × 2 experiment, random assignment to all four cells maintains a concurrent control for each contrast. When executed correctly, it can be more efficient than running separate tests sequentially because each participant contributes information about both factors. But efficiency is not automatic: the cell count, sample size, treatment compatibility, and risk of interactions must fit the decision.

Factorial structure and estimands

Suppose factor A is a simplified pricing page and factor B is an annual-billing prompt. With outcome Y, the four cell means are Y₀₀, Y₁₀, Y₀₁, and Y₁₁, where 1 indicates the new level. The average main effect of A in equal allocation is [(Y₁₀ + Y₁₁) − (Y₀₀ + Y₀₁)] / 2. The main effect of B is calculated analogously.

The A-by-B interaction is (Y₁₁ − Y₀₁) − (Y₁₀ − Y₀₀). It asks whether the A effect changes when B changes. A near-zero interaction means additive effects on that scale, not that the factors are unimportant. A large interaction can be productive—two features reinforce one another—or harmful—one message distracts from another.

State the estimand before launch. A main effect averages over the other factor’s assigned levels. That is appropriate if the intended future environment uses that mix or if the interaction is negligible for the decision. If the product will only ever ship A together with B, the relevant estimand is often the combined cell versus the current experience, not a marginal main effect.

When factorial experiments are valid

Use a factorial design when factors can be assigned independently, all combinations are feasible and safe, and the team has a credible rollout decision for the possible outcomes. The same randomization unit should normally receive a stable combination for the relevant outcome window. If a customer sees A in one session and B in another because separate systems assign them independently, the observed cells may not correspond to the planned treatment.

Each eligible unit needs a known nonzero probability of every applicable combination. Exposure must be logged separately for each factor as well as for the final combination. Define shared eligibility carefully: if B only appears after an action influenced by A, the design no longer has a simple fully crossed structure. For factorial inference, analyze all assigned eligible units for the primary outcome, not only people who reached a treatment-affected surface.

Factorial experiments retain ordinary controlled-experiment requirements: reliable assignment, concurrent comparison, outcome maturity, stable definitions, and analysis at the unit of randomization. Check allocation across cells, not just an aggregate sample ratio mismatch. A balanced total can conceal an empty or misrouted combination.

Planning sample size and analysis

A 2 × 2 design splits traffic into four cells. If stakeholders need a precise cell-versus-control conclusion, every cell needs enough sample for that contrast, and the required total can be much larger than a single A/B test. Main effects can be estimated more efficiently because they pool across the other factor, but an interaction is often less precise than a main effect of the same practical size. Plan power for the smallest decision-relevant contrast, not for a convenient omnibus statement.

For a continuous metric, a regression with coded A, B, and A × B terms is common. For binary outcomes, a risk-difference model, logistic model, or another prespecified approach may be suitable; communicate effects in a decision-friendly scale. Include baseline covariates only when they are pre-treatment and fixed in the analysis plan. Adjust standard errors for clusters or repeated observations when assignment is not independent.

Multiplicity deserves a clear policy. A factorial design can produce main effects, interactions, cell contrasts, several metrics, and segments. Classify one primary decision and limited secondary decisions. If several claims are confirmatory, control the relevant familywise error rate or false-discovery policy. A p-value below 0.05 for one of many unplanned contrasts is not a clean discovery; see multiple comparisons in A/B testing.

Worked scenario: improving trial-to-paid conversion

A software company evaluates two onboarding changes: A is a progress checklist and B is a contextual upgrade prompt. New trial accounts are randomized 1:1:1:1 to the current flow, checklist only, prompt only, or both. The primary outcome is paid conversion within 30 days per randomized account. Time to first value, support contacts, and cancellation requests are guardrails. The protocol says the team will ship an independent factor only if its average effect is practically useful and the interaction does not indicate meaningful harm.

After all accounts mature, conversion is 8.0% for current, 9.1% for checklist only, 8.8% for prompt only, and 10.5% for both. The checklist’s average main effect is 1.4 percentage points; the prompt’s is 1.1 points. The interaction is 0.6 points: the combined result exceeds the sum expected from the separate effects, but its interval includes a small negative interaction. Support contacts rise for the prompt, especially in the checklist-plus-prompt cell.

The team does not announce two universal winners. It examines the predeclared interaction and guardrails, verifies that exposure and 30-day outcomes are complete, and decides to ship the checklist broadly. It runs a focused follow-up on prompt timing before pairing the prompt with the checklist. This preserves a valid conclusion about A while treating the uncertain bundle decision with appropriate caution.

Analysis and decision process

1. List factors, levels, combinations, compatibility constraints, and the product decisions each result could support.
2. Choose the assignment unit, primary outcome window, guardrails, target population, and exposure rule for every factor.
3. Define primary main effects, required interaction tests, cell comparisons, multiplicity handling, and minimum useful effects.
4. Calculate sample requirements for the decision-relevant contrasts; reserve traffic for every valid cell.
5. Run assignment and tracking QA by factor and full combination before relying on outcome data.
6. Analyze the locked population with the prespecified model, reporting cell means, marginal effects, interaction estimates, intervals, and guardrails.
7. Ship factors independently only when the estimand matches the rollout; otherwise choose a tested combination or run a confirmation experiment.

Limitations and common errors

• Too many cells: adding factors multiplies combinations and can leave no cell with adequate precision.
• Incompatible experiences: a combination may be technically possible but confusing, unsafe, or unlike any intended shipped product.
• Interaction blindness: averaging a factor across another factor can hide an important dependency.
• Post-treatment eligibility: analyzing only users who reached a surface affected by one factor changes the causal question.
• Cell shopping: declaring the best observed cell a winner without planned multiplicity control exaggerates chance variation.
• Operational overlap: independent flags or releases can corrupt the planned combinations.

Frequently asked questions

Is a factorial design the same as multivariate testing?

It is a form of multivariate experimentation with a specified factor structure. “Multivariate testing” is often used more loosely, so state the factors and analysis explicitly.

Can we run a 2 × 2 design with uneven allocation?

Yes, but plan it deliberately. Unequal allocation changes precision and may be reasonable for risk management, although it complicates simple formulas and can weaken interaction estimates.

What if the interaction is not statistically significant?

It means the data do not establish an interaction at the planned precision. It does not prove additivity; use the interval and practical interaction threshold when deciding whether independent shipment is safe.

How many factors should we include?

Usually as few as needed for a real decision. More factors are worthwhile only when combinations are feasible and traffic supports the corresponding cells.

Can factors have more than two levels?

Yes. The same logic applies, but cell counts grow quickly. A 3 × 3 design already has nine combinations.

Summary

A factorial design randomizes participants to planned combinations of factors so teams can estimate main effects and interactions. It can answer several connected product questions efficiently, but only when all cells are feasible, sample size supports the intended contrasts, and analysis respects interactions and multiplicity. The correct rollout follows the estimand: a marginal factor effect supports independent deployment only when the future environment makes that average meaningful.

Sources

• NIST/SEMATECH e-Handbook, “Factorial Designs”
• Stark, Randomized Experiments lecture notes
• U.S. Food and Drug Administration, Multiple Endpoints in Clinical Trials

Related terms

• Multivariate Testing
• Multivariate A/B Test
• Full Factorial Design
• Fractional Factorial Design
• Interaction Effect