Quick definition: A full factorial design assigns experimental units to every possible combination of the selected factor levels, making all planned main effects and interactions estimable from the experiment.
What is a full factorial design?
A full factorial design is the complete version of a factorial experiment. If an experiment has factors with L1, L2, and L3 levels, it contains L1 × L2 × L3 treatment combinations. A two-level design with three factors has eight cells; a three-factor design with one three-level factor and two two-level factors has 12 cells. Every combination that the theory and product scope permit is intentionally represented.
Completeness is the defining feature. A full factorial design does not omit selected combinations to save traffic or infer an unobserved combination from observed ones. It can estimate each factor’s average effect, pairwise interactions, and higher-order interactions within the defined levels.
In online experimentation, it may test copy, a default setting, and support treatment together. The approach is valuable when components might reinforce or interfere and the organization needs evidence about the bundle. It is more ambitious than a simple A/B test, so the decisions must justify its traffic and operational cost.
How full factorial structure works
Consider a 2 × 2 × 2 checkout experiment. Factor A is a concise price explanation, B is a delivery-date message, and C is a trust badge. Each visitor is randomly assigned to one of eight combinations. The current experience is A0/B0/C0; the fully changed experience is A1/B1/C1. With balanced allocation, each cell receives one eighth of eligible traffic.
A main effect of A averages its contrast over every B and C combination. A two-factor A × B interaction asks whether the A effect differs by B after averaging C. An A × B × C interaction asks whether the A-by-B dependency itself changes when C changes. These effects depend on the outcome scale: additivity in conversion-rate points is not the same as additivity in log odds or revenue dollars.
| Effect | Question it answers | Typical rollout implication |
|---|---|---|
| Main effect | Does one factor help on average across the others? | Potential independent default change. |
| Two-way interaction | Do two factors work differently together? | Ship or avoid a pair as a bundle. |
| Higher-order interaction | Does a pair’s relationship depend on another factor? | Use caution; often needs strong prior justification. |
| Cell contrast | How does a complete experience compare with current? | Select a tested complete configuration. |
Randomization should be implemented as one coherent assignment to a combination, or as independent balanced assignments that are demonstrably equivalent. Persist the assignment for the observation window and log factor-level and combination-level exposure. If feature flags have constraints, eligibility filters, or conflicts, design those rules before launch; a missing combination means the design is not full factorial.
Assumptions and valid application
Full factorial experiments depend on the general assumptions of randomized controlled experiments: units must be assigned according to the documented probabilities, outcome measurement must be comparable across arms, and analysis must use an appropriate unit and mature observation window. The method adds an important practical requirement: every factor combination must be deliverable and ethically acceptable. A combination that produces conflicting messages, inaccessible navigation, or an unsafe operational process should not be randomized merely to complete a table.
All factors must be defined before assignment. A “level” cannot be a behavior that occurs after treatment, such as whether a visitor saw a message after choosing a product. Conditioning on that behavior can introduce selection bias. Similarly, treatment combinations must remain distinct in practice. If an automatic personalization system replaces a control headline for some visitors, observed differences no longer correspond cleanly to the designed factors.
Interpretation also assumes the selected levels describe a meaningful decision domain. A result for a small font change, a medium font change, and a large font change does not automatically generalize to an untested design. The experiment estimates effects for its population, time, and levels. When users interfere with one another, inventory is shared, or treatment is assigned by store, use a design and variance estimate that respect that dependence.
Practical rule: A full factorial design estimates all planned combinations; it does not make every possible interaction a credible product discovery. Preselect the interactions that could change a decision.
Planning traffic, precision, and analysis
The number of cells grows multiplicatively. Four binary factors create 16 combinations, so traffic ample for one A/B comparison may leave each cell underpowered. Before launch, calculate the required sample for the most important complete-experience contrast and for interactions that would alter shipment. Designating higher-order interactions exploratory does not make a design capable of precisely resolving them.
Balanced allocation is usually efficient when combinations have comparable risk and cost. Unequal allocation can protect users from a risky configuration, but analysis weights must match the planned assignment and target population. A pre-experiment covariate adjustment can improve precision if it uses only baseline information; see CUPED.
A regression model containing coded factor terms and the planned interactions is a clear analysis framework. For the checkout example, include A, B, C, A × B, A × C, B × C, and A × B × C when the full three-way effect is in scope. Report raw cell sizes and outcomes alongside model estimates so stakeholders can see the experience behind a coefficient. Define a multiplicity strategy for primary effects, interactions, and selected cell contrasts rather than promoting whichever result has the smallest p-value.
Worked scenario: configuring a merchant activation flow
A platform wants to improve merchant activation within 14 days. It can vary A, a setup checklist; B, an in-product sample catalog; and C, a live-chat invitation. Each has current and new levels, so it runs an eight-cell full factorial experiment. Activation is the primary metric; support cost, time to publish, and merchant opt-out rate are guardrails. The team decides in advance that it will evaluate A and B main effects, the A × B interaction, and the complete new bundle versus the current bundle. It will not make a confirmatory claim about all possible three-way patterns.
After mature outcomes are available, the checklist helps in most cells. The sample catalog helps without chat but has little effect with chat. The A × B interaction suggests that the catalog’s benefit is larger when the checklist is absent, likely because both features solve the same early uncertainty. Chat raises activation for low-usage merchants but also support cost; that segment result is exploratory because it was not powered as a primary decision.
The team ships the checklist as a default and runs a targeted follow-up to choose between catalog and chat for different merchant needs. It does not ship the all-new bundle just because it has the highest raw activation rate: its support-cost guardrail exceeds the preset limit, and the factorial results show that the bundle is not necessary for the checklist’s value.
Analysis and decision process
- Define the factors, all levels, feasible combinations, treatment conflicts, and the decision each potential effect could affect.
- Set the randomization unit, exposure event, primary estimand, outcome window, guardrails, and treatment persistence rule.
- Classify main effects, interactions, and cell contrasts as primary, secondary, or exploratory; set meaningful-effect thresholds and error control.
- Plan sample size at the cell level and test flag logic, eligibility, and exposure logging for every combination.
- During the experiment, monitor allocation and data quality by cell without selecting winners from immature results.
- At analysis, use the locked model and report cell values, planned contrasts, uncertainty, multiplicity-adjusted conclusions where applicable, and protocol deviations.
- Map results to a concrete configuration and validate that its operational guardrails remain acceptable before rollout.
When the decision deadline is fixed, a full factorial design can still be preferable to serial tests, but only if there is enough traffic for all cells. A fixed analysis horizon avoids opportunistic stopping; details are covered in how long an A/B test should run.
Limitations and common errors
- Combinatorial growth: each added factor divides traffic and increases QA burden.
- Low-value interactions: testing every high-order term can consume power and produce unstable stories.
- Missing cells: incompatible flags or implementation failures destroy the full structure.
- Overgeneralization: results apply to tested levels, not every possible setting.
- Confusing marginals and bundles: a positive average factor effect does not guarantee that every cell containing it is better.
- Ignoring guardrails: choosing a best conversion cell without operational costs or safety checks can cause a bad rollout.
Frequently asked questions
How is full factorial different from fractional factorial?
A full factorial design includes every combination, so planned effects can be separated without intentional aliasing. A fractional factorial design runs a selected subset and accepts assumptions about which effects are negligible.
Must every factor have two levels?
No. Factors may have several levels, though each additional level expands the combination count and sample requirement.
Can we drop a poorly performing cell during the test?
Only under a prespecified sequential or safety rule. Removing a cell after viewing outcomes changes the design and analysis; emergency harm mitigation is separate from a confirmatory claim.
Should we always estimate three-way interactions?
No. Include them when a credible mechanism and decision justify the sample requirement. Otherwise designate them as exploratory or simplify the design.
Does full factorial eliminate multiple-comparison risk?
No. It organizes contrasts but can create many of them. The protocol still needs a clear primary question and multiplicity policy.
Summary
A full factorial design assigns every planned combination of factor levels, enabling direct estimation of main effects, interactions, and complete-experience contrasts. It is the right choice when combinations are feasible and interaction evidence matters enough to fund every cell. Its rigor comes from predefinition, sufficient cell-level information, design-aware analysis, and rollout decisions that reflect both effectiveness and guardrails.
Sources
- NIST/SEMATECH e-Handbook, “Factorial Designs”
- Stark, Randomized Experiments lecture notes
- Penn State STAT 503, “Factorial Designs”