Experiment Design

What is experiment design?

Experiment design is the set of choices that make a test answer a particular question credibly. It specifies what changes, who is eligible, which entity receives assignment, what the control is, how allocation works, which outcomes matter, how long the test runs, and how a result will change a decision. It is not simply the visual design of a variant or a statistical calculation performed after launch.

In digital experimentation, the common design is a parallel randomized A/B test. Eligible units are allocated concurrently to a current experience or a treatment, allowing their outcomes to be compared in the same operating conditions. A/B/n designs add several variants; factorial designs test combinations of changes; cluster and geo experiments assign groups rather than people; sequential designs permit planned interim decisions. The appropriate design follows the causal question and constraints, not a preference for a fashionable method.

A sound design makes the eventual result interpretable. Without a defined population and denominator, a conversion lift is ambiguous. Without stable assignment, returning users can receive mixed treatment. Without a primary metric and stopping rule, the readout can become a search for any favorable chart. Design is therefore an act of decision governance as well as statistics.

Core elements of an A/B-test design

First formulate the decision and hypothesis. A useful hypothesis names a population, intervention, mechanism, primary outcome, and expectation: for eligible returning shoppers, a saved-address selector will reduce checkout friction and increase completed orders per visitor. The decision may be “roll out if the improvement exceeds the practical threshold without material guardrail harm,” not merely “see whether it is significant.”

Then define population and assignment. Eligibility must be based on information available before treatment; excluding people after a treatment-affected step can bias the comparison. Choose a unit that follows the experience—often a user or account—and assign it persistently. If customers share inventory, settings, or a social network, individual assignment may cause interference. A larger cluster, holdout, or different design may be required.

Measurement includes a primary metric, diagnostics, and guardrails. The primary metric decides the main claim. Diagnostics explain where behavior changed. Guardrails constrain a decision by measuring cost, reliability, quality, fraud, complaints, or downstream value. Define event logic, denominator, attribution window, exclusions, missing-data handling, and data-maturity time before launch. Log assignment and exposure separately so delivery can be audited.

Finally, plan information and inference. Determine the minimum effect worth acting on, baseline rate or variance, allocation, desired power, and false-positive policy. These determine the needed independent sample and approximate runtime. Choose fixed-horizon analysis or a valid sequential design. The plan should also state treatment of multiple variants, metrics, segments, outliers, and deviations.

Scenario: changing trial pricing presentation

A SaaS company wants to show annual-plan savings more prominently on its trial upgrade page. It worries that a higher annual-plan share could improve cash flow while lowering overall paid conversion if the presentation feels coercive. The team designs a two-arm test for eligible trial owners, assigned persistently by workspace. Control keeps the current monthly-first layout; treatment shows annual savings next to the annual option while keeping monthly pricing equally accessible.

The primary metric is paid conversion per eligible workspace within 21 days. Annual-plan share and revenue per eligible workspace are diagnostic measures. Cancellation requests, support contacts about pricing, payment failures, and 60-day retention are guardrails or longer-term follow-up measures. The plan selects a minimum lift that would justify design and billing work, calculates the needed workspace count, and requires whole-week coverage plus the conversion window before final analysis.

If treatment raises annual-plan share but not paid conversion, the design has still produced useful evidence: the presentation changes mix, not necessarily total demand. If a primary-metric improvement is paired with more cancellations, the team needs an explicit trade-off decision rather than a simplistic winner label.

Experiment design workflow

1. Write the causal question: define the change, reference experience, target population, outcome, and action that follows each plausible result.
2. Assess feasibility and risk: confirm that randomized variation is safe, legal, technically possible, and unlikely to create unacceptable spillovers.
3. Select the design type: choose A/B, A/B/n, factorial, cluster, geo, switchback, or holdout based on the intervention and unit boundaries.
4. Specify treatment delivery: describe every material difference, fallback behavior, assignment persistence, and exposure event.
5. Define analysis up front: set primary and guardrail metrics, denominators, windows, estimator, segments, error policy, and stopping rules.
6. Size and schedule the test: use baseline behavior and the minimum worthwhile effect to calculate information needs, then cover relevant business cycles.
7. Run a preflight review: test event definitions, targeting, bucketing, rendering, allocation, latency, and rollback before exposing meaningful traffic.

Pre-registration need not be bureaucratic. A concise design record makes disagreements visible before data can influence them. It also distinguishes confirmatory analysis, which follows the plan, from exploratory analysis, which can generate valuable next hypotheses but should not carry the same strength of claim.

Limitations and common design errors

Good design does not eliminate uncertainty. An experiment estimates an effect for the tested implementation, population, and time window. It may not generalize to a different country, device, season, or future product state. Long-term consequences may be unavailable at the initial decision point. Tests can also be constrained by low traffic, delayed outcomes, or systems that make clean randomization impossible.

• Testing an undefined bundle: changing copy, price, layout, and eligibility at once estimates the package, not each component’s independent effect.
• Choosing metrics after launch: it invites selection of a favorable outcome rather than an honest decision criterion.
• Underpowering on convenience: a short test with a wide interval cannot establish that an important effect is absent.
• Ignoring multiple comparisons: adding variants and segments without a plan creates accidental winners.
• Using post-treatment eligibility: filtering on users who clicked, activated, or reached a later page can break randomization.
• Missing operational guardrails: a treatment can improve conversion while causing outages, fraud, cost overruns, or customer harm.

Design should be revised when implementation facts change, but amendments must be documented before inspecting the relevant outcome where possible. Changing allocation, audience, metric definitions, or stopping rules midstream can be defensible for safety or operational reasons; it also changes the inferential meaning of the test and must not be hidden.

Frequently asked questions

What is the most important part of experiment design?

A clear causal question and decision rule. They determine the appropriate population, unit, metrics, sample, and interpretation. Statistical machinery cannot rescue a test that lacks a meaningful contrast.

Should every test use a 50/50 split?

No. Equal allocation is often efficient for two similar arms, but uneven allocation may reduce risk, preserve a holdout, support several variants, or accommodate operational constraints. Account for it in sample planning.

When should I use A/B/n instead of A/B?

Use A/B/n when selecting among several coherent alternatives is worth the additional traffic and analysis complexity. Each added arm receives fewer observations and increases the need for a multiple-comparison plan.

Can I change a test after it starts?

Only with care. Safety fixes and documented implementation repairs may be necessary, but material changes can create a new treatment or population. Record the change, assess data before and after it separately, and consider a restart.

How do guardrails fit into a decision rule?

Define unacceptable harm in advance. A treatment can be considered for rollout only when the primary effect is worthwhile and guardrail estimates remain within the agreed limits, subject to uncertainty and operational judgment.

Summary

Experiment design turns a product idea into an interpretable causal comparison. It aligns the decision, unit, intervention, measurement, sample, monitoring, and action rule before outcomes arrive. The right design is the one that makes the needed decision credible while respecting real-world risks and constraints.

Sources

1. Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments
2. NIST/SEMATECH e-Handbook: hypothesis testing
3. AB-Labz: How to write an A/B-test hypothesis
4. AB-Labz: How to calculate sample size

Related terms

• A/B Testing
• A/B Test
• A/B/n Testing
• A/A Testing
• Split Testing