Variation

What does “variation” mean in an A/B test?

A variation is a deliberate difference introduced into an experiment. It might be a shorter headline, a different product image, a new CTA label, a revised form, or a complete landing-page concept.

Omniconvert describes a variation as a modified version of a control element used to measure performance differences [1]. The term is broad enough to cover a page-level alternative, while “variant” often names one specific arm of an experiment.

There is no universal vocabulary across tools. Some platforms call every arm a variation; others reserve variation for a complete combination in a multivariate design. This is why an experiment brief should define the control, each factor, each version, and the complete experience shown to users.

Variation, variant, and treatment

For a simple A/B test, the distinctions rarely affect the analysis. For a multivariate test, they matter: a factor can have several variants, and the resulting variation is the combination users actually see.

A variation should answer a question

Start with the problem and hypothesis, then design the variation. Do not begin with a list of visual changes and search for a metric afterward.

Example:

The cleanest version changes the delivery-information block. If the team also changes the hero image, headline, price display, and button, it may still test a coherent redesign—but it should not pretend to have isolated the effect of shipping information.

Single-change variations

Changing one meaningful factor is useful when the team wants to understand a specific lever. The rest of the experience should remain equivalent in layout, loading, eligibility, tracking, and timing.

Nielsen Norman Group recommends keeping the proposed difference narrow when the goal is to understand a design element’s impact [2]. This is not a law against larger redesigns; it is a rule about what the result can legitimately tell you.

Bundled variations and radical redesigns

Sometimes the hypothesis is about a complete package. A new onboarding experience may need a different sequence, copy, navigation, and interaction model to work as intended. A bundled variation can be valid when the decision is “Does this whole concept work?”

When a bundle wins, document it as a bundle. Follow-up variations can isolate the parts that appear most important, or a replication test can validate the package under a new period or audience.

Variation in multivariate testing

Suppose a team tests two headlines, three images, and two CTA labels. The full factorial design contains:

Each complete combination is a variation. “Headline B” is a level of the headline factor; “Headline B + Image C + CTA A” is a complete experience.

ActiveCampaign explains that MVT tests combinations to reveal how elements interact [3]. A/B/n testing is different when the team manually selects several complete alternatives without testing every combination.

Worked example: pricing-page variations

A subscription product wants to test how pricing is presented.

The primary metric might be paid conversion or revenue per exposed visitor. Trial starts can be a driver metric. Refunds, support requests, plan downgrades, and retention are guardrails. The highest trial-start rate does not automatically represent the best variation if it brings lower-quality customers.

How to measure a variation

Example: control converts at 4.0%; variation converts at 4.4%.

Report the point estimate with an interval, sample counts, exposure definition, and data-quality checks. A variation can have the highest observed rate and still be inconclusive.

Technical QA for variations

Before users see a variation in a live test, verify:

• the correct audience can receive it;
• the variation renders on the target browsers, devices, and viewport sizes;
• links, forms, payment, and error recovery work;
• page performance is not materially worse;
• accessibility and keyboard flows remain usable;
• events include the experiment and variation identifiers;
• exposure is logged when the user actually sees the variation;
• the user does not switch between versions unexpectedly.

A variation that loads slowly or fails on mobile is not testing only the design idea. It is testing the design plus the implementation defect.

Name and archive variations clearly

Store the screenshot, copy, configuration, code version, audience, exposure rule, metrics, dates, and result. If the variation becomes the new control, preserve its original experiment name and identifier.

Reading the result

A neutral variation is not necessarily a wasted test. It may reject a weak assumption, reveal that the issue is elsewhere, or suggest the change was too small to detect.

Common mistakes with variations

1. “Make it pop” as a hypothesis: the change has no defined mechanism.
2. Multiple unrelated changes: the result cannot be attributed to a useful cause.
3. Variant and variation used inconsistently: analysts do not know which unit is being compared.
4. Too many combinations: each version receives too little data.
5. Different technical behavior: a slow or broken version wins or loses for the wrong reason.
6. Changing the variation mid-test: two different experiences share one label.
7. Declaring a winner from ranking: the leader may be noise.
8. Ignoring downstream value: a micro-conversion rises while retention falls.

Variation checklist

• Variation is based on a clear observation and hypothesis.
• Intended difference from control is written in plain language.
• Single factor, coherent bundle, or factorial combination is identified.
• Primary and guardrail metrics are selected before launch.
• Traffic, sample size, MDE, duration, and comparison plan are defined.
• Variation has a descriptive name and version.
• Rendering, performance, accessibility, and functionality are QA’d.
• Assignment, exposure, and outcome events are linked consistently.
• SRM, identity, and cross-device behavior are checked.
• Result and screenshots are archived for future experiments.

FAQ

Is a variation the same as a variant?

Often, yes. In simple A/B testing they are commonly interchangeable. In multivariate testing, “variant” may describe a level of one factor, while “variation” describes the complete combination shown to a user.

How many variations should an A/B test include?

As many as the question and traffic support. Every additional version divides traffic and increases analysis complexity. Test a small number of meaningful alternatives rather than many cosmetic copies.

Can a variation contain more than one change?

Yes, when the hypothesis concerns a coherent package or redesign. The result then estimates the package, not the isolated effect of each component.

What is the difference between A/B/n and multivariate testing?

A/B/n tests several selected complete alternatives. Multivariate testing systematically combines levels of multiple factors to estimate main and interaction effects.

What should happen to a winning variation?

Validate the result, inspect guardrails, roll it out safely, preserve the experiment record, and use it as the new baseline only after implementation is stable.

Sources

1. Omniconvert: Variation in A/B Testing
2. Nielsen Norman Group: Multivariate Testing
3. ActiveCampaign: Multivariate Testing
4. Awa Digital: A/B Testing Best Practices
5. First Principles: A/B Experimentation
6. Unbounce: A/B Testing Guide
7. Statsig: MVT vs A/B Testing

Related terms

• A/B Testing
• A/B Test
• A/B/n Testing
• A/A Testing
• Split Testing