Exposure

What is exposure in experimentation?

Exposure describes when an experiment’s intervention became available to the unit whose outcome will be measured. For a button-color test, it may occur when the button is rendered in the viewport. For a new ranking model, it may occur when results generated by that model are served. For an email experiment, exposure may occur when the message is sent, not when it is opened. The appropriate definition follows the causal question and the product’s delivery behavior.

Exposure is often confused with assignment. Assignment is a randomization decision: a user or account is placed in a treatment or control arm. Exposure is evidence that the assigned experience was actually delivered or available. A person may be assigned to a new checkout but never visit checkout; an account may be assigned to an email policy but be ineligible for a send; a feature flag may assign treatment but fall back because of an error. These differences are useful diagnostic information.

It is also not the same as engagement. A user can be exposed to a recommendation without clicking it. Requiring a click, purchase, activation, or another post-treatment action before including someone in an analysis can bias the comparison because treatment may change who reaches that action. The central question is not “who engaged?” but “what population and treatment opportunity does the estimate represent?”

Exposure mechanics in an A/B test

A reliable experiment records several related states. First, an eligibility event shows that the unit met pre-treatment criteria. Second, an assignment event identifies the intended arm and randomization key. Third, an exposure event records the version, timestamp, experiment identifier, and context when the version was available. Outcome events then use compatible identifiers and windows. Each state should be attributable to the same unit definition, especially when anonymous visitors later sign in or a person uses multiple devices.

The primary analysis population should be prespecified. Many product experiments use assignment-based or intention-to-treat analysis: include all eligible units assigned to each arm, whether or not they later use the surface. This estimates the effect of offering the change and preserves randomization. An exposure-based analysis can be valid for a narrowly defined question when exposure is mechanically determined by eligibility and assignment before behavior diverges, but it requires careful review. If treatment affects the chance of exposure, conditioning on exposure can create selection bias.

Consider a treatment that loads a new payment widget only after a user clicks “continue.” An analysis restricted to exposed users compares people who clicked continue in treatment with people who clicked continue in control. If the widget changes that click, the groups are no longer comparable. The treatment could appear beneficial or harmful for reasons created by the filtering rule. Assignment-based analysis answers the cleaner question: what happened when eligible users were offered the new flow?

Exposure logging also supports quality assurance. Compare planned assignment counts with exposure counts by arm, browser, app version, country, and time. A sudden treatment-only decline may reveal rendering failures, latency, targeting mistakes, cached code, or a tracking regression. It should trigger investigation before outcome differences are interpreted as product effects.

Scenario: testing a search ranking model

An ecommerce company wants to replace its search ranking model for queries with at least twenty eligible products. It assigns visitors 50/50 by persistent user ID. Control receives the existing model; treatment receives the new model. The team defines exposure as the first successful search-response impression in which the assigned model returned results. It logs the model version, query category, result count, response status, and assignment arm.

The primary metric is seven-day orders per assigned eligible searcher. Search result click-through rate, add-to-cart rate, zero-result rate, latency, complaint rate, and product-category diversity are diagnostics or guardrails. A visitor assigned to treatment who leaves before searching remains in the primary assigned population; excluding them would answer a different question and could distort the comparison if the experiment affects the likelihood of searching.

At readout, researchers can use exposure data to explain delivery and heterogeneity, but they should clearly label whether figures are based on assignment, eligible units, or confirmed exposures. A conversion rate without its denominator is not interpretable. “Orders per exposed searcher” and “orders per assigned eligible searcher” answer different questions even when their numerical values look similar.

How to define and use exposure

1. Start with the intervention: identify the moment at which the treatment can plausibly affect behavior. Avoid using a generic page view if the changed component is not available then.
2. Separate pre-treatment states: document eligibility, assignment, delivery, exposure, and outcome as different events where the system can distinguish them.
3. Choose the estimand first: decide whether the decision needs the effect of assignment, an offered experience, a successfully delivered experience, or another precisely stated contrast.
4. Log sufficient context: include experiment and variant IDs, unit IDs, timestamps, platform or app version, targeting inputs, response status, and relevant fallback conditions.
5. Validate before outcomes mature: inspect allocation, duplicate events, missing values, treatment rendering, and exposure rates across important technical slices.
6. Report denominators: state exactly whether rates use assigned, eligible, exposed, or another population, and do not switch the denominator after seeing results.
7. Respond to delivery failures: distinguish a treatment’s intended effect from an implementation incident. A broken delivery path is a safety and engineering issue first.

A clear exposure definition can improve sensitivity by excluding exposure-impossible traffic only when the exclusion is truly pre-treatment and symmetric. For example, a test that only runs on a supported app version can define eligibility using that version before assignment. But moving the same condition later in the funnel, after behavior has been affected, changes the population and may invalidate the causal comparison.

Limitations and common errors

Exposure data can be imperfect. Client-side tracking can be blocked, delayed, duplicated, or lost when the page closes. Server logs can record a response even when the client never renders it. Identity can change between anonymous browsing and authentication. These are reasons to design redundant, privacy-conscious instrumentation and to quantify missingness, not reasons to assume every exposure record is literal ground truth.

• Calling assignment exposure: this hides the difference between allocation and delivery, making troubleshooting much harder.
• Logging after interaction: if the exposure event fires only after a click, it cannot establish that the treatment was available to non-clickers.
• Filtering on a treatment-affected step: selecting only exposed, activated, or purchasing users can introduce post-treatment bias.
• Asymmetric instrumentation: a treatment-specific component may log more events simply because its code path differs from control.
• Unstable identity: assignment at device level and outcomes at account level can create duplicate, missing, or cross-arm records.
• Ignoring latency and fallbacks: a valid variant may be assigned but never meaningfully experienced when it times out or reverts.

Exposure is not a license to erase noncompliance. If a policy is assigned but only partly delivered, the assignment effect may be exactly the decision-relevant quantity: what will happen if the organization deploys that policy with its actual delivery reliability? A separate engineering analysis can estimate and improve delivery. The correct choice depends on the business decision, and it should be made before the result is known.

Frequently asked questions

Is exposure the same as a page view?

Only when the tested experience is genuinely available on every such page view and the page-view event is recorded consistently. A component that loads later, is below the fold, or can fall back needs a more specific definition.

Should I analyze only users who were exposed?

Usually not by default. Assignment-based analysis preserves randomization and estimates the effect of offering treatment. Exposure-only analysis needs a justification showing that the filter is pre-treatment and does not differ because of the variant.

Why log both assignment and exposure?

Assignment validates randomization and defines the intended arms. Exposure validates delivery and helps diagnose technical failures. Together they reveal whether a measured result is likely product behavior, implementation behavior, or a mixture.

What if exposure logging is missing for one variant?

Do not use the logged exposure population as though it were comparable. Assess the scope of missingness, repair the event, and determine whether the primary assignment-based analysis remains valid or a rerun is required.

Can exposure happen more than once?

Yes. A user may see an experience repeatedly. Define whether analysis uses first exposure, any exposure in a window, or repeated impressions, and align that choice with the mechanism and attribution window.

Summary

Exposure records a real opportunity for an experimental unit to receive a treatment. Separating it from eligibility, assignment, and engagement makes delivery auditable and analysis more honest. The key is to select the analysis population before launch and avoid filtering on actions that treatment can change.

Sources

1. Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments
2. NIST/SEMATECH e-Handbook: hypothesis testing
3. AB-Labz: Sample ratio mismatch in A/B testing
4. AB-Labz: Primary vs. guardrail metrics

Related terms

• A/B Testing
• A/B Test
• A/B/n Testing
• A/A Testing
• Split Testing