Quick definition: Incrementality is the extent to which an action causes outcomes that would not otherwise have happened.
What is incrementality?
Incrementality separates causal impact from correlation and credit assignment. A customer who buys after seeing an ad, receiving an email, or using a new product experience may have bought anyway. The incremental outcome is the portion created because the intervention occurred. This question matters whenever a team decides whether to invest, expand, or keep a change.
Attribution is not incrementality. Attribution models allocate credit among observed touchpoints using rules—last click, first click, linear, or data-driven. They can be useful for reporting journeys, but they do not observe what would have happened without the touchpoint. Incrementality requires a defensible counterfactual. Randomized experiments make a comparable untreated group; quasi-experiments attempt the same comparison with stronger assumptions.
Incrementality is also broader than conversion lift. It can apply to revenue, profit, activation, retention, emissions, support demand, or any outcome. “Incremental” does not automatically mean positive: an intervention can create incremental cost, churn, or harm. The term identifies the causal contrast, not its direction.
Incrementality framework
The fundamental estimand is the average treatment effect: incremental effect = average outcome if treated − average outcome if not treated. For a randomized test, the practical estimator is mean outcome among assigned treatment − mean outcome among assigned control. For a binary conversion metric, this is an absolute percentage-point difference; for revenue, it is incremental revenue per assigned unit.
Define five elements before measuring: the intervention; the eligible population; the randomization unit; the outcome and maturity window; and the counterfactual group. The unit may be a person, account, household, store, city, or time period. It should minimize interference: if one user’s treatment changes another user’s outcome, individual assignment may be unsuitable. A geo holdout, cluster experiment, switchback design, or matched design can be more credible.
| Approach | Counterfactual | Main risk |
|---|---|---|
| User A/B test | Randomized control users | Spillovers and incomplete exposure |
| Audience holdout | Eligible users withheld from campaign | Contamination across channels |
| Geo experiment | Randomized regions | Few units and regional differences |
| Difference-in-differences | Untreated trend | Unproven parallel-trends assumption |
Incrementality in A/B testing and product work
A/B testing is a direct incrementality method when assignment is random, stable, and measured before treatment exposure. A product team can ask whether a redesigned onboarding actually increases activation, whether a recommendation creates purchases rather than redirects them, or whether a notification causes retained use rather than a temporary click. Analyze all eligible assigned units to estimate the offer’s effect, rather than only people who clicked or rendered an element.
Predefine the primary outcome and a small set of guardrails. For a promotional intervention, incremental orders may rise while incremental margin falls due to discount cost. For a product prompt, initial engagement may rise while unsubscribe rate or long-term retention declines. The decision framework in primary and guardrail metrics helps make that trade-off explicit. Use a test long enough for outcomes to mature; see A/B test duration.
Do not inspect every segment, time window, and outcome until one appears incremental. That selection makes chance findings likely. State the hypothesis and analysis plan before launch, then label unplanned cuts exploratory. Multiple-comparisons guidance explains the false-positive problem.
Worked scenario
A B2B platform wants to know whether a renewal reminder email creates renewals or merely receives credit for renewals already planned. It randomly withholds the email from 40,000 renewal-eligible accounts and sends it to 40,000 comparable accounts. Renewal within 30 days is 18.0% in control and 18.7% in treatment.
incremental renewal rate = 18.7% − 18.0% = 0.7 percentage pointsestimated incremental renewals = 40,000 × 0.007 = 280
The email platform may report far more “email-attributed” renewals because it credits anyone who opened or clicked. The holdout estimate indicates the causal quantity. The team then evaluates uncertainty, net recurring revenue, discounts offered by the email, opt-outs, and renewal quality. It does not assume that 280 is a guaranteed future count, and it checks that the withheld group was truly excluded from all reminder variants.
Data-quality limitations
A valid design can still fail in measurement. Log assignment, eligibility, actual delivery or exposure, outcome event, timestamp, and stable identity. Delivery systems may suppress messages, users can receive a campaign through another channel, and accounts can merge. Reconcile payment or subscription outcomes to server-side records. Differences in missingness by arm can manufacture or hide an effect.
Check balance and allocation before interpretation. A sample-ratio mismatch, consent change, or targeting bug undermines the comparability that incrementality needs; investigate using SRM diagnostics. For geo and time-based studies, also examine seasonality, concurrent campaigns, stock availability, and policy changes. Sophisticated modeling cannot fully recover a counterfactual when the untreated comparison is systematically different.
Common mistakes
- Calling attribution incremental: exposure before purchase is not proof of causation.
- Using clickers as the control population: click behavior is post-treatment selection.
- Ignoring spillover: treatment can affect people assigned to control.
- Comparing unequal maturity windows: newer cohorts have less chance to convert.
- Measuring only revenue: incremental margin, retention, and customer cost may tell another story.
- Overgeneralizing: an effect for eligible users in one market may not transfer elsewhere.
Frequently asked questions
Why is incrementality important?
It prevents investment decisions based on outcomes that would have occurred without the intervention.
Is an A/B test always incremental?
It estimates an incremental effect only if assignment, exposure, measurement, and interference assumptions are credible.
Can incrementality be measured without randomization?
Sometimes, using quasi-experimental methods, but their causal claims rely on assumptions that should be tested and stated.
What is incrementality versus lift?
Lift is a difference or percentage change. Incrementality specifies that the difference represents a causal contrast against a counterfactual.
Summary
Incrementality measures what an intervention actually caused, not what an attribution rule happened to credit. Establish a credible counterfactual, measure all assigned eligible units over a mature window, and evaluate uncertainty, spillovers, and downstream trade-offs before acting.
Sources
- National Bureau of Economic Research: Field experiments
- Stanford King Center: Experimental research
- OECD: Using experimental methods