Quick definition: CUPED, or controlled experiments using pre-experiment data, reduces variance in an experiment outcome by adjusting for a covariate measured before treatment assignment.
What is CUPED?
CUPED is a variance-reduction method for randomized experiments. It retains the randomized treatment comparison but uses a predictive baseline variable to remove predictable noise from the outcome. A common example is a user’s revenue in the 28 days before assignment when the outcome is revenue after assignment. Users with high baseline revenue tend to remain relatively high-revenue users; accounting for that information makes the treatment estimate more precise.
The method does not create information about the treatment effect or make a biased experiment valid. Its value comes from a pre-treatment covariate that is correlated with the outcome and unaffected by the assigned experience. Randomization still identifies causality. CUPED changes the variance of the estimator, not the target population or the requirement to define the metric, eligibility, and analysis unit in advance.
CUPED mechanics and formula
Let Y be the post-assignment outcome and X a baseline covariate. Form the adjusted outcome Ycuped = Y − θ(X − E[X]), where θ = Cov(Y,X) / Var(X) is estimated from the planned analysis population. Centering preserves the mean treatment difference while removing variation associated with X.
Under the simple model, the remaining variance is approximately Var(Y)(1 − ρ²), where ρ is the correlation between baseline covariate and outcome. A correlation of 0.6 can therefore reduce variance by about 36%, which can shorten a fixed-horizon test or narrow its interval. The realized benefit depends on data quality, correlation, and the final analysis method.
Regression adjustment is an equivalent practical implementation: regress the user-level outcome on treatment and pre-treatment covariates, using a specification fixed before results are examined. Include the treatment coefficient and an appropriate uncertainty estimate. For unequal allocation, missing baseline data, or nonlinear relationships, use a documented estimator rather than applying a formula mechanically.
Correct use and assumptions
The covariate must be available before assignment. Purchase amount after exposure, pages viewed in the new flow, or eligibility determined by a treatment-affected event are invalid CUPED covariates because they can introduce post-treatment bias. Record the extraction cutoff, source, and user identity rule. A delayed warehouse backfill can otherwise leak future behavior into the baseline.
Analyze at the unit of randomization. If users are randomized, aggregate events to users. If stores are randomized, a store-level or cluster-aware approach is needed; pretending each customer is independent makes the interval too narrow. CUPED also assumes the adjustment is applied consistently across arms. It cannot fix broken assignment, tracking loss, or a sample ratio mismatch.
CUPED in A/B testing
CUPED is especially useful for noisy continuous metrics such as revenue per eligible user, engagement counts, or historical activity where a strong baseline exists. It can also support binary outcomes through regression-based covariate adjustment, though the estimand and model must be communicated clearly. Predeclare the covariate window, transformations, missing-value handling, and whether treatment-by-covariate interactions are permitted.
Use the unadjusted estimate as a diagnostic and report the adjusted estimate as the prespecified primary analysis. If they differ materially, investigate baseline construction, missingness, and model behavior rather than choosing the favorable one. A variance-reduction plan belongs in sample-size planning; see how to calculate sample size.
Worked scenario: subscription revenue
A subscription service tests a renewal reminder on 100,000 users. The primary outcome is 30-day net revenue per randomized user. Each user has prior-30-day net revenue measured before assignment. Historical analysis estimates a 0.5 correlation between prior and future revenue. CUPED is expected to reduce outcome variance by roughly 25%, improving precision without withholding additional traffic.
The team estimates theta on the locked analysis population, centers baseline revenue, and compares mean adjusted revenue by assignment. It reports both the dollar lift and its interval, as well as renewal rate, refunds, and unsubscribe guardrails. If high-value users have missing baseline records because an identity pipeline changed, it does not silently treat those users as zero; it follows the predeclared missingness rule and investigates the pipeline.
Interpretation
A CUPED result estimates the same average treatment effect as the corresponding randomized comparison when used correctly. Its narrower interval reflects explained baseline variation, not stronger treatment. Explain the covariate, timing, expected variance reduction, and model. Evaluate practical impact with the same care as an unadjusted result: statistical precision is not a business decision.
Limitations and common mistakes
- Using post-treatment behavior as a covariate.
- Building the baseline from a changing or mismatched identity system.
- Fitting many covariates after observing outcomes.
- Ignoring clustered assignment or repeated users.
- Assuming CUPED helps when the covariate barely predicts the outcome.
- Claiming it repairs poor randomization or incomplete measurement.
Frequently asked questions about CUPED
Does CUPED change the treatment effect?
No. It aims to estimate the same effect with less sampling variance.
Can I use several baseline covariates?
Yes, through prespecified regression or machine-learning adjustment with appropriate validation, but complexity raises implementation and overfitting risk.
Does every experiment need CUPED?
No. It is most useful when reliable baseline variables predict the outcome strongly.
Can it use revenue from after launch?
No. A CUPED covariate must be measured before assignment.
Summary
CUPED uses reliable pre-experiment data to remove predictable outcome variation and improve A/B-test precision. It is valid only when covariates are truly pre-treatment, analysis respects assignment, and the adjustment is planned and reported transparently.
Sources
- Deng et al., Improving the Sensitivity of Online Controlled Experiments
- CUPED overview
- NIST Statistical Engineering resources