Quick definition: Linear regression models the expected value of a continuous outcome as a linear combination of predictors. In a randomized A/B test, a regression with a treatment indicator estimates the adjusted mean treatment effect and can improve precision when it includes pre-treatment covariates.
What is linear regression?
Linear regression relates an outcome Y to one or more predictors X. In its simplest form, Y = beta0 + beta1X + error. The intercept is the modeled outcome when X is zero, and beta1 is the expected change in Y for a one-unit increase in X, holding other included predictors fixed. With several predictors, the model is Y = beta0 + beta1X1 + ... + betapXp + error.
Ordinary least squares (OLS) estimates coefficients by minimizing the sum of squared residuals: the differences between observed outcomes and fitted values. A coefficient is an association conditional on the variables in the model; it is not automatically causal. In a randomized experiment, however, a treatment indicator has a causal interpretation under valid randomization and delivery because treatment assignment is independent of potential outcomes. Regression then provides a convenient way to estimate the treatment-control mean difference and adjust for pre-treatment predictors.
Model mechanics and coefficient interpretation
For an A/B test, let T equal one for treatment and zero for control. The model Y = beta0 + betaTT + error is algebraically equivalent to comparing group means: beta0 is the control mean and betaT is treatment minus control. Add a baseline covariate Z: Y = beta0 + betaTT + betaZZ + error. If Z predicts the outcome and is measured before assignment, residual variance can fall, narrowing the standard error of betaT.
An interaction allows effects to vary by a predeclared characteristic. With device indicator D, include T × D. The coefficient on that product tests whether the treatment effect differs between the reference device and D, not whether the treatment is significant in one subgroup but not another. Subgroup interactions need power and multiplicity planning; comparing two separate p-values is not a valid test of heterogeneity.
Coefficient estimates require uncertainty estimates. Conventional OLS standard errors assume independent, homoscedastic residuals. Heteroscedasticity—residual spread changing with fitted values or groups—is common in product metrics. Heteroscedasticity-consistent robust standard errors protect asymptotic inference against variance misspecification, but they do not address correlated observations. Cluster-robust standard errors or cluster-level analysis are required when users are nested in accounts, stores, or experiments randomized by cluster.
Assumptions and limitations
Linearity concerns the conditional mean: the model should describe how the expected outcome changes with predictors, not require every individual outcome to lie on a line. For a treatment indicator, the estimated difference in conditional means is naturally linear. Continuous covariates may need transformations, splines, or flexible pre-specified functions if their relationship with the outcome is curved. Adding arbitrary functions after looking for significance risks overfitting and invalid inference.
OLS coefficients are unbiased for a randomized treatment effect under valid assignment even if unrelated covariates are omitted, but model misspecification can affect efficiency and interval accuracy. Avoid adjusting for variables affected by treatment, such as purchases, post-exposure engagement, or delivered experience, because they can block part of the effect or open biasing paths. Adjust only for pre-treatment variables chosen based on availability and predictive value before analysis. A guide to A/B-test hypotheses can help distinguish outcomes from adjustment variables.
Outliers and influential points warrant investigation, especially for revenue and time metrics. A high-leverage predictor value can strongly influence the fitted relationship; a rare high outcome can affect a mean that may still be the relevant estimand. Do not remove observations because they hurt the result. Diagnose data validity, state pre-specified robust or winsorization rules, and use sensitivity analyses that answer clearly defined alternate estimands.
Linear regression for experiment analysis
Regression is often used as analysis of covariance (ANCOVA) in online experiments. Suppose prior 30-day spend predicts post-assignment 14-day spend. Including prior spend can reduce noise and increase power without changing the causal target, because it was observed before treatment. This has the same broad purpose as CUPED: use pre-experiment information to improve precision. The model should retain the treatment indicator and report the treatment coefficient in original business units.
For multi-variant experiments, use treatment indicators with one omitted reference group, normally control. Each coefficient estimates a treatment-versus-control difference. Predeclare whether the decision family includes all coefficients, selected contrasts, or an omnibus test, then apply a suitable multiplicity procedure. A regression table with ten unadjusted variant p-values does not solve the problem discussed in multiple comparisons for A/B tests.
Linear regression is not automatically the best model for every metric. A 0/1 conversion outcome has a linear probability model interpretation, but logistic regression keeps fitted probabilities in range and is often preferable. Revenue can have a heavy right tail and many zeros; the mean can remain the product estimand, while robust uncertainty or a suitable resampling approach handles distributional behavior. The analysis unit must remain the randomized unit or use a valid dependence adjustment.
Worked example: adjusting for pre-period behavior
A team randomizes 80,000 eligible users to a recommendation algorithm and measures 28-day net revenue per assigned user. It fits revenue = beta0 + betaTtreatment + betaPpre-period revenue + error. The treatment coefficient is $0.38, with a robust 95% confidence interval of $0.10 to $0.66 and p = 0.008. The unadjusted mean difference was also $0.38, but its interval was $0.04 to $0.72.
The equal point estimates are expected in a large randomized experiment: covariate adjustment mainly improves precision, not the target effect. The narrower adjusted interval reflects that prior revenue explained some variation in future revenue. The team reports that the covariate was pre-assignment, the unit was a user, robust standard errors were used, and the interval represents incremental revenue per eligible user. It also verifies exposure logging, refunds, support burden, and latency before rollout.
If the team had adjusted for “number of recommendations clicked,” the coefficient would answer a different and potentially biased question because clicks occur after treatment and may mediate its effect. A smaller treatment coefficient after such adjustment would not mean the algorithm failed; it could mean the model removed the very pathway through which it works.
Practical workflow
- Specify the outcome, randomized population, treatment coding, covariates, interactions, clustering, and estimand before analysis.
- Use only pre-treatment covariates for precision adjustment and verify their definitions are stable across variants.
- Aggregate to the appropriate unit, inspect distributions, residual behavior, influential points, and missingness.
- Use robust or cluster-robust standard errors when warranted, and state the exact variance estimator.
- Report treatment estimates and intervals in business units, alongside diagnostic and guardrail evidence rather than a model fit statistic alone.
Risks and common mistakes
- Calling every coefficient causal. Only a credibly randomized treatment variable supports that interpretation; ordinary predictors are conditional associations.
- Adjusting for post-treatment variables. This can distort or redefine the treatment effect.
- Ignoring clustered data. Robust errors for heteroscedasticity do not correct within-account or within-store correlation.
- Using R-squared as treatment success. It measures fit to observed variation, not incremental product value or valid causal evidence.
- Searching interactions after the fact. Many subgroup terms create false positives unless planned or confirmed.
- Reporting transformed effects without translation. Product decisions need an estimand and effect expressed in a meaningful scale.
Frequently asked questions about linear regression
Is linear regression the same as a t-test for two groups?
With only an intercept and a binary treatment indicator, OLS gives the same group-mean difference as the corresponding linear comparison. Regression becomes especially useful when adding planned covariates, interactions, or clustering adjustments.
Should I adjust for baseline variables in every A/B test?
Adjusting for strongly predictive pre-treatment variables can improve precision, but it should be planned and kept simple. It does not compensate for invalid assignment or tracking.
What does a negative treatment coefficient mean?
With treatment coded one and control zero, it means the estimated treatment outcome is lower by that many outcome units, conditional on the specified model. Confirm coding and metric direction before interpreting it as harm.
Can linear regression model conversion?
It can estimate an absolute probability difference using a linear probability model, but fitted values can leave the 0–1 range. Logistic regression or a direct proportion method is often more appropriate for binary outcomes.
Do robust standard errors fix all model problems?
No. They address a class of heteroscedasticity issues asymptotically. They do not fix biased covariate selection, clustering, invalid randomization, or a mismatched estimand.
Summary
Linear regression estimates an expected continuous outcome from predictors. In an A/B test, a treatment indicator estimates a mean effect, while carefully selected pre-treatment covariates can reduce variance. Valid use depends on the randomized unit, a stable estimand, appropriate dependence-aware uncertainty, and avoiding post-treatment adjustment. Regression output is evidence to interpret with data checks and product thresholds, not an automatic launch decision.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, Linear Regression.
- Freedman, D. A. (2008), On Regression Adjustments in Experiments with Several Treatments.
- Imbens and Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences.