Logistic Regression

What is logistic regression?

Logistic regression is a generalized linear model for a binary outcome. Rather than model the probability directly as a straight line, it models the log-odds: log(p / (1 − p)) = beta₀ + beta₁X₁ + ... + beta_kX_k. The inverse-logit function converts the fitted value back to a probability between zero and one. This makes logistic regression a natural option for user-level conversion, retention, fraud flag, or purchase outcomes.

Exponentiating a coefficient gives an odds ratio. For a binary treatment indicator, exp(beta_T) is the ratio of treatment odds to control odds conditional on included covariates. Odds are p/(1-p), not probabilities. An odds ratio of 1.20 means odds are estimated to be 20% higher; it does not generally mean conversion probability is 20% higher. The difference matters materially when the baseline outcome is common.

How the model is fitted and interpreted

Logistic regression estimates coefficients by maximum likelihood. For each independent unit, the model assigns a probability based on its predictors; estimation finds coefficients that make the observed pattern most likely under the Bernoulli model. A treatment coefficient can be tested with a Wald test, likelihood-ratio test, or score test. For an A/B test, the choice should be documented, but the effect estimate and confidence interval are more important than picking the most favorable test.

Translate coefficients into decision-friendly quantities. If control conversion is 10%, its odds are 0.10/0.90 = 0.111. An odds ratio of 1.20 implies treatment odds 0.133, corresponding to probability 0.133/(1 + 0.133) = 11.76%. The resulting absolute lift is 1.76 percentage points, not 2.0 points and not necessarily “20% conversion lift.” Marginal predicted probabilities or a directly estimated risk difference often communicate experiment effects more clearly than conditional odds ratios.

With only an intercept and a binary treatment indicator, logistic regression tests the same broad conversion comparison as standard two-group methods, though estimates and intervals are on the odds scale before transformation. Add pre-treatment covariates to improve precision or account for a planned design feature. Include treatment-by-covariate interactions only for justified, predeclared heterogeneity questions. A significant treatment coefficient in one subgroup and non-significant coefficient in another does not show that subgroup effects differ; test the interaction directly.

Assumptions and boundaries

The outcome must be coded consistently as a binary event over a stated window. A user who activates twice is still one success for a user-level activation metric; event rows must not be treated as independent Bernoulli trials if users were randomized once. Observations also need appropriate independence conditional on the model. Shared accounts, stores, or repeated user records require clustered standard errors, generalized estimating equations, mixed-effects logistic models, or a cluster-level design.

Logistic regression assumes the log-odds is linear in continuous predictors as represented in the model. A nonlinear pre-period activity relationship may require splines or categories chosen before outcome analysis. It also assumes relevant interactions and major predictors have been handled adequately for a conditional association interpretation. In a randomized treatment analysis, correct treatment assignment supplies causal identification; adding covariates is not a license to adjust for post-treatment behavior.

Rare events, complete separation, and small sample cells can destabilize maximum-likelihood estimates. If every user in one tiny subgroup converts or none does, the coefficient can diverge. Penalized methods such as Firth logistic regression may help in small or rare-event settings, but their estimand and reporting must be clear. Large-sample software output can conceal sparse-data problems, so inspect event counts by variant and key covariate levels.

Logistic regression in A/B testing

Use logistic regression when the primary question concerns a user-level binary outcome and the team wants a model-based estimate, planned covariate adjustment, or interaction. Begin with a metric specification: eligible population, success event, denominator, assignment unit, attribution window, deduplication, and missing-data policy. A conversion percentage without a consistent denominator is not repaired by a sophisticated model.

A simple experiment model might be logit(P(convert)) = beta₀ + beta_Ttreatment + beta_Ccountry + beta_Ddevice, where country and device were recorded before exposure. Report an adjusted odds ratio with an interval, plus standardized or marginal predicted conversion in each variant and absolute percentage-point lift. For a decision, stakeholders usually need the latter, expected incremental conversions, and guardrail behavior—not only an odds ratio.

Do not condition on variables caused by treatment. Including “saw pricing modal,” “number of product views,” or “started checkout” may estimate a controlled direct association rather than the total effect of the assigned experience, and may introduce bias. The primary analysis should generally follow assigned eligible users according to the predeclared experiment hypothesis and metric. Per-protocol or mediator analyses, if useful, need their own causal assumptions and labels.

Worked example: activation probability

A team randomizes 30,000 new users equally to standard or guided onboarding. Activation within 14 days occurs for 2,250 of 15,000 control users (15.0%) and 2,520 of 15,000 treatment users (16.8%). An unadjusted logistic model estimates an odds ratio of about 1.14. Its 95% confidence interval is 1.07 to 1.22, and the two-sided p-value is below 0.001.

The odds-ratio statement is correct but incomplete. The directly observed absolute lift is 1.8 percentage points, or 12% relative to the 15.0% baseline. A model with pre-treatment acquisition channel and platform yields a similar adjusted marginal difference of 1.7 points, with a 95% interval of 0.8 to 2.6. The team therefore describes the result in probability units, confirms consistent activation logging, and checks 30-day retention, support contacts, and onboarding latency.

If the team discovers that treatment users have more missing activation events, the model output is not trustworthy merely because the odds ratio is precise. Differential measurement can mimic a treatment effect. Reconcile assigned, exposed, and analyzed users and investigate allocation problems such as sample ratio mismatch before interpreting statistical significance.

Analysis and reporting workflow

1. Define the binary outcome, unit, exposure rule, attribution window, and estimand before the test begins.
2. Choose covariates and interactions based on pre-treatment availability and the analysis plan, not their observed p-values.
3. Check event counts, allocation, missingness, sparse cells, dependence, and whether all users have equal outcome opportunity.
4. Fit the planned model with appropriate cluster-aware uncertainty where necessary, and retain model diagnostics.
5. Report event counts, probabilities, absolute differences, odds ratios if useful, confidence intervals, multiplicity treatment, and product guardrails.

Risks and common mistakes

• Reading odds ratios as risk ratios. Convert results to probabilities or percentage-point changes at relevant baselines.
• Adjusting for mediators. Post-treatment covariates can change or bias the causal question.
• Ignoring clustered or repeated data. One user’s multiple events are not independent binary observations.
• Using only model fit metrics. AUC, pseudo-R-squared, and likelihood do not answer the incremental business question.
• Hiding sparse cells or separation. Inspect raw counts before trusting enormous coefficients and intervals.
• Mining many interactions. Treat unplanned segment patterns as exploratory and account for multiplicity.

Frequently asked questions about logistic regression

Is logistic regression better than a conversion-rate test?

Neither is universally better. A simple proportion comparison is transparent for a two-group primary metric. Logistic regression is useful for planned covariate adjustment, interactions, and more complex designs.

What is an odds ratio?

It is the ratio of two odds, where odds equal probability divided by one minus probability. It should not be reported as a probability increase without translating it at a stated baseline.

Can logistic regression establish causality?

Not from observational data alone. In a valid randomized experiment, the assigned-treatment coefficient can estimate a causal effect; the model still depends on correct data, outcome definition, and analysis unit.

Should I include every available variable?

No. Include pre-specified, meaningful pre-treatment variables. Adding variables indiscriminately can create sparse cells, unstable estimates, overfitting, and hard-to-audit analysis choices.

How do I report a treatment effect clearly?

Give variant event counts and conversion rates, absolute percentage-point difference with an interval, the model and covariates, and an odds ratio only as a supplementary conditional measure when it helps.

Summary

Logistic regression models binary outcomes through log-odds and is valuable for conversion-style A/B-test metrics, especially with planned covariate adjustment or interactions. Its treatment coefficient is naturally an odds ratio, but decisions should be translated into absolute probability changes. Reliable use requires a coherent binary metric, appropriate independence or clustering treatment, pre-treatment adjustment only, and rigorous data-quality checks.

Sources

• Hosmer, Lemeshow, and Sturdivant, Applied Logistic Regression.
• NIST/SEMATECH e-Handbook of Statistical Methods, Logistic Regression.
• Agresti, A., Categorical Data Analysis.

Related terms

• Two-Proportion Z-Test
• Z-Test
• T-Test
• Independent Samples T-Test
• Paired T-Test