Quick definition: An estimator is a rule that uses sample data to calculate an estimate of an unknown population quantity, such as a conversion rate or a treatment effect.
What is an estimator?
An estimator is a formula, algorithm, or procedure for learning about an unknown parameter from observed data. The parameter is the target quantity in a population or data-generating process; the estimate is the numerical result produced in one sample. For example, the sample mean estimates a population mean, and the difference between treatment and control conversion rates estimates an average treatment effect for a defined eligible population.
The distinction matters because estimates vary across comparable samples. If an experiment were repeated with new randomized users, its observed difference would not be identical every time. An estimator describes the repeated-sampling rule; an estimate is one realization of that rule. Standard errors, confidence intervals, hypothesis tests, and Bayesian posterior summaries all concern uncertainty around a target, but they begin with a clearly defined quantity and a method for estimating it.
Choosing an estimator is a scientific and product decision, not merely a coding choice. “Average revenue per assigned user,” “median order value among purchasers,” and “conversion among users who reached checkout” are different quantities. The first can answer total value of assignment; the second describes a conditional typical purchase; the third conditions on a potentially treatment-affected step. A mathematically correct estimator can still answer the wrong question.
Properties of a useful estimator
Bias describes an estimator’s average error under a model: Bias(θ̂) = E[θ̂] − θ. An unbiased estimator has expected value equal to the target parameter, but unbiasedness alone does not make an estimator preferable. Precision concerns how much estimates vary across samples, often described by variance or standard error. Mean squared error combines variance and bias as MSE(θ̂) = Var(θ̂) + Bias(θ̂)².
Consistency means that an estimator approaches the target as the amount of relevant information grows under stated assumptions. Efficiency compares precision among estimators targeting the same quantity; see efficient estimator. Robustness concerns sensitivity to departures from assumptions, such as outliers or a misspecified distribution. No property can be assessed without specifying the estimand, data structure, and model.
An estimator also needs a credible uncertainty procedure. A difference in means may be easy to compute, but its standard error must reflect the unit of randomization, repeated observations, clustering, unequal variance, and stopping design. Reporting many decimal places does not create precision; the interval and assumptions communicate what the data can support.
Common estimators and formulas
For observations X1, …, Xn, the sample mean is X̄ = (1/n) ΣXi. For a correctly defined binary outcome, the sample proportion is the mean of 0/1 indicators: p̂ = successes / eligible units. In a two-arm randomized experiment, the simple difference-in-means estimator is τ̂ = ȲT − ȲC.
Regression estimators can adjust for pre-treatment covariates to improve precision, while preserving a treatment indicator as the causal contrast under appropriate design. Ratio metrics, quantiles, survival estimates, and cluster-level means need methods appropriate to their definitions. Do not replace an assigned-user mean with an event-level ratio because the latter is easier to query. The analysis unit must match the question and randomization.
Estimators in A/B testing
For a standard A/B test, begin by writing the estimand in words: “the difference in seven-day purchase probability among all users eligible at assignment.” A simple intention-to-treat estimator compares the mean outcome of all eligible assigned users in treatment with the corresponding mean in control. It preserves randomization and includes zero outcomes for users who do not convert.
Do not condition the primary estimator on post-assignment behavior such as exposure to a widget, reaching checkout, or remaining active, unless the causal question explicitly requires a more specialized method. Treatment may affect membership in that subset, so a conditional comparison can be biased. Exposure failures should be diagnosed and reported; they are not usually a reason to remove affected users from the primary denominator.
Estimator selection should be prespecified with the metric, eligibility rule, observation window, missing-data treatment, exclusions, variance method, and stopping rule. This reduces the chance that analysts select the procedure whose result looks best. It also makes results reproducible. Related planning concepts include A/B testing, primary and guardrail metrics, and confidence intervals.
Worked scenario: signup conversion
A product team randomizes 20,000 eligible visitors to each signup page. It defines Y=1 when a visitor creates and confirms an account within 24 hours and Y=0 otherwise. Control has 900 successes, so p̂C=900/20,000=4.5%. Treatment has 1,020 successes, so p̂T=5.1%.
The estimator of absolute effect is τ̂=5.1%−4.5%=0.6 percentage points. This is an estimate, not proof that every future user has a 0.6-point gain. The team calculates a confidence interval with a proportion method appropriate to the design, checks allocation and tracking, and compares downstream activation and support contacts. It reports the original denominator, not just users who saw the form render.
If one visitor can have many sessions, session rows are not interchangeable independent observations. The estimator can instead aggregate the outcome at the user, account, or randomization unit. If randomization occurs by store, analysis must respect stores rather than treating individual transactions as independent treatment assignments.
Assumptions and interpretation
Estimator validity depends on the data-generating conditions. For a randomized treatment-control difference, key requirements include valid assignment, no material interference beyond the estimand, consistent eligibility, accurate outcomes, and an uncertainty calculation that matches the assignment structure. Missing outcome data can create bias when missingness differs by arm or relates to the unobserved outcome. More data reduce random variation but do not fix a systematically missing denominator.
Interpret estimates on their natural operational scale and with uncertainty. A statistically significant estimate may be too small to justify a launch, while an imprecise estimate may be compatible with both useful benefit and unacceptable harm. Predeclared practical thresholds and guardrails turn estimation into a decision process rather than a hunt for a favorable coefficient.
Limitations and common mistakes
- Confusing the estimate with the parameter: one observed value is subject to sampling variation.
- Choosing a convenient denominator: it can change the causal question or introduce post-treatment bias.
- Ignoring clustered assignment: standard errors can become far too small.
- Adjusting for treatment-affected variables: it can distort the total effect.
- Selecting a method after seeing results: it weakens confirmatory interpretation.
- Using an estimator without checking data quality: no formula repairs broken exposure or outcome logging.
Frequently asked questions about estimators
Is an estimator the same as an estimate?
No. An estimator is the rule; an estimate is the number produced when that rule is applied to data.
Must an estimator be unbiased?
Not always. Small, controlled bias can sometimes reduce overall error, but the target and trade-off must be explicit and appropriate to the decision.
What estimator should an A/B test use?
Use the estimator that targets the prespecified business question and respects the randomization unit. For many primary binary outcomes, that is a difference in assigned-user conversion rates.
Can regression improve an A/B-test estimator?
Yes. Pre-treatment covariates can improve precision when applied with a valid, documented model. They should not replace the randomized comparison or adjust for post-treatment behavior.
Why is the standard error important?
It describes sampling variability under assumptions and is used to quantify uncertainty. A point estimate without uncertainty is incomplete evidence.
Summary
An estimator is a data-based rule for learning about an unknown quantity. In A/B testing, define the causal target first, choose an estimator and uncertainty method that match assignment and measurement, then interpret the resulting estimate with practical thresholds, guardrails, and design limitations.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods
- OpenIntro Statistics
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments