Quick definition: Bias is a systematic deviation between an estimate, measurement, or comparison and the quantity it is meant to represent. Unlike random sampling noise, bias does not reliably disappear when an A/B test receives more traffic.
What is bias?
In statistics, bias describes a directional error. An estimator is biased when its expected value differs from the true parameter it targets. More broadly in experimentation, bias refers to a systematic process that makes the observed treatment-control difference differ from the causal effect the team wants to know. It can come from who enters the analysis, which users receive the variant, what is recorded, how outcomes are attributed, or how results are selected and reported.
Bias differs from variance. Variance is ordinary random fluctuation across samples; larger samples usually narrow it. Bias shifts the result in a consistent direction. A precisely measured but biased estimate can be dangerously persuasive because its confidence interval is narrow around the wrong answer. More traffic will not fix a broken exposure event, a post-treatment denominator, or self-selection into variants.
The target quantity should be stated first. For example: “the seven-day net-revenue-per-assigned-user effect of offering the new checkout to eligible mobile visitors.” This target identifies the population, treatment, outcome, and window. Bias is assessed relative to that target, not as a vague label for any unwanted result.
Common sources of bias
| Type | How it arises | Example in experimentation |
|---|---|---|
| Selection bias | Analyzed groups differ before treatment | Visitors choose which design to see. |
| Measurement bias | Outcome is recorded differently or inaccurately | Only the new checkout fires a purchase event. |
| Attrition bias | Missing outcomes differ by arm | Treatment crashes more often before telemetry loads. |
| Survivorship bias | Only units that remain are examined | Comparing retention among users who reached a later step. |
| Reporting or multiple-testing bias | Favorable analyses are selected after inspection | Publishing one winning segment from fifty explored. |
Confounding is closely related. In a before-and-after rollout, a new design may coincide with a holiday, campaign, inventory change, or product release. The observed change mixes treatment with those other causes. Concurrent random assignment is designed to balance measured and unmeasured pre-treatment factors in expectation, making the control group a credible counterfactual.
Randomization is necessary but not sufficient. It can be undermined by faulty bucketing, eligibility logic applied after assignment, uneven treatment delivery, identity loss, or an analysis that filters groups differently. A randomized experiment with biased measurement is not automatically trustworthy.
Bias in statistical estimation
For an estimator θ̂ of a parameter θ, formal statistical bias is:
Bias(θ̂) = E[θ̂] − θ
An estimator with zero bias is unbiased in repeated samples under its model assumptions. That does not guarantee it is best for every use: an estimator can trade a small bias for lower variance, and real-world assumptions can fail. In A/B testing, the more practical concern is whether the estimated difference in observed metrics corresponds to the intended causal effect.
Suppose the true treatment effect on purchase probability is zero, but a logging defect records 5% of treatment purchases twice while control events are deduplicated. The measured treatment rate will be systematically high. The estimate is biased even if the sample contains millions of users and the statistical test returns an extremely small p-value.
Bias in A/B testing
Well-designed A/B tests prevent the most serious selection bias by assigning eligible units randomly and persistently before they encounter a variant. Do not let users choose a version, assign by a mutable property influenced by treatment, or compare two time periods as if they were concurrent arms. The A/B testing design provides a causal comparison only for the population that was actually eligible and correctly randomized.
Preserve the assignment-based denominator. Imagine a recommendation module shown only after a user reaches a product page. If treatment changes product-page reach, comparing conversion “among module viewers” conditions on a post-treatment behavior. It may make treatment and control viewer groups non-comparable. Analyze all eligible assigned users for the overall effect, then use viewer analyses only as clearly labeled diagnostics.
Measurement requires equal treatment. Event names, client versions, consent handling, attribution windows, refund treatment, bot filters, and identity joins should be the same across arms unless their difference is explicitly the treatment. Server-side records can reduce client-blocking problems for transactions, but they still require a reliable link to assignment. An A/A test can expose unexpected rate differences when both arms intentionally receive identical experiences.
Worked example: a biased checkout result
A retailer tests a simplified checkout. It assigns 50,000 eligible users to each arm by user ID. The dashboard reports purchase conversion of 4.8% in treatment and 4.2% in control, apparently a +0.6-point lift. The result looks decisive because the sample is large.
During quality assurance, the team finds that the new checkout logs its client-side “purchase completed” event immediately after payment authorization. Control logs only after server confirmation. Some treatment authorizations later fail or are canceled, and the treatment event is also emitted twice on an app retry. Reconciled order data shows treatment conversion is 4.28% and control is 4.21%, a much smaller difference with a different uncertainty range.
The original analysis was not merely noisy; it measured different outcomes in the two arms. Increasing the sample would have made the misleading result more statistically certain. The team switches the primary metric to deduplicated, server-confirmed paid orders within seven days, documents the incident, reruns the test if needed, and retains payment failures and refunds as guardrails.
Diagnosing and preventing bias
Start before launch. Write eligibility, assignment unit, exposure rule, metric formulas, exclusions, and stopping rules in a test plan. Validate assignment balance and sample-ratio mismatch. Run event-level QA on both experiences, including reloads, failures, consent states, cross-device identity, and delayed outcomes. Reconcile crucial business outcomes to source-of-truth systems.
At readout, compare pre-treatment attributes and data completeness, but avoid treating every small imbalance as evidence of failure. Randomization permits chance differences. Investigate large, systematic, or operationally implausible discrepancies, especially if they align with a deployment, browser, geography, or variant. Check exposure volume, zero-event share, event timing, missing IDs, duplicate records, and delayed maturation.
Pre-specification reduces analysis-selection bias. Select one primary metric, a fixed population, a stopping method, and a limited set of planned segments. Exploring additional cuts can create hypotheses, but it cannot supply the same confirmatory evidence. Multiple comparisons increase the chance that at least one chance result looks impressive.
Do not “correct” bias by quietly dropping inconvenient observations. Exclusions need a rule that is independent of variant and outcome, a documented rationale, and sensitivity analysis when the decision could change. Undocumented filtering can introduce a second bias.
Interpretation and limits
No experiment is free from every threat. The goal is to identify material pathways that could change the decision and make evidence proportionate to risk. Some bias concerns affect internal validity: whether the variant caused the observed effect for tested units. Others affect external validity: whether the result generalizes to another platform, country, season, or future implementation. A clean experiment on desktop visitors does not automatically estimate the effect on all customers.
Bias should not be invoked as a generic reason to reject an unfavorable result. Ask a concrete question: what mechanism would shift the estimate, in what direction, and what observable diagnostic would confirm or refute it? This makes investigation useful rather than post-hoc rationalization. When the answer is uncertain, report the limitation and choose a staged rollout, replication, or additional measurement.
Common mistakes
- Confusing bias with random error: bigger samples reduce noise, not systematic defects.
- Using historical performance as randomized control: concurrent conditions may differ in many ways.
- Analyzing only exposed or surviving users: treatment may affect entry into that subset.
- Ignoring unequal telemetry: an event can be accurate in one arm and biased in comparison.
- Searching until a segment wins: post-hoc selection inflates apparent effects.
- Overgeneralizing a valid result: a causal effect in one tested population may not transfer elsewhere.
Frequently asked questions
Does randomization eliminate all bias?
No. It addresses pre-treatment selection and confounding in expectation, but delivery, measurement, attrition, interference, analysis choices, and generalization can still bias a result.
Can a large sample remove bias?
No. A large sample can make a biased estimate very precise. Fix the process that causes systematic error.
What is post-treatment bias?
It occurs when analysis conditions on a variable affected by treatment, such as viewing a page, completing an intermediate step, or remaining active. That can make the groups no longer comparable.
How does an A/A test help?
Because both arms receive the same experience, persistent metric differences, allocation anomalies, or exposure gaps can reveal implementation or analysis problems.
Is bias always intentional?
No. Most experimentation bias comes from ordinary system behavior, unclear definitions, tracking gaps, or well-intentioned analysis decisions rather than deliberate manipulation.
What should I do after finding a bias?
Assess whether it can change the decision, stop relying on affected results, document the cause, correct the measurement or design, and rerun or reanalyze only with a defensible pre-specified rule.
Summary
Bias is systematic error between an observed estimate and the effect or quantity it is intended to represent. In A/B testing, random assignment is a strong defense against selection bias, but valid results also require stable eligibility, consistent exposure and measurement, complete outcomes, disciplined analysis, and appropriate limits on generalization. Diagnose concrete pathways, not just suspicious results, because additional sample size cannot repair a biased comparison.
Sources
- NIST/SEMATECH e-Handbook: Sources of variation and bias
- CDC: Bias and confounding
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments