Quick definition: Missing data is information that should be available for a defined record or analysis but is absent, delayed, unreadable, or excluded. In experiments, missingness is dangerous when it differs by variant or is related to the outcome.
What is missing data?
Missing data includes absent events, null properties, broken joins, late records outside an observation window, consent-restricted rows, and records removed by a transformation rule. It is not simply an empty cell. A value can be unknown, not applicable, deliberately withheld, invalid, or not yet received; these states need separate representation because they imply different remedies and different limits on interpretation.
The term does not mean every incomplete record is unusable. A missing optional display property may be irrelevant to a purchase metric. A missing variant, timestamp, identity key, or primary outcome, however, can change the analysis population. Teams should document which fields are required for each metric, why they can be missing, and whether the omission occurs before or after a user receives treatment.
How missingness enters a data pipeline
Collection can fail because of network loss, browser blocking, consent status, SDK initialization, app crashes, offline queues, or a code path that never fires an event. Transport and storage can reject malformed schemas, truncate properties, or delay batches. Transformation can create missingness through failed parsing, a left join with no match, deduplication, bot filtering, or a timezone boundary. Reporting can hide it by dropping nulls or converting them to a default category.
Record failure reasons and preserve raw arrival evidence. Monitor event volume, field completeness, rejected records, queue age, join rate, and data maturity by producer version and platform. Distinguish event time from ingestion time: a purchase that arrives late is not necessarily absent, but it is unavailable to a report run before its maturity date. A metric contract should state a cutoff and backfill policy so analysts do not compare an immature treatment week with a mature control week.
Statistical terminology is useful but should not obscure product mechanics. Missing completely at random is rare; missingness often depends on observed factors such as device type or on unobserved behavior such as purchase intent. If a new checkout causes browsers with stricter tracking protection to lose events, excluding those records may change the estimand. Diagnose operational causes before selecting an imputation method.
Experiment impact
Randomization applies to units at assignment, not automatically to units remaining after missing records are filtered out. If exposure or conversion is missing more often in one arm, the reported groups may no longer be comparable. Missing outcome data can falsely raise or lower a conversion rate; missing identity data can remove cross-device conversions; missing timestamps can shift users into the wrong window. The direction is not always obvious, so do not assume missing data is conservative.
Define the primary analysis population before launch. For an intent-to-treat estimate, retain all eligible assigned units in the denominator whenever possible and obtain outcomes from a common source. For an exposure-based estimate, state the exposure rule and evaluate exposure completeness by arm. Compare missingness rates and reasons across variants, then investigate material differences before interpreting a p-value. Guidance on p-values in A/B testing applies only after the data represents the planned comparison.
Imputation is not a universal repair. Replacing missing outcomes with zero assumes a behavioral outcome that may not be true; carrying forward a value can create post-treatment bias; model-based imputation adds assumptions that must be justified. Sensitivity analyses can show how conclusions change under plausible bounds, but they cannot turn unknown values into observed evidence. Report coverage and the population represented by the final estimate.
Practical QA scenario
A travel site tests a faster search results page. The primary metric is booking within twenty-four hours of exposure. Treatment uses a new client rendering path; on older Android devices, the exposure event fires before consent initialization and is discarded. Booking records remain available from the backend. A naive exposed-user analysis therefore excludes more treatment users, making treatment conversion look unusually high.
The team compares assigned traffic with exposure logs, identifies the missing path by app version, and retains an assigned-user analysis linked to backend bookings as the launch decision metric. It repairs exposure collection, runs an A/A check on affected versions, and labels the prior exposed-user estimate as invalid. Its dashboard now separates “not applicable,” “consent denied,” “late,” “rejected,” and “unmatched” rather than displaying one generic null rate.
- List required fields and events for the metric and their acceptable latency.
- Measure missingness and reason codes by arm, platform, version, and time.
- Trace the first pipeline boundary where coverage diverges.
- Choose correction, restriction, or sensitivity analysis before viewing business results.
- Record the population excluded and the residual uncertainty in the decision.
Limitations and failure modes
Completeness alone is not quality. A field can be nearly complete but semantically wrong, and a low completion rate can be harmless for an optional diagnostic. Backfills may repair historical records but can use information unavailable at decision time, so retain original and corrected versions. Privacy choices must be respected; attempting to infer every withheld value can be inappropriate as well as statistically fragile.
- Null collapse: unknown, unavailable, and not-applicable values are combined.
- Asymmetric filtering: a post-treatment condition removes records from one arm.
- Premature reporting: late outcomes are treated as non-outcomes.
- Unexamined joins: missing identity mappings silently shrink the numerator.
- Convenient imputation: values are filled to preserve a preferred result.
Important: Never describe a result as representative of all assigned users when missingness has silently restricted the analysis to a different population.
FAQ
Is a null value always missing data?
No. It can mean unknown, intentionally withheld, not applicable, or a valid empty value. Store and document the state rather than treating all nulls alike.
Can late events be included?
Yes if the metric’s maturity rule permits them. Recompute all relevant arms with the same cutoff instead of adding late data selectively.
When should a test be stopped for missing data?
Stop or block the decision when missingness materially affects required exposure, outcome, or identity evidence and is unexplained or asymmetric by variant.
Does imputation solve experiment missingness?
Only under explicit assumptions. It should be a documented analysis choice, not a substitute for fixing collection or reporting coverage.
Summary
Missing data is a pipeline and analysis problem, not just a blank value. Classify why data is absent, monitor it by variant, preserve the intended population, and communicate any coverage limits before acting on an experiment result.
Useful incident notes answer practical questions: when did the gap begin, which records are affected, whether it is balanced by variant, whether raw evidence remains available, and whether a corrected dataset can be produced without guessing. Add a recovery date and repeat the original validation on a fresh interval after the fix. This makes missingness observable over time rather than allowing it to reappear as a dashboard anomaly. If customers who decline measurement are excluded by design, document that scope honestly instead of representing the result as all-user behavior.
Sources
- NIST/SEMATECH Statistical Handbook
- NIST Privacy Framework
- Microsoft Research: Online Experimentation at Microsoft