Quick definition: Data leakage occurs when an analysis, model, metric, or decision uses information that would not have been available at the point it claims to measure, or uses post-treatment information in a way that biases a causal comparison.
What is data leakage?
Data leakage is an information-boundary failure. It can make a predictive model look unusually accurate because it has learned from the future, and it can make an experiment look persuasive because an analysis conditioned on an outcome affected by treatment. The details differ, but the common error is allowing information from outside the intended decision point to influence the estimate.
In an A/B test, the most consequential form is post-treatment leakage or selection. A team might compare checkout completion only among people who clicked a new call to action, even though the call to action itself changes who clicks. That comparison no longer estimates the effect on all eligible assigned users. It estimates behavior in groups whose membership was partly caused by the variants, so control and treatment are not generally comparable. Leakage can also occur when an outcome is joined using a future status, when a segment is defined after exposure, or when a feature used for targeting includes events from after assignment.
Not all use of later information is invalid. A cumulative seven-day revenue outcome necessarily occurs after assignment; it is valid when measured for every planned eligible unit over the same window. The issue is using later information to decide who enters the analysis, what treatment they are assigned, or what features were supposedly known before a decision.
Common leakage patterns
Target leakage occurs when a predictor contains the outcome itself or a downstream proxy: for example, using a “refund issued” field to predict whether an order will be refunded. Temporal leakage occurs when a feature is calculated using future events, such as a 30-day activity total for a model intended to score users on day seven. Train-test leakage occurs when preprocessing, aggregation, or identity reconciliation lets test-set facts influence training. In experiments, conditioning leakage occurs when analysis filters or segments use a behavior influenced by the treatment.
| Pattern | Example | Why it misleads |
|---|---|---|
| Post-treatment filter | Analyze payment success only for checkout starters | The variant may change who starts checkout. |
| Future feature | Use next-month spend to target a current offer | The feature was unavailable when targeting occurred. |
| Outcome proxy | Predict churn with a cancellation-confirmed event | The proxy nearly reveals the label. |
| Cross-split contamination | Fit a normalizer using all training and evaluation rows | Evaluation data influences model choices. |
| Identity backfill | Join anonymous exposure to a later account only for registrants | Successful users have better observability. |
| Dashboard hindsight | Pick the best-looking metric after examining many | Selection inflates apparent evidence. |
Leakage is often subtle because the leaked field is technically available in a warehouse. Warehouse availability is not the same as decision-time availability. Every source field should carry an event time, availability time, and semantic description. A historical backfill must reproduce what would have been known at the intended cutoff rather than reading a current table whose values have been revised.
Instrumentation and data mechanics
Prevent leakage by recording a causal timeline. For each randomized unit, retain eligibility time, assignment time, exposure time, outcome time, ingestion time, and any identity-resolution time. Define a feature cutoff separately from an outcome window. For example, an email recommendation model may use activity observed through midnight before send time and evaluate purchase through seven days after send time. Late-arriving events should be handled according to their event time and documented availability policy, not accidentally incorporated into a past snapshot.
Experiment datasets should begin from the planned assignment population. Join pre-treatment attributes using values frozen before assignment or explicitly constructed as of that time. Then attach outcomes in the specified follow-up period, retaining zero outcomes. If downstream events are used for diagnostics, label them as post-treatment and do not use them to redefine eligibility. Stable IDs and event IDs are important: a fuzzy match performed only after conversion can create more linkage for one arm or outcome status than another.
Data-access practices matter too. Version metric definitions, save analysis extracts, and record query cutoffs. A dashboard that automatically includes corrected historical data can be excellent for operations but unsuitable for reproducing an interim experiment decision unless its revision behavior is understood. Keep an auditable snapshot or queryable raw lineage for decision-critical analyses.
Experimental impact and causal interpretation
Randomization establishes comparable groups before treatment. Conditioning on a post-treatment variable can break that comparability. Suppose a redesigned onboarding flow reduces the number of people who reach a final form, but makes the remaining form users more motivated. Completion among final-form visitors might increase even as completed accounts per assigned user decreases. The former may diagnose form usability; it cannot substitute for the primary intention-to-treat outcome because the set of final-form visitors differs by variant.
Similarly, excluding users with a missing exposure event can bias a comparison if the treatment changes loading or tracking failures. Exposure-based analyses can be appropriate for a defined estimand, but they require a defensible opportunity-to-expose rule and careful reporting of who was excluded. Start with assignment-based outcomes for the rollout question whenever that aligns with eligibility, then use exposure and funnel analyses as secondary evidence.
Leakage can also appear in exploratory segmentation. Finding that a treatment “works” for users who made three purchases during the test uses post-assignment behavior to define the segment. Prefer segments based on attributes measured before assignment, such as prior purchase history or declared device type. Even then, prespecify important subgroup questions and account for multiplicity; the article on multiple comparisons explains why many opportunistic cuts produce fragile findings.
Practical workflow: an offer-targeting experiment
A subscription company tests whether a retention offer improves renewal. It plans to target users whose pre-period activity score is below a threshold. The team first builds a time-indexed feature table: each score is calculated only from events with event times before the weekly assignment cutoff, and the job records when the score became available. It then randomizes eligible users, logs assignment and offer exposure, and observes paid renewal for 30 days after assignment.
- Write the timeline. List every field and classify it as pre-eligibility, pre-assignment, post-assignment diagnostic, or outcome.
- Freeze eligibility. Save the target population and score version at assignment. Do not recalculate it later using new activity.
- Build from assignments. Create one row per eligible assigned user and attach a zero or one renewal outcome for a uniform window.
- Audit filters. For every exclusion, state when the criterion is observed and whether treatment could affect it. Escalate post-treatment filters.
- Evaluate alternatives carefully. Report renewal among all assigned users as primary; report offer-view and redemption rates as diagnostics without claiming they are unbiased treatment effects.
During review, an analyst proposes excluding users who contacted support because the offer was not shown to some of them. Support contact occurs after assignment and can be influenced by the offer, so the exclusion would leak treatment effects into membership. The team instead reports support contacts as a guardrail and investigates delivery failure separately. This maintains the planned population while revealing an implementation problem.
Assumptions and limitations
A strict time boundary does not solve every bias. Pre-treatment attributes may themselves be missing, stale, or affected by an earlier intervention. Identity resolution can introduce selection when only logged-in users have a stable history. In some systems, event arrival order differs from event time because of offline clients or delayed partners. Document which clock governs eligibility and which records can be revised, then assess whether the policy affects variants comparably.
Some analyses intentionally estimate conditional effects, such as the effect among users who were actually exposed under a carefully designed instrumental-variable framework. These require stronger assumptions and specialized methods; they are not justified by simply filtering a standard A/B dataset. Clarity about the estimand is more useful than claiming one universal “true” effect.
Common leakage failures
- Defining a segment with events during the test. Use attributes measured before assignment for causal subgroup comparisons.
- Recomputing eligibility after outcomes mature. Persist the original assignment population.
- Joining from converters outward. Begin with all eligible assignments so non-converters remain represented.
- Using current-state tables for historical features. Reconstruct data as it was known at the decision cutoff.
- Hiding post-treatment filters in SQL. Label each filter’s time relationship to treatment in the metric contract.
- Calling a funnel diagnostic the primary effect. Pair downstream rates with an upstream, pre-specified denominator.
Frequently asked questions
Is a conversion outcome itself data leakage?
No. Outcomes occur after treatment by design. Leakage occurs when post-treatment information changes eligibility, feature construction, assignment, or an analysis comparison without a valid causal method.
Can I segment results by users who saw the variant?
Only with caution. Seeing the variant may depend on treatment delivery and behavior. Assignment-based segments defined before treatment are usually safer for standard A/B reporting.
How do late events affect leakage?
Late events can leak future knowledge into a historical feature if availability is ignored. Use clear as-of cutoffs and record both event and arrival time.
Does preprocessing cause leakage?
Yes. Fitting imputers, normalizers, vocabulary, or feature selection on data that includes an evaluation set lets evaluation information influence a model.
What is the first check when a result seems too strong?
Review the population construction, all joins and filters, feature cutoffs, and whether any post-assignment behavior determined inclusion or segmentation.
Summary
Data leakage lets information cross a boundary it should not cross. Preserve the assignment population, distinguish pre-treatment attributes from later outcomes, construct historical features as of their decision time, and treat post-treatment filters as potential bias. These controls keep a convincing-looking result tied to the causal question it claims to answer.
Sources
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments
- NIST: Data Quality Guidelines
- AB-Labz: How to Write an A/B-Test Hypothesis