Quick definition: Data validation is the repeatable process of checking that collected, transformed, and analyzed data conforms to an explicit contract and is credible for its intended use.
What is data validation?
Data validation turns an informal belief—“the dashboard looks right”—into testable rules. A rule may verify that required fields are present, values belong to an allowed set, timestamps are plausible, an event can be joined to assignment, or a metric reconciles to an independent system. In experimentation, validation applies not only to the final metric table but to the full measurement path from eligibility and randomization through exposure, outcome capture, ingestion, transformation, and reporting.
Validation differs from cleaning. Cleaning changes or removes records; validation measures whether records satisfy a stated expectation. A pipeline may quarantine an invalid currency or deduplicate a retry, but it should retain counts of what was changed and why. Otherwise a convenient “clean” output can conceal an arm-specific defect. It also differs from statistical testing: a p-value assesses a model-based pattern in observed data, while validation asks whether the observed data was produced and interpreted as intended.
The central discipline is to write the contract before relying on the data. “Purchase event is valid” is too vague. A usable contract says that a completed purchase has an immutable order ID, a nonnegative net amount in a known currency, a server timestamp, a stable customer or guest key where available, and exactly one terminal status for an order. It identifies the authoritative source when fields conflict and the version of the application that emitted the record.
Validation layers and controls
Strong validation uses layers because no single check detects every failure. At collection time, client and server instrumentation should validate schema, types, event name, required properties, and payload size. During transport, monitor delivery success, retry behavior, ordering guarantees, and duplicate-message risk. At ingestion, test freshness, volume, partition completeness, and schema evolution. At transformation, test uniqueness, referential integrity, join coverage, business rules, and aggregate reconciliation. At reporting, test whether filters, date boundaries, and metric labels match the documented definition.
| Layer | Example assertion | Failure it catches |
|---|---|---|
| Event collection | experiment_id and variant are valid enums | Undeclared or misspelled variants |
| Raw ingestion | Hourly volume is within an expected range | Dropped stream or duplicate replay |
| Identity and joins | Outcome-to-assignment match rate is stable by arm | Variant-specific identity loss |
| Metric model | One row per planned analysis unit | Accidental event-level weighting |
| Business reconciliation | Net orders agree with the payment ledger | Wrong status, refund, or currency logic |
| Release regression | Golden journey emits the expected event sequence | Instrumentation removed by a UI change |
Assertions require tolerances. A zero-null rule is appropriate for an event key that is technically required; a freshness rule might allow a 30-minute delay for 99% of events. Express the population, time window, segmentation, threshold, owner, severity, and response. “Traffic looks low” cannot be automated or audited. “Treatment exposure volume on each platform must be within 5% of expected allocation after 1,000 assignments” can be monitored and investigated.
Instrumentation and data mechanics
An event contract should be versioned alongside the feature. Additive optional fields are generally safer than reusing a property to mean something new. If an event must change semantics, emit a version field or a distinct event and maintain a migration plan. Timestamps need similarly precise meaning: client-render time, server-receipt time, payment settlement time, and warehouse-load time answer different questions. Store enough context to diagnose skew and late arrivals without using sensitive values unnecessarily.
For experiment data, validate three linked records. First, the assignment record identifies the randomization unit, experiment, variant, eligibility result, and assignment time. Second, the exposure record identifies the opportunity to receive the change and delivery version. Third, the outcome record identifies the action, outcome key, and time. The analysis can join these records through stable keys and documented fallback logic. If one is absent, do not invent an exposure or treat an outcome-only record as proof of randomized eligibility.
Use synthetic or controlled test accounts to validate end-to-end paths. Their expected IDs and events are known, making it possible to observe whether data arrives at every destination and whether the metric model returns the planned result. Protect these accounts from production reporting or label them explicitly so that their test traffic does not contaminate outcomes. Production monitoring is still necessary because real devices, consent choices, network conditions, and payment states produce paths that a test fixture may not cover.
Why validation protects experimental validity
An A/B result relies on comparable measurement, not only comparable assignment. Consider a treatment that adds a client-side step and delays the event listener. If the primary conversion event is more likely to be lost in that path, treatment conversion will appear lower even if true conversion is unchanged. A schema check may pass because the surviving events are well formed; only coverage checks by arm, platform, and release reveal the differential loss.
Validation also catches denominator errors. A metric defined as converters divided by eligible assigned users should preserve all eligible assigned users, including those with zero outcomes. If a transformation joins from outcomes to assignments, it can silently drop non-converters and turn the rate into a different, often meaningless ratio. Compare the assignment universe to the final analysis universe at every stage. The sample ratio mismatch article provides a related diagnostic for allocation, but matching allocation alone is not proof that the outcome table is complete.
Metric validation should precede result interpretation. A large lift in a secondary dashboard may be an early signal, not a decision. Confirm the event definition, maturity window, deduplication, attribution, and arm-level quality checks before selecting a winner. Then report uncertainty using appropriate methods; confidence intervals describe sampling variation, not the validity of an unchecked pipeline.
Practical workflow: validating a checkout metric
A retailer launches an experiment that changes the payment form. Its primary metric is paid orders per eligible assigned user within 14 days. The analytics team creates a validation matrix before ramping traffic.
- Specify the source of truth. Payment settlement records determine paid status and net amount; the browser event supplies user experience diagnostics but is not the sole financial source.
- Test known paths. Use controlled orders for success, decline, retry, cancellation, refund, guest checkout, and cross-device return. Verify one immutable order ID maps to one terminal state.
- Check experiment linkage. Confirm every eligible assignment has an analysis row, treatment exposure is recorded after rendering, and settlement records join to assignment at comparable rates across variants.
- Set live monitors. Track assignment, exposure, payment attempts, settled orders, duplicate order IDs, null currencies, and ingestion lag by variant and platform. Alert on unexpected differences.
- Reconcile after maturity. Compare warehouse net revenue and paid orders to the ledger for the fixed 14-day window. Investigate differences before publishing an effect.
Suppose browser purchase events are 7% lower than the payment ledger on mobile Safari, but the gap is equal in control and treatment. The team can use ledger-backed orders for the primary metric and retain browser events for funnel diagnostics, while documenting the limitation. If the gap is 7% in treatment and 1% in control, it has evidence of differential measurement and should not claim the browser-event lift or loss represents customer behavior.
Assumptions and limitations
Passing rules establishes only that the rules were adequate and satisfied. Validation cannot detect a missing field that no one specified, a shared upstream error in both systems, or a business definition that is conceptually wrong. Periodic review by product, engineering, analytics, and finance is needed to challenge the metric itself. Independent reconciliation is valuable because two transformations of the same faulty event stream are not independent confirmation.
Thresholds are decisions, not universal constants. An alert threshold that is too tight produces noise and normalization of alarms; one that is too loose misses meaningful defects. Base thresholds on historical variation, expected traffic, the outcome’s rarity, and the cost of a wrong rollout. Use severity levels: a broken exposure log may require an immediate pause, while a modest delay in a non-primary diagnostic may warrant an annotation and later backfill.
Common validation failures
- Validating only final totals. Test each stage and key segment, especially treatment versus control.
- Hard-coding undocumented business logic. Put definitions and owners in a visible metric contract.
- Testing the happy path only. Include retries, offline behavior, consent changes, refunds, and failures.
- Silently filtering invalid data. Measure rejected records and make arm-specific exclusions visible.
- Accepting a passing schema as accuracy. Reconcile material outcomes with an independent operational system.
- Adding checks after a surprising result. Build a reusable pre-launch checklist and preserve the evidence.
Frequently asked questions
What is the difference between validation and monitoring?
Validation defines and executes assertions; monitoring observes their results over time and alerts when a threshold is breached. A robust program needs both.
Should invalid records be deleted?
Usually quarantine or label them and retain audit information. Delete only when required by policy or privacy rules. Silent deletion makes it difficult to measure impact and reproduce a result.
How do I validate a metric with late outcomes?
Define a maturity window, report provisional completeness separately, and compare variants only after equivalent follow-up. Track the arrival curve to detect treatment-specific lag.
Can an A/A test replace data validation?
No. An A/A test can reveal aggregate assignment or metric anomalies, but it may miss a specific release, platform, schema, or rare-payment path. It complements targeted checks.
Who owns validation rules?
Engineering commonly owns collection reliability, analytics owns transformation and metric tests, and product or finance confirms decision semantics. Assign named owners for each critical rule.
Summary
Data validation makes measurement contracts executable. Test collection, transport, joins, metric construction, and reconciliation; monitor the results by experiment arm; and define responses before a decision is at risk. Passing checks are evidence, not a substitute for clear definitions and independent review.
Sources
- NIST: Data Quality Guidelines
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments
- AB-Labz: How to Write an A/B-Test Hypothesis