Data quality·Glossary term

Outlier

Outlier A/B testing Reference guide

Outlier is a concept used in data quality & diagnostics.

Quick definition: An outlier is an observation that is unusually large, small, rare, or otherwise distant from the typical pattern under a stated metric and population. It may be an error, a real but influential outcome, or evidence that the metric combines different behaviors.

What is an outlier?

An outlier is not defined by surprise alone. It is an observation judged unusual relative to a distribution, context, and rule. A $100,000 order might be a data-entry error in a consumer storefront, a normal enterprise contract in a B2B product, or a genuine but influential revenue outcome. The same value can be ordinary in one country, customer tier, or currency and extreme in another.

Outliers differ from invalid records. An invalid record violates a known contract, such as a negative duration where negatives are impossible or an event from a test account. An outlier may be perfectly valid. It also differs from a heavy-tailed distribution: revenue, session duration, and usage counts often have many legitimate high values. The goal is to understand the generating process before deciding whether and how an observation should influence a metric.

Data mechanics and detection

Start with raw evidence. Preserve record ID, producer, timestamp, unit, currency or scale, status, schema version, and transformations. Check for parsing errors, unit conversion mistakes, duplicated transactions, retry loops, clock faults, test traffic, bot activity, and broken joins. A sudden cluster of extreme values after a release often points to an implementation issue; a stable long tail may be normal customer behavior.

Detection rules should match the metric. Percentiles, interquartile-range rules, robust z-scores based on median absolute deviation, business limits, and log-scale visualizations can each be useful. None is universally correct. A normal-distribution z-score can be misleading for skewed revenue; a fixed percentile may remove a strategically important enterprise segment. Evaluate rules by platform, geography, customer type, and time only when those distinctions have a defined purpose, not to search for a convenient answer.

Document every transformation. A dashboard that silently clips values, takes logs, or excludes large accounts makes historical results difficult to reproduce. Store original and derived values, exclusion reason, and rule version. Monitor the number and contribution of extreme observations, because a stable mean can still be dominated by a few units. Robust summaries such as medians or trimmed means can complement the primary metric, but they answer a different business question.

Experiment impact

Outliers can dominate mean revenue, time, and count metrics in A/B tests. One unusually large transaction in treatment may create an attractive lift and a wide confidence interval; one runaway client retry loop can make a feature appear to increase engagement. Removing values after seeing which arm benefits creates bias. Define eligibility and treatment rules before launch, or perform symmetric sensitivity analyses that show conclusions with and without a predeclared robust method.

Choose the estimand first. If the business decision concerns total expected revenue per assigned user, legitimate high-value orders belong in the primary estimate, even if they increase variance. If the decision concerns typical customer experience, a median or capped operational metric may be more useful. Often the right response is to report both: a mean-value primary metric with uncertainty, plus distributional diagnostics and guardrails. For skewed outcomes, see log-transforming revenue metrics before assuming a transformation preserves the same decision meaning.

Outlier checks are also data-quality checks. Compare extreme-value rate, source status, duplicate rate, and identity join behavior by variant. A treatment-only tail can be a real product effect, but it should be traced to underlying records. Retain the audit trail for any exclusion, cap, winsorization, or transformation and apply the rule consistently across arms and periods.

Practical QA scenario

A SaaS company tests a self-serve upgrade page using revenue per visitor. Treatment shows a 40% lift driven by two accounts that each appear to purchase hundreds of seats. QA traces the records to an API retry defect that created repeated client purchase events, while the billing system contains one settled invoice per account. The values are extreme because of duplication, not because the redesigned page won enterprise business.

The team uses settled invoice amount as the authoritative numerator, deduplicates by invoice ID, and keeps client events for funnel debugging. It runs a controlled retry test, compares transaction counts and value distributions across variants, and publishes a sensitivity report showing the original client metric, corrected mean revenue, median revenue, and number of high-value accounts. The launch decision is delayed until the event path is repaired and validated on fresh A/A traffic.

  1. Inspect extreme records at the source before applying a statistical rule.
  2. Classify them as invalid, duplicated, legitimate, or uncertain with evidence.
  3. Predefine the primary estimand and any robust sensitivity analyses.
  4. Apply documented rules symmetrically to every arm.
  5. Report how much extreme observations contribute to the estimate and uncertainty.

Limitations and failure modes

Outlier detection cannot determine business relevance automatically. Removing valid large customers may make a metric more stable while hiding the economic effect of a treatment. Keeping invalid values can be equally misleading. Distributional methods require sufficient context and should not turn routine product diversity into a defect. Robust metrics can reduce sensitivity but may obscure a change in tail risk that deserves a guardrail.

  • Blind trimming: values are removed because they are inconvenient.
  • Distribution assumptions: a normal-rule threshold is used for heavy-tailed data.
  • Variant-specific cleanup: different rules are applied after results are visible.
  • Duplicate masking: a cap hides retry or join defects.
  • Mean-only reporting: influential observations are not disclosed.

Important: An extreme observation is not permission to delete it. Establish whether it is valid, what decision population it represents, and how any rule changes the estimand.

FAQ

Are outliers always errors?

No. They can be legitimate rare outcomes, especially in revenue and usage data. Investigate provenance before treating them as bad data.

Should outliers be removed before an A/B test?

Only under a documented, symmetric rule justified by the metric and available before results. Preserve raw data and report sensitivity when influential.

Is the median better than the mean?

Neither is universally better. The mean describes average total value; the median describes a typical observation. Choose according to the decision.

Can transformations solve outlier problems?

They can stabilize analysis, but change interpretation. A log-transformed estimate is not automatically the same business quantity as raw average revenue.

Summary

An outlier is an unusual observation whose meaning depends on context. Trace its source, distinguish valid tails from defects, predefine symmetric treatment rules, and report the influence of extreme values on experiment decisions.

Distribution review should be routine for metrics with money, duration, count, or latency values. Track quantiles, tail contribution, maximum values, and record provenance alongside the mean. This provides early warning of both genuine customer shifts and technical defects. When the outcome has high business stakes, retain a small record-level review sample for the largest contributions, subject to appropriate access controls. The goal is not surveillance of individual customers; it is enough evidence to verify that the aggregate metric remains connected to real, correctly represented events.

Sources