A/B Testing & Statistics Glossary

Available now

A/A Testing

Experiment QA, SRM, instrumentation, false positives, and platform validation.

Available now

A/B Test

Control and treatment, assignment, metrics, result interpretation, and test structure.

Available now

A/B Testing

Definition, design, statistics, MDE, power, pitfalls, and a pre-launch checklist.

Available now

A/B/n Testing

Multiple variants, allocation, sample size, multiple comparisons, and analysis.

Available now

Alternative Hypothesis

The predicted effect, how it differs from the null, and its role in an experiment decision.

Available now

Baseline

Starting metrics, control context, MDE planning, and sample-size assumptions.

Available now

Bucket Testing

Stable assignment, hashing, buckets, allocation, and experiment diagnostics.

Available now

Bucketing

How stable assignment buckets users, supports allocation, and prevents cross-variant exposure.

Available now

Carryover Effect

Definition, experimental context, and practical decision considerations.

Available now

Causal Effect

Definition, experimental context, and practical decision considerations.

Available now

Control Experience

Definition, experimental context, and practical decision considerations.

Available now

Control Group

Baseline, counterfactual, randomization, holdouts, and A/B examples.

Available now

Controlled Experiment

Control groups, treatment, variables, random assignment, and causal design.

Available now

Experiment

Definition, design, causal inference, lifecycle, validity threats, and an expert checklist.

Available now

Experiment Contamination

Definition, experimental context, and practical decision considerations.

Available now

Experiment Design

Definition, experimental context, and practical decision considerations.

Available now

Experiment Lifecycle

Definition, experimental context, and practical decision considerations.

Available now

Experiment Unit

Definition, experimental context, and practical decision considerations.

Available now

Experimentation

Continuous learning, methods, program maturity, culture, metrics, and governance.

Available now

Exposure

Definition, experimental context, and practical decision considerations.

Available now

Holdout Group

Definition, experimental context, and practical decision considerations.

Available now

Learning Effect

Definition, experimental context, and practical decision considerations.

Available now

Novelty Effect

Definition, experimental context, and practical decision considerations.

Available now

Null Hypothesis

Definition, experimental context, and practical decision considerations.

Available now

Online Controlled Experiment

Architecture, randomization, telemetry, ramping, and safety checks.

Available now

Practical Significance

Definition, experimental context, and practical decision considerations.

Available now

Random Assignment

Definition, experimental context, and practical decision considerations.

Available now

Randomization

Fair assignment, stable IDs, hashing, stratification, and diagnostics.

Available now

Reproducibility

Definition, experimental context, and practical decision considerations.

Available now

Rollout

Definition, experimental context, and practical decision considerations.

Available now

Split Testing

Same-URL and split-URL methods, routing, SEO, performance, and implementation.

Available now

Test Plan

Definition, experimental context, and practical decision considerations.

Available now

Test Prioritization

Definition, experimental context, and practical decision considerations.

Available now

Test Validity

Definition, experimental context, and practical decision considerations.

Available now

Test Winner

Definition, experimental context, and practical decision considerations.

Available now

Treatment Effect

Definition, experimental context, and practical decision considerations.

Available now

Treatment Group

Intervention, variant, exposure, compliance, and treatment-effect analysis.

Available now

Unit of Randomization

Definition, experimental context, and practical decision considerations.

Available now

Variant

How variants are designed, named, assigned, measured, and evaluated.

Available now

Variation

Single changes, bundles, combinations, and clearer experimental learning.

Available now

Alpha (α)

The false-positive threshold, why it is chosen before launch, and its link to power.

Available now

Alpha Spending

How sequential tests allocate false-positive risk across planned interim looks.

Available now

ANOVA

Comparing group means, interpreting the F-test, and when it fits multi-variant experiments.

Available now

Average

The arithmetic mean, when it can mislead, and how to interpret average experiment metrics.

Available now

Bayesian Inference

How priors and observed data produce posterior estimates for experiment decisions.

Available now

Bayesian vs Frequentist

Different questions, outputs, priors, stopping, and practical choice.

Available now

Bayes’ Theorem

The relationship between prior beliefs, new evidence, and updated probabilities.

Available now

Bernoulli Distribution

The probability model for one binary outcome such as a conversion.

Available now

Beta (β)

Type II error, its relationship to statistical power, and planning trade-offs.

Available now

Beta Spending

Managing false-negative risk and futility boundaries in sequential designs.

Available now

Bias

Sources of systematic error that can distort an experiment estimate or decision.

Available now

Binary Metric

Defining and analyzing outcomes with two states, such as converted or not converted.

Available now

Binomial Distribution

Success counts, assumptions, and why the distribution matters for conversion metrics.

Available now

Binomial Metric

A metric based on successes and opportunities, with a clearly defined denominator.

Available now

Bonferroni Correction

A conservative way to control false positives across multiple comparisons.

Available now

Bootstrap

Resampling observed data to estimate uncertainty when analytic formulas are difficult.

Available now

Causal Inference

Definition, assumptions, interpretation, and analysis context.

Available now

Causation

Definition, assumptions, interpretation, and analysis context.

Available now

Chi-Square Test

Definition, assumptions, interpretation, and analysis context.

Available now

Confidence Bound

Definition, assumptions, interpretation, and analysis context.

Available now

Confidence Interval

Formula intuition, uncertainty, width, and experiment reporting.

Available now

Confidence Level

Definition, assumptions, interpretation, and analysis context.

Available now

Confidence Threshold

Definition, assumptions, interpretation, and analysis context.

Available now

Consistent Estimator

Definition, assumptions, interpretation, and analysis context.

Available now

Continuous Metric

Definition, assumptions, interpretation, and analysis context.

Available now

Correlation

Definition, assumptions, interpretation, and analysis context.

Available now

Credible Interval

Definition, assumptions, interpretation, and analysis context.

Available now

Descriptive Statistics

Definition, assumptions, interpretation, and analysis context.

Available now

Distribution

Definition, assumptions, interpretation, and analysis context.

Available now

Efficient Estimator

Definition, assumptions, interpretation, and analysis context.

Available now

Error-Spending Function

Definition, assumptions, interpretation, and analysis context.

Available now

Estimator

Definition, assumptions, interpretation, and analysis context.

Available now

Expected Loss

Definition, assumptions, interpretation, and analysis context.

Available now

Expected Uplift

Definition, assumptions, interpretation, and analysis context.

Available now

External Validity

Definition, assumptions, interpretation, and analysis context.

Available now

False Discovery Rate (FDR)

Definition, assumptions, interpretation, and analysis context.

Available now

False Negative

Definition, assumptions, interpretation, and analysis context.

Available now

False Negative Rate (FNR)

Definition, assumptions, interpretation, and analysis context.

Available now

False Positive

Definition, assumptions, interpretation, and analysis context.

Available now

False Positive Rate (FPR)

Definition, assumptions, interpretation, and analysis context.

Available now

Family-Wise Error Rate (FWER)

Definition, assumptions, interpretation, and analysis context.

Available now

Fisher’s Exact Test

Definition, assumptions, interpretation, and analysis context.

Available now

Frequentist Statistics

Definition, assumptions, interpretation, and analysis context.

Available now

Holm–Bonferroni Correction

Definition, assumptions, interpretation, and analysis context.

Available now

Independent Samples T-Test

Definition, assumptions, interpretation, and analysis context.

Available now

Inferential Statistics

Definition, assumptions, interpretation, and analysis context.

Available now

Internal Validity

Definition, assumptions, interpretation, and analysis context.

Available now

Linear Regression

Definition, assumptions, interpretation, and analysis context.

Available now

Logistic Regression

Definition, assumptions, interpretation, and analysis context.

Available now

Mann–Whitney U Test

Definition, assumptions, interpretation, and analysis context.

Available now

Mean

Definition, assumptions, interpretation, and analysis context.

Available now

Median

Definition, assumptions, interpretation, and analysis context.

Available now

Minimum Detectable Effect

Business value, sample-size trade-offs, and realistic MDE setting.

Available now

Mode

Definition, assumptions, interpretation, and analysis context.

Available now

Multiple Comparisons

Definition, assumptions, interpretation, and analysis context.

Available now

Normal Distribution

Definition, assumptions, interpretation, and analysis context.

Available now

P-value

Correct interpretation, null hypothesis, common misreadings, and examples.

Available now

Paired T-Test

Definition, assumptions, interpretation, and analysis context.

Available now

Posterior Distribution

Definition, assumptions, interpretation, and analysis context.

Available now

Power Analysis

Definition, assumptions, interpretation, and analysis context.

Available now

Prior Distribution

Definition, assumptions, interpretation, and analysis context.

Available now

Probability of Being Best

Definition, assumptions, interpretation, and analysis context.

Available now

Probability of Superiority

Definition, assumptions, interpretation, and analysis context.

Available now

Ratio Metric

Definition, assumptions, interpretation, and analysis context.

Available now

Regression Analysis

Definition, assumptions, interpretation, and analysis context.

Available now

Regression to the Mean

Definition, assumptions, interpretation, and analysis context.

Available now

Resampling

Definition, assumptions, interpretation, and analysis context.

Available now

Sample Size Calculator

Definition, assumptions, interpretation, and analysis context.

Available now

Selection Bias

Definition, assumptions, interpretation, and analysis context.

Available now

Significance Level

Definition, assumptions, interpretation, and analysis context.

Available now

Standard Deviation

Definition, assumptions, interpretation, and analysis context.

Available now

Standard Error

Definition, assumptions, interpretation, and analysis context.

Available now

Standardization

Definition, assumptions, interpretation, and analysis context.

Available now

Statistical Power

Power, beta, sample size, MDE, and underpowered decisions.

Available now

Statistical Precision

Definition, assumptions, interpretation, and analysis context.

Available now

Statistical Significance

Evidence, practical significance, confidence, and decision boundaries.

Available now

Statistical Test

Definition, assumptions, interpretation, and analysis context.

Available now

Survivorship Bias

Definition, assumptions, interpretation, and analysis context.

Available now

T-Test

Definition, assumptions, interpretation, and analysis context.

Available now

Two-Proportion Z-Test

Definition, assumptions, interpretation, and analysis context.

Available now

Type I Error

Definition, assumptions, interpretation, and analysis context.

Available now

Type II Error

Definition, assumptions, interpretation, and analysis context.

Available now

Variance

Definition, assumptions, interpretation, and analysis context.

Available now

Z-Test

Definition, assumptions, interpretation, and analysis context.

Available now

Adaptive Experimentation

Changing allocation or design as evidence arrives while preserving valid inference.

Available now

Adaptive Sequential Design

Planned interim analyses and adaptations that maintain statistical guarantees.

Available now

Always-Valid Inference

Methods designed for continuous monitoring without invalid optional stopping.

Available now

Before-and-After Analysis

What change-over-time comparisons can show and why they rarely establish causality alone.

Available now

Blocking

Grouping similar units before randomization to improve precision and balance.

Available now

Cluster Randomized Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Cluster-Level Randomization

Design choices, assumptions, and practical guidance for experiments.

Available now

Conversion Lift

Design choices, assumptions, and practical guidance for experiments.

Available now

CUPED

Design choices, assumptions, and practical guidance for experiments.

Available now

Difference-in-Differences

Design choices, assumptions, and practical guidance for experiments.

Available now

Experiment Duration

Business cycles, seasonality, novelty, and stopping rules.

Available now

Factorial Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Fixed-Horizon Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Fractional Factorial Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Full Factorial Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Geo Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Geo Lift Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Group Sequential Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Heterogeneous Treatment Effect

Design choices, assumptions, and practical guidance for experiments.

Available now

Hypothesis

How to write falsifiable, mechanism-based product hypotheses.

Available now

Incrementality Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Interaction Effect

Design choices, assumptions, and practical guidance for experiments.

Available now

Interrupted Time Series

Design choices, assumptions, and practical guidance for experiments.

Available now

Lift Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Marketing Holdout

Design choices, assumptions, and practical guidance for experiments.

Available now

Multiple Testing

Bonferroni, Holm, FDR, variants, metrics, and segments.

Available now

Multivariate A/B Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Multivariate Testing

Design choices, assumptions, and practical guidance for experiments.

Available now

Natural Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Optional Stopping

Design choices, assumptions, and practical guidance for experiments.

Available now

Peeking

Design choices, assumptions, and practical guidance for experiments.

Available now

Pre-Experiment Covariate

Design choices, assumptions, and practical guidance for experiments.

Available now

Quasi-Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Sample Size

Baseline, MDE, power, alpha, allocation, and duration.

Available now

Segment-Level Analysis

Design choices, assumptions, and practical guidance for experiments.

Available now

Sequential Analysis

Design choices, assumptions, and practical guidance for experiments.

Available now

Sequential Testing

Design choices, assumptions, and practical guidance for experiments.

Available now

Stratification

Design choices, assumptions, and practical guidance for experiments.

Available now

Subgroup Analysis

Design choices, assumptions, and practical guidance for experiments.

Available now

Switchback Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Synthetic Control

Design choices, assumptions, and practical guidance for experiments.

Available now

Treatment Effect Heterogeneity

Design choices, assumptions, and practical guidance for experiments.

Available now

Variance Reduction

Design choices, assumptions, and practical guidance for experiments.

Available now

Anti-Flicker Script

Reducing visual flashes in client-side tests while managing page-performance costs.

Available now

API Experimentation

Testing API behavior with appropriate assignment, exposure, and latency safeguards.

Available now

Application Programming Interface (API)

The delivery contract behind systems that can be measured and experimented on.

Available now

Asynchronous Loading

Non-blocking delivery, timing risks, and exposure gaps in web experiments.

Available now

Audience Eligibility

Pre-treatment rules that define who belongs in an experiment denominator.

Available now

Audience Targeting

Selecting an experiment population and documenting the scope of valid conclusions.

Available now

Canary Release

Technical context, delivery considerations, and experiment risks.

Available now

Client-Side Rendering (CSR)

Technical context, delivery considerations, and experiment risks.

Available now

Client-Side Testing

Rendering, flicker, scripts, performance, and use cases.

Available now

Custom Audience

Technical context, delivery considerations, and experiment risks.

Available now

Data Layer

Technical context, delivery considerations, and experiment risks.

Available now

Experiment Trigger

Technical context, delivery considerations, and experiment risks.

Available now

Experimentation Platform

Technical context, delivery considerations, and experiment risks.

Available now

Feature Flag

Delivery control, rollout, experiment support, and technical debt.

Available now

Feature Toggle

Technical context, delivery considerations, and experiment risks.

Available now

Flicker Effect

Technical context, delivery considerations, and experiment risks.

Available now

Latency

Technical context, delivery considerations, and experiment risks.

Available now

Mobile App Experimentation

Technical context, delivery considerations, and experiment risks.

Available now

Monitoring

Technical context, delivery considerations, and experiment risks.

Available now

Observability

Technical context, delivery considerations, and experiment risks.

Available now

Page Load Time

Technical context, delivery considerations, and experiment risks.

Available now

Progressive Rollout

Technical context, delivery considerations, and experiment risks.

Available now

Release Management

Technical context, delivery considerations, and experiment risks.

Available now

Remote Configuration

Technical context, delivery considerations, and experiment risks.

Available now

Rollback

Technical context, delivery considerations, and experiment risks.

Available now

Server-Side Rendering (SSR)

Technical context, delivery considerations, and experiment risks.

Available now

Server-Side Testing

Architecture, consistency, latency, and backend experiments.

Available now

Single-Page Application (SPA)

Technical context, delivery considerations, and experiment risks.

Available now

Software Development Kit (SDK)

Technical context, delivery considerations, and experiment risks.

Available now

Synchronous Loading

Technical context, delivery considerations, and experiment risks.

Available now

Tag Management System

Technical context, delivery considerations, and experiment risks.

Available now

Time to Interactive

Technical context, delivery considerations, and experiment risks.

Available now

Web Experimentation

Technical context, delivery considerations, and experiment risks.

Available now

Anonymization

Removing or transforming identifiers, re-identification limits, and experiment-data use.

Available now

Attribution

Source, touchpoints, causal experiments, and marketing measurement.

Available now

Attribution Model

Rules for assigning credit across touchpoints and their limits for causal decisions.

Available now

Consent Management

Definition, governance context, and responsible measurement considerations.

Available now

Consent Management Platform (CMP)

Definition, governance context, and responsible measurement considerations.

Available now

Consent Mode

Definition, governance context, and responsible measurement considerations.

Available now

Customer Data Platform (CDP)

Definition, governance context, and responsible measurement considerations.

Available now

Data Governance

Definition, governance context, and responsible measurement considerations.

Available now

Data Minimization

Definition, governance context, and responsible measurement considerations.

Available now

Data Residency

Definition, governance context, and responsible measurement considerations.

Available now

Data Sovereignty

Definition, governance context, and responsible measurement considerations.

Available now

First-Click Attribution

Definition, governance context, and responsible measurement considerations.

Available now

First-Party Data

Consent-aware measurement, identity, and durable analytics.

Available now

GDPR

Experimentation, analytics, consent, minimization, and lawful measurement.

Available now

Identity Graph

Definition, governance context, and responsible measurement considerations.

Available now

Incremental Attribution

Definition, governance context, and responsible measurement considerations.

Available now

K-Anonymity

Definition, governance context, and responsible measurement considerations.

Available now

Last-Click Attribution

Definition, governance context, and responsible measurement considerations.

Available now

Marketing Attribution

Definition, governance context, and responsible measurement considerations.

Available now

Media Mix Modeling

Definition, governance context, and responsible measurement considerations.

Available now

Multi-Touch Attribution

Definition, governance context, and responsible measurement considerations.

Available now

Personal Data

Definition, governance context, and responsible measurement considerations.

Available now

Personally Identifiable Information (PII)

Definition, governance context, and responsible measurement considerations.

Available now

Privacy by Design

Definition, governance context, and responsible measurement considerations.

Available now

Pseudonymization

Definition, governance context, and responsible measurement considerations.

Available now

Source / Medium

Definition, governance context, and responsible measurement considerations.

Available now

Third-Party Data

Definition, governance context, and responsible measurement considerations.

Available now

UTM Parameters

Definition, governance context, and responsible measurement considerations.

Available now

Zero-Party Data

Definition, governance context, and responsible measurement considerations.