A/B Testing & Statistics Glossary

Available now

A/A Testing

Experiment QA, SRM, instrumentation, false positives, and platform validation.

Available now

A/B Test

Control and treatment, assignment, metrics, result interpretation, and test structure.

Available now

A/B Testing

Definition, design, statistics, MDE, power, pitfalls, and a pre-launch checklist.

Available now

A/B/n Testing

Multiple variants, allocation, sample size, multiple comparisons, and analysis.

Available now

Alternative Hypothesis

The predicted effect, how it differs from the null, and its role in an experiment decision.

Available now

Baseline

Starting metrics, control context, MDE planning, and sample-size assumptions.

Available now

Bucket Testing

Stable assignment, hashing, buckets, allocation, and experiment diagnostics.

Available now

Bucketing

How stable assignment buckets users, supports allocation, and prevents cross-variant exposure.

Available now

Carryover Effect

Definition, experimental context, and practical decision considerations.

Available now

Causal Effect

Definition, experimental context, and practical decision considerations.

Available now

Control Experience

Definition, experimental context, and practical decision considerations.

Available now

Control Group

Baseline, counterfactual, randomization, holdouts, and A/B examples.

Available now

Controlled Experiment

Control groups, treatment, variables, random assignment, and causal design.

Available now

Experiment

Definition, design, causal inference, lifecycle, validity threats, and an expert checklist.

Available now

Experiment Contamination

Definition, experimental context, and practical decision considerations.

Available now

Experiment Design

Definition, experimental context, and practical decision considerations.

Available now

Experiment Lifecycle

Definition, experimental context, and practical decision considerations.

Available now

Experiment Unit

Definition, experimental context, and practical decision considerations.

Available now

Experimentation

Continuous learning, methods, program maturity, culture, metrics, and governance.

Available now

Exposure

Definition, experimental context, and practical decision considerations.

Available now

Holdout Group

Definition, experimental context, and practical decision considerations.

Available now

Online Controlled Experiment

Architecture, randomization, telemetry, ramping, and safety checks.

Available now

Randomization

Fair assignment, stable IDs, hashing, stratification, and diagnostics.

Available now

Split Testing

Same-URL and split-URL methods, routing, SEO, performance, and implementation.

Available now

Treatment Group

Intervention, variant, exposure, compliance, and treatment-effect analysis.

Available now

Variant

How variants are designed, named, assigned, measured, and evaluated.

Available now

Variation

Single changes, bundles, combinations, and clearer experimental learning.

Available now

Alpha (α)

The false-positive threshold, why it is chosen before launch, and its link to power.

Available now

Alpha Spending

How sequential tests allocate false-positive risk across planned interim looks.

Available now

ANOVA

Comparing group means, interpreting the F-test, and when it fits multi-variant experiments.

Available now

Average

The arithmetic mean, when it can mislead, and how to interpret average experiment metrics.

Available now

Bayesian Inference

How priors and observed data produce posterior estimates for experiment decisions.

Available now

Bayesian vs Frequentist

Different questions, outputs, priors, stopping, and practical choice.

Available now

Bayes’ Theorem

The relationship between prior beliefs, new evidence, and updated probabilities.

Available now

Bernoulli Distribution

The probability model for one binary outcome such as a conversion.

Available now

Beta (β)

Type II error, its relationship to statistical power, and planning trade-offs.

Available now

Beta Spending

Managing false-negative risk and futility boundaries in sequential designs.

Available now

Bias

Sources of systematic error that can distort an experiment estimate or decision.

Available now

Binary Metric

Defining and analyzing outcomes with two states, such as converted or not converted.

Available now

Binomial Distribution

Success counts, assumptions, and why the distribution matters for conversion metrics.

Available now

Binomial Metric

A metric based on successes and opportunities, with a clearly defined denominator.

Available now

Bonferroni Correction

A conservative way to control false positives across multiple comparisons.

Available now

Bootstrap

Resampling observed data to estimate uncertainty when analytic formulas are difficult.

Available now

Causal Inference

Definition, assumptions, interpretation, and analysis context.

Available now

Causation

Definition, assumptions, interpretation, and analysis context.

Available now

Chi-Square Test

Definition, assumptions, interpretation, and analysis context.

Available now

Confidence Bound

Definition, assumptions, interpretation, and analysis context.

Available now

Confidence Interval

Formula intuition, uncertainty, width, and experiment reporting.

Available now

Confidence Level

Definition, assumptions, interpretation, and analysis context.

Available now

Confidence Threshold

Definition, assumptions, interpretation, and analysis context.

Available now

Consistent Estimator

Definition, assumptions, interpretation, and analysis context.

Available now

Continuous Metric

Definition, assumptions, interpretation, and analysis context.

Available now

Correlation

Definition, assumptions, interpretation, and analysis context.

Available now

Credible Interval

Definition, assumptions, interpretation, and analysis context.

Available now

Descriptive Statistics

Definition, assumptions, interpretation, and analysis context.

Available now

Distribution

Definition, assumptions, interpretation, and analysis context.

Available now

Efficient Estimator

Definition, assumptions, interpretation, and analysis context.

Available now

Error-Spending Function

Definition, assumptions, interpretation, and analysis context.

Available now

Estimator

Definition, assumptions, interpretation, and analysis context.

Available now

Expected Loss

Definition, assumptions, interpretation, and analysis context.

Available now

Expected Uplift

Definition, assumptions, interpretation, and analysis context.

Available now

External Validity

Definition, assumptions, interpretation, and analysis context.

Available now

False Discovery Rate (FDR)

Definition, assumptions, interpretation, and analysis context.

Available now

False Negative

Definition, assumptions, interpretation, and analysis context.

Available now

False Negative Rate (FNR)

Definition, assumptions, interpretation, and analysis context.

Available now

False Positive

Definition, assumptions, interpretation, and analysis context.

Available now

False Positive Rate (FPR)

Definition, assumptions, interpretation, and analysis context.

Available now

Family-Wise Error Rate (FWER)

Definition, assumptions, interpretation, and analysis context.

Available now

Fisher’s Exact Test

Definition, assumptions, interpretation, and analysis context.

Available now

Frequentist Statistics

Definition, assumptions, interpretation, and analysis context.

Available now

Holm–Bonferroni Correction

Definition, assumptions, interpretation, and analysis context.

Available now

Minimum Detectable Effect

Business value, sample-size trade-offs, and realistic MDE setting.

Available now

P-value

Correct interpretation, null hypothesis, common misreadings, and examples.

Available now

Statistical Power

Power, beta, sample size, MDE, and underpowered decisions.

Available now

Statistical Significance

Evidence, practical significance, confidence, and decision boundaries.

Available now

Adaptive Experimentation

Changing allocation or design as evidence arrives while preserving valid inference.

Available now

Adaptive Sequential Design

Planned interim analyses and adaptations that maintain statistical guarantees.

Available now

Always-Valid Inference

Methods designed for continuous monitoring without invalid optional stopping.

Available now

Before-and-After Analysis

What change-over-time comparisons can show and why they rarely establish causality alone.

Available now

Blocking

Grouping similar units before randomization to improve precision and balance.

Available now

Cluster Randomized Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Cluster-Level Randomization

Design choices, assumptions, and practical guidance for experiments.

Available now

Conversion Lift

Design choices, assumptions, and practical guidance for experiments.

Available now

CUPED

Design choices, assumptions, and practical guidance for experiments.

Available now

Difference-in-Differences

Design choices, assumptions, and practical guidance for experiments.

Available now

Experiment Duration

Business cycles, seasonality, novelty, and stopping rules.

Available now

Factorial Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Fixed-Horizon Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Fractional Factorial Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Full Factorial Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Geo Experiment

Design choices, assumptions, and practical guidance for experiments.

Available now

Geo Lift Test

Design choices, assumptions, and practical guidance for experiments.

Available now

Group Sequential Design

Design choices, assumptions, and practical guidance for experiments.

Available now

Heterogeneous Treatment Effect

Design choices, assumptions, and practical guidance for experiments.

Available now

Hypothesis

How to write falsifiable, mechanism-based product hypotheses.

Available now

Multiple Testing

Bonferroni, Holm, FDR, variants, metrics, and segments.

Available now

Sample Size

Baseline, MDE, power, alpha, allocation, and duration.

Available now

Anti-Flicker Script

Reducing visual flashes in client-side tests while managing page-performance costs.

Available now

API Experimentation

Testing API behavior with appropriate assignment, exposure, and latency safeguards.

Available now

Application Programming Interface (API)

The delivery contract behind systems that can be measured and experimented on.

Available now

Asynchronous Loading

Non-blocking delivery, timing risks, and exposure gaps in web experiments.

Available now

Audience Eligibility

Pre-treatment rules that define who belongs in an experiment denominator.

Available now

Audience Targeting

Selecting an experiment population and documenting the scope of valid conclusions.

Available now

Canary Release

Technical context, delivery considerations, and experiment risks.

Available now

Client-Side Rendering (CSR)

Technical context, delivery considerations, and experiment risks.

Available now

Client-Side Testing

Rendering, flicker, scripts, performance, and use cases.

Available now

Custom Audience

Technical context, delivery considerations, and experiment risks.

Available now

Data Layer

Technical context, delivery considerations, and experiment risks.

Available now

Experiment Trigger

Technical context, delivery considerations, and experiment risks.

Available now

Experimentation Platform

Technical context, delivery considerations, and experiment risks.

Available now

Feature Flag

Delivery control, rollout, experiment support, and technical debt.

Available now

Feature Toggle

Technical context, delivery considerations, and experiment risks.

Available now

Flicker Effect

Technical context, delivery considerations, and experiment risks.

Available now

Server-Side Testing

Architecture, consistency, latency, and backend experiments.

Available now

Anonymization

Removing or transforming identifiers, re-identification limits, and experiment-data use.

Available now

Attribution

Source, touchpoints, causal experiments, and marketing measurement.

Available now

Attribution Model

Rules for assigning credit across touchpoints and their limits for causal decisions.

Available now

Consent Management

Definition, governance context, and responsible measurement considerations.

Available now

Consent Management Platform (CMP)

Definition, governance context, and responsible measurement considerations.

Available now

Consent Mode

Definition, governance context, and responsible measurement considerations.

Available now

Customer Data Platform (CDP)

Definition, governance context, and responsible measurement considerations.

Available now

Data Governance

Definition, governance context, and responsible measurement considerations.

Available now

Data Minimization

Definition, governance context, and responsible measurement considerations.

Available now

Data Residency

Definition, governance context, and responsible measurement considerations.

Available now

Data Sovereignty

Definition, governance context, and responsible measurement considerations.

Available now

First-Click Attribution

Definition, governance context, and responsible measurement considerations.

Available now

First-Party Data

Consent-aware measurement, identity, and durable analytics.

Available now

GDPR

Experimentation, analytics, consent, minimization, and lawful measurement.