Glossary · Fundamentals
Fundamentals
The conceptual foundation: what experiments are, how variants are compared, and how a testing program creates evidence.
Glossary terms
Browse this topic
A/A Testing
Experiment QA, SRM, instrumentation, false positives, and platform validation.
A/B Test
Control and treatment, assignment, metrics, result interpretation, and test structure.
A/B Testing
Definition, design, statistics, MDE, power, pitfalls, and a pre-launch checklist.
A/B/n Testing
Multiple variants, allocation, sample size, multiple comparisons, and analysis.
Alternative Hypothesis
The predicted effect, how it differs from the null, and its role in an experiment decision.
Baseline
Starting metrics, control context, MDE planning, and sample-size assumptions.
Bucket Testing
Stable assignment, hashing, buckets, allocation, and experiment diagnostics.
Bucketing
How stable assignment buckets users, supports allocation, and prevents cross-variant exposure.
Carryover Effect
Definition, experimental context, and practical decision considerations.
Causal Effect
Definition, experimental context, and practical decision considerations.
Control Experience
Definition, experimental context, and practical decision considerations.
Control Group
Baseline, counterfactual, randomization, holdouts, and A/B examples.
Controlled Experiment
Control groups, treatment, variables, random assignment, and causal design.
Experiment
Definition, design, causal inference, lifecycle, validity threats, and an expert checklist.
Experiment Contamination
Definition, experimental context, and practical decision considerations.
Experiment Design
Definition, experimental context, and practical decision considerations.
Experiment Lifecycle
Definition, experimental context, and practical decision considerations.
Experiment Unit
Definition, experimental context, and practical decision considerations.
Experimentation
Continuous learning, methods, program maturity, culture, metrics, and governance.
Exposure
Definition, experimental context, and practical decision considerations.
Holdout Group
Definition, experimental context, and practical decision considerations.
Learning Effect
Definition, experimental context, and practical decision considerations.
Novelty Effect
Definition, experimental context, and practical decision considerations.
Null Hypothesis
Definition, experimental context, and practical decision considerations.
Online Controlled Experiment
Architecture, randomization, telemetry, ramping, and safety checks.
Practical Significance
Definition, experimental context, and practical decision considerations.
Random Assignment
Definition, experimental context, and practical decision considerations.
Randomization
Fair assignment, stable IDs, hashing, stratification, and diagnostics.
Reproducibility
Definition, experimental context, and practical decision considerations.
Rollout
Definition, experimental context, and practical decision considerations.
Split Testing
Same-URL and split-URL methods, routing, SEO, performance, and implementation.
Test Plan
Definition, experimental context, and practical decision considerations.
Test Prioritization
Definition, experimental context, and practical decision considerations.
Test Validity
Definition, experimental context, and practical decision considerations.
Test Winner
Definition, experimental context, and practical decision considerations.
Treatment Effect
Definition, experimental context, and practical decision considerations.
Treatment Group
Intervention, variant, exposure, compliance, and treatment-effect analysis.
Unit of Randomization
Definition, experimental context, and practical decision considerations.
Variant
How variants are designed, named, assigned, measured, and evaluated.
Variation
Single changes, bundles, combinations, and clearer experimental learning.
Explore the glossary
Other topics
CRO & funnels
CRO & funnels
Research-led optimization of the complete path from discovery to meaningful action.
Statistics
Statistics
Statistical terms will be published with notation, formulas, worked examples, assumptions, and interpretation warnings.
Metrics
Metrics
Metric pages will connect formulas to denominators, business outcomes, guardrails, and experiment use cases.
Design
Design
Plan tests that can answer a useful question without creating avoidable validity risks.
Data quality
Data quality
Checks that determine whether an experiment result can be trusted before it informs a decision.
Implementation
Implementation
The delivery and measurement foundations behind reliable product and web experiments.
Product analytics
Product analytics
Concepts for understanding user behavior, product value, and the outcomes experiments are meant to improve.
Privacy
Privacy
Responsible measurement concepts for experimentation, marketing, and product analytics.