Quick definition: Sample ratio mismatch, usually shortened to SRM, is a statistically unlikely difference between observed and planned experiment-group counts. For a planned 50/50 split, far more assigned or exposed units in one arm than the other is a diagnostic signal that assignment, eligibility, delivery, logging, or analysis population may be broken.
What is sample ratio mismatch (srm)?
Sample ratio mismatch, usually shortened to SRM, is a statistically unlikely difference between observed and planned experiment-group counts. For a planned 50/50 split, far more assigned or exposed units in one arm than the other is a diagnostic signal that assignment, eligibility, delivery, logging, or analysis population may be broken.
In experimentation, the useful boundary is operational rather than rhetorical: define the unit, the time at which the rule is evaluated, the system that produces the record, and the exact population affected. That makes the concept reviewable before an outcome is known and prevents a plausible label from becoming an untestable explanation.
Data and implementation mechanics
The standard check compares observed counts with expected counts using a chi-square goodness-of-fit test. The expectation must reflect the configured allocation and the exact population being tested: assignment logs, exposed users, or analysis users are different denominators. A low SRM p-value signals that the discrepancy warrants investigation; it does not identify the cause or prove that the estimated treatment effect is biased.
Instrumentation should preserve raw decision and event records with a timestamp, configuration or schema version, source, and stable join key. Build a governed derived dataset from those records, then reconcile counts and distinct units at each step. Monitor volume, nulls, duplicates, late arrivals, join success, and platform distribution. A dashboard aggregate is useful, but it is not evidence that all of its inputs are correct.
Experiment implications
Sample ratio mismatch (SRM) affects whether the control and treatment estimates represent comparable populations and the intended causal question. Write the analysis contract before launch: hypothesis, target population, randomization unit, allocation, exposure definition, primary metric, guardrails, attribution window, data-maturity time, and stopping rule. This separates planned measurement from result-driven interpretation.
When reading a result, report the estimate with uncertainty rather than a winner label alone. sample ratio mismatch checks, confidence intervals, and sample-size planning help frame practical and statistical uncertainty, but neither replaces investigation of a material data-quality discrepancy. If a failure can affect arms differently, pause the decision until its direction, size, and affected population are understood.
QA scenario and workflow
A 50/50 checkout test shows 51,200 control and 48,800 treatment exposures. QA first checks the configured allocation and decision logs, then compares eligibility, assignment, exposure, and metric-population counts by platform and release version. It finds a treatment template error that prevents exposure logging for an older browser and blocks the result until the path is repaired.
- Review the written experiment and measurement contract, including the authoritative source for each field.
- Trace controlled identities from eligibility and assignment through trigger, exposure, outcome, and warehouse aggregation.
- Reconcile raw and derived counts, distinct units, timestamps, and key properties by variant, platform, and release version.
- Investigate an unexpected difference before inspecting the business outcome in detail.
- Record the fix, rerun validation on fresh data, and retain the evidence with the experiment decision.
Diagnostics and controls
Use layered checks. Start with configuration and decision logs, then compare assignment, trigger, exposure, and analysis populations. Run an SRM diagnostic against the configured ratio where it applies; compare baseline covariates chosen before launch; and inspect high-risk slices such as platform, app version, locale, consent state, and login status. Every diagnostic needs a defined population and a known owner.
Set release tolerances in advance. A threshold should prompt a specific response—alert, incident review, hold, or documented acceptance—not a debate about whether the current result is attractive. Independent reconstruction of a primary metric from raw records is especially valuable after a new SDK, flag system, identity path, or warehouse transformation is introduced.
Limitations and common errors
SRM can be induced by bot filtering, deduplication, identity joins, delayed ingestion, or an incorrect expected ratio, not only randomization. Conversely, a non-significant SRM test does not validate metrics, exposure quality, or baseline balance. Repeatedly checking many arbitrary slices can create false alarms; predefine high-risk slices and use the check with engineering traces rather than as a winner-selection rule.
- Unversioned changes: a rule, flag, schema, or query changes without an effective timestamp.
- Variant-asymmetric loss: a retry, rendering path, join, or consent branch removes records differently by arm.
- Post-hoc filtering: an analyst redefines the population after seeing outcomes.
- Aggregate-only validation: totals are accepted without tracing records through the pipeline.
Important: A statistically significant outcome cannot repair a material, unexplained implementation or measurement difference. Establish the affected population and validity before making a rollout decision.
Governance and decision use
Assign a named owner for the definition and keep a compact audit trail: experiment configuration, release identifier, schema or rule version, data-maturity timestamp, certified query, and validation results. Product, engineering, analytics, and experimentation owners can share review, but a material change should have one accountable approver. Keep only the data necessary for the decision and apply identity and consent rules consistently.
If a discrepancy is repaired with a backfill, document which records changed, whether the correction is balanced by variant, and whether a fresh observation window is required. If it cannot be repaired, quantify the impact and narrow the conclusion instead of silently extrapolating. The durable goal is a result that another reviewer can reproduce without relying on memory.
FAQ
Why does sample ratio mismatch (srm) matter in an A/B test?
It can change who is compared, what is measured, or how reliably records are joined. Any of those can alter an estimated effect independently of customer behavior.
What should QA check before launch?
Validate the written contract with controlled identities, confirm configuration and telemetry paths, and reconcile the key populations and properties before relying on production outcomes.
Can a dashboard prove that it is working?
No. A dashboard can show a plausible aggregate while a source event, identity join, filter, or variant-specific path is wrong. Trace raw evidence and independently rebuild critical counts.
Does a passing SRM check prove validity?
No. It supports the expected count distribution for the tested population, but it does not validate metric semantics, exposure quality, identity resolution, or every baseline characteristic.
When should a result be paused or rerun?
Pause when a material issue is unexplained or variant-asymmetric. Rerun when the affected population cannot be confidently corrected or when a design change breaks comparability with the original plan.
Summary
Sample ratio mismatch (SRM) is manageable when its definition, data path, and analysis use are specified before launch. Preserve the underlying evidence, validate every population boundary, investigate asymmetric discrepancies, and limit conclusions to the population that the data can support.
Sources
- Microsoft Research: Trustworthy Online Controlled Experiments
- NIST/SEMATECH e-Handbook of Statistical Methods
- Gerber and Green: Field Experiments