What Is SRM in A/B Testing — And What to Do When It Fires

You planned a 50/50 split. After two weeks, the analysis table shows 62% of users in variant B and 38% in control. The conversion lift looks fantastic. Before you celebrate, check SRM — Sample Ratio Mismatch. That skewed split is a red flag that your experiment data may not represent a fair random assignment.

SRM is one of the highest-signal quality checks in A/B testing. When it fires, your p-value and confidence interval are built on a broken foundation. This guide explains what SRM is, how the test works, what usually causes it, and the playbook for responding without throwing away good experiments unnecessarily.

What Sample Ratio Mismatch means

SRM occurs when the observed allocation of users across experiment arms differs significantly from the intended split. You configured 50/50 but received something else — not because randomness always produces exact balance (it does not), but because the deviation is larger than sampling noise alone can explain.

Think of it as a sanity check on your randomization pipeline. If assignment is working correctly and traffic is stable, the observed ratio should be close to the target. A large, persistent gap suggests something interfered: a bucketing bug, filtered traffic, bot skew, or data pipeline drops.

The chi-square test — intuition, not just formula

The standard SRM check is a chi-square goodness-of-fit test. You compare observed counts in each arm to the counts you would expect under the planned ratio.

Example: 10,000 users assigned, 50/50 planned. You observe 4,800 control and 5,200 variant — a 4% absolute skew. With enough traffic, that gap can be statistically significant even though it looks "close enough" to a PM. The test quantifies whether "close enough" is believable under random assignment.

Do not eyeball ratios on small samples. Do run SRM after you have meaningful volume — typically thousands of assignments per arm for a 50/50 test, though thresholds depend on how strict you want to be.

Common causes of SRM

SRM almost always traces to infrastructure or filtering — not to the variants themselves. Frequent culprits:

Assignment and bucketing bugs

• Hash function changes mid-experiment, re-bucketing users inconsistently.
• Sticky assignment broken — users see multiple variants across sessions.
• Off-by-one in experiment key or layer configuration, sending extra traffic to one arm.

Traffic filtering and redirects

• One variant crashes or loads slower on certain devices — those users drop from the denominator.
• Redirect rules send mobile users disproportionately to one experience.
• Bot or fraud filters remove users unevenly across arms.

Data pipeline issues

• Event loss on one variant's tracking implementation.
• Delayed events counted in one arm but not another during partial exports.
• Join bugs between assignment logs and outcome events.

How serious is it?

Severity depends on magnitude, timing, and whether the skew correlates with user types:

• Mild, early drift that stabilizes — sometimes startup transients or cache warm-up. Still investigate; do not ignore.
• Persistent 55/45 or worse on large volume — strong evidence of a systemic issue. Do not ship on primary metrics.
• SRM plus segment skew — e.g., all missing iOS users are from control — suggests the bug is platform-specific and may completely bias lift estimates.

Even small ratio errors can inflate false positives or hide real effects if the missing users are not random. The whole point of randomization is that arms are comparable; SRM breaks that contract.

What to do when SRM fires

Follow a disciplined response — panic-shipping is the worst outcome:

• Stop the launch review. Treat the experiment as invalid for decision-making until resolved.
• Verify assignment logs vs analysis counts. Compare raw bucketing tables to the metrics warehouse. The bug is often between them.
• Check timing. Did SRM appear after a deploy, flag change, or marketing spike? Correlate with incident timelines.
• Segment diagnostics. Break counts by platform, country, new vs returning — find where the ratio breaks.
• Fix and restart if needed. Once assignment is repaired, old data is usually not salvageable for causal inference. A clean rerun beats patching biased samples.
• Document. Log the cause so the same bucketing bug does not invalidate the next three tests.

AB-Labz Workbench runs an SRM check automatically in the Analysis Engine when you upload or connect experiment data — so a skewed split surfaces before you dive into lift and p-values.

Summary

Sample Ratio Mismatch means your observed traffic split does not match what you configured — beyond what random chance would produce. The chi-square test formalizes that check. Causes are usually bucketing bugs, uneven drops, or pipeline errors — not the variant creative. When SRM fires, pause shipping, debug assignment, and prefer a clean restart over trusting a biased result. No amount of statistical sophistication rescues broken randomization.