Quick definition: Optional stopping is the practice of choosing when to stop data collection after seeing accumulating results. In a conventional fixed-horizon A/B test, it can inflate false-positive rates unless the stopping rule and analysis are designed to remain valid over time.
What is optional stopping?
Optional stopping occurs when an experimenter may end a test based on interim evidence rather than at the sample size or date specified before launch. A team might check conversion daily and stop when a p-value falls below 0.05, keep collecting when a result is disappointing, or extend the test after a near-significant result. These choices use the same data both to decide when to look and to decide what conclusion to report.
With an ordinary fixed-horizon test, a 5% significance threshold assumes one planned analysis. Repeating it at many looks creates extra opportunities for random noise to cross the threshold. Under no true effect, the chance of eventually observing a “significant” result can be far greater than 5%. This is why uncontrolled monitoring is a validity problem rather than harmless curiosity.
Optional stopping is not intrinsically invalid. It can be handled by a group sequential design, an alpha-spending procedure, a confidence sequence, or a clearly stated Bayesian decision framework. What matters is that the method’s guarantee matches the allowed stopping behavior.
Why optional stopping changes inference
Imagine flipping a fair coin and checking after every ten flips whether heads is unusually common. If the rule is “stop and celebrate at the first p-value below 0.05,” occasional favorable runs become selected. The final p-value no longer has its usual fixed-horizon interpretation because the stopping time depended on the observed contrast. The same selection can exaggerate the estimated effect: a large estimate is often what triggered the stop.
Continuing selectively is equally problematic. Stopping for a favorable result but extending an unfavorable one changes the reporting process even if the final analysis uses all accumulated users. A test with a stated target but an unrecorded extension rule has a different operating characteristic from the planned test.
| Monitoring behavior | Fixed-horizon validity | Safer approach |
|---|---|---|
| One analysis at prespecified maximum. | Usually valid if other assumptions hold. | Lock the data at the planned horizon. |
| Scheduled interim decisions. | Not valid with repeated ordinary thresholds. | Use sequential boundaries or alpha spending. |
| Continuous dashboard monitoring. | Not valid for ordinary p-values. | Use always-valid inference or restrict decisions. |
| Emergency rollback for harm. | Separate from efficacy claim. | Document safety rule and analyze transparently. |
Valid methods for planned stopping
A group sequential design schedules a limited number of information-based looks and sets boundaries that control the total Type I error. Alpha-spending designs can accommodate a flexible timing of analyses by allocating the error budget as information accumulates. Confidence sequences provide intervals designed to retain stated coverage at arbitrary stopping times, subject to their assumptions and implementation. These methods require decisions about the primary metric, effect direction, maximum information or observation process, and outcome maturity before results drive behavior.
Bayesian monitoring can report posterior probabilities after each update, but it is not automatically immune to poor decision practice. Priors, likelihood, loss function, stopping and rollout criteria, and reporting must be specified. A posterior probability answers a different question from a frequentist p-value; do not borrow an interpretation from one framework while using the other operationally.
Some stopping is administrative rather than evidence-driven: a launch deadline, traffic loss, legal requirement, or instrumentation failure. It may not inflate error in the same way if independent of outcomes, but it can reduce power or change the population. Record it, report the achieved sample and interval, and avoid claiming the original planned precision.
Optional stopping in A/B tests
Product teams often need dashboards for safety and operations. The practical solution is not to hide data; it is to separate monitoring from confirmatory decisions. Define a primary endpoint and a valid efficacy procedure. Define guardrails and emergency thresholds that can trigger rollback. Restrict success declarations to scheduled reviews or an always-valid rule. Make the dataset mature before evaluating delayed outcomes such as refund-adjusted revenue or retention.
Traffic changes introduce another reason to avoid arbitrary stops. Ending after a campaign spike may select an unusual visitor mix. Keep concurrent control, document changes in eligibility and allocation, and consider whether the result transports to normal traffic. If the product must ship before the planned test completes, treat the evidence as an interim decision with its actual method and uncertainty rather than retrofitting a final claim.
Worked scenario: a subscription offer
A subscription service tests annual-plan framing with a 28-day net-revenue outcome. After two weeks, the daily dashboard shows a favorable conversion p-value. The owner wants to stop, but only half the enrolled users have reached the revenue window and the test was planned for 40,000 mature users. Stopping now would mix immature outcomes with uncontrolled peeking.
The team continues to the next prespecified group sequential look, after 20,000 mature users. The sequential efficacy boundary is not crossed and refunds are elevated. At the final planned look, the revenue interval includes the minimum useful gain and a modest loss. The team concludes that the variant has not met the stated ship criterion. It does not present the early dashboard result as evidence because that result was not a valid mature primary analysis.
Practical workflow
- Decide before launch whether analysis is fixed-horizon, group sequential, always-valid, or Bayesian.
- Specify the primary outcome, maturation rule, maximum information, review cadence, alpha or decision budget, and guardrails.
- Test data locks and dashboards so that interim data cannot silently change after review.
- At a look, use only the allowed analysis and record the exact decision, data cutoff, and deviation.
- Separate emergency safety action from the primary efficacy claim.
- Report the full monitoring and stopping history with the final estimate and uncertainty.
Interpreting a stopped test
A valid early success decision means the procedure crossed its prespecified threshold, not that the observed lift is a precise forecast. Early-stopped effects can be selected upward. A non-crossing interim look means continue, unless a valid futility or harm rule says otherwise. A test stopped for operational reasons may yield useful descriptive evidence but often has less precision than planned.
“We checked but did not decide” is acceptable only when checks truly did not influence stopping, metric selection, or product action. If interim results shaped decisions, disclose that fact and use a method whose interpretation fits the actual process.
Limitations and common mistakes
- Using p < 0.05 repeatedly: repeated fixed-horizon thresholds do not control the advertised error rate.
- Counting assigned users as mature outcomes: delayed metrics require completed windows.
- Stopping only winners: selective continuation biases both significance and effect size.
- Adding looks silently: unrecorded reviews change the design.
- Confusing safety with efficacy: a harm rollback does not create a success result.
- Overclaiming early estimates: threshold crossing does not guarantee stable long-run impact.
Frequently asked questions
Can we look at data without invalidating the test?
Viewing data is not magic, but using it to change a decision creates optional stopping. Keep exploratory monitoring distinct from the confirmatory procedure.
Can we extend a fixed-horizon test?
Not without considering the extra look. Use a prespecified extension rule or a valid sequential update; otherwise describe the result as exploratory.
Does a larger sample fix optional stopping?
No. More data may improve precision, but data-dependent repeated thresholds still alter false-positive behavior.
Are confidence sequences always best?
No. They support continuous validity but can be wider and require careful implementation. A few planned sequential looks are often simpler for product decisions.
Summary
Optional stopping is choosing when to stop after observing evidence. It invalidates ordinary fixed-horizon inference when repeated looks drive decisions, but planned sequential, always-valid, or coherent Bayesian methods can support legitimate monitoring. Reliable A/B testing makes the stopping rule, mature outcome, guardrails, and complete decision path visible.
Sources
- Pocock, “Group Sequential Methods in the Design and Analysis of Clinical Trials”
- Howard et al., “Time-Uniform, Nonparametric, Nonasymptotic Confidence Sequences”
- Armitage, McPherson and Rowe, “Repeated Significance Tests on Accumulating Data”