How Long Should You Run an A/B Test? Avoiding the Peeking Problem

An experiment goes live on Monday. By Wednesday, the variant is up 9% and statistically significant. You ship it. Two months later, the metric is exactly where it was before. What happened?

You stopped the test too early. This is the single most common execution error in A/B testing, and it happens in teams of every size. The fix isn't complicated — but it requires deciding how long to run your experiment before you look at the data.

The right way to determine test duration

Test duration is not an arbitrary choice. It follows directly from your sample size calculation:

If you haven't calculated the required sample size, you haven't determined the required duration. The two are inseparable. Start with your baseline conversion rate, MDE, α, and power — compute sample size — then divide by your daily traffic.

Two minimums that override the formula

Even if the math says you'll hit your sample size in three days, two minimums apply:

1. At least one full week

User behaviour varies significantly by day of the week. Monday traffic has different intent from Saturday traffic. A test that runs only on weekdays will measure a different population than one that includes weekends. Always run experiments for a minimum of seven consecutive days to capture at least one full weekly cycle.

For B2B products where weekends are essentially zero traffic, you may need to measure in full business weeks instead.

2. At least one full business cycle

If your product has monthly billing cycles, payroll-based purchase patterns, or quarterly review periods, a one-week experiment may not capture representative behaviour. Map your known purchase and engagement cycles before setting duration.

The novelty effect: why longer isn't always better

Running an experiment indefinitely introduces a different problem: novelty effects decay. When users first see a new UI, their engagement is elevated simply because it's different. Over weeks, behaviour reverts toward their baseline. An experiment that ran for two months may show a smaller (or reversed) effect than the same experiment at two weeks — not because the variant is worse, but because the novelty wore off.

The practical implication: run experiments long enough to capture your target sample size and complete at least one business cycle — then stop. Don't run "a little longer to be sure" after you've hit your pre-specified criteria.

Why you cannot stop when the result "looks good"

The peeking problem is statistical, not intuitive. Here's why it matters:

Suppose you run an A/A test (no real difference between groups) and check the p-value every day for 30 days. The p-value fluctuates around 0.5 on average — but it will dip below 0.05 temporarily on several of those days just by random chance. If your rule is "stop when p < 0.05," you will almost certainly stop at a false positive in a sufficiently long experiment.

Simulations show that peeking daily over 30 days at α = 0.05 produces an effective false positive rate of roughly 25–30% — five times higher than intended.

When you genuinely need to stop early

Some situations legitimately require early stopping: a bug in the variant is harming users, a metric has moved dramatically in the wrong direction, or a business deadline requires a decision. In these cases, use methods designed for early stopping:

Sequential testing (SPRT / mSPRT)

Sequential Probability Ratio Tests define a stopping boundary before the experiment starts. As data accumulates, the test statistic is checked against this boundary. If it crosses, you can stop early with the same false positive guarantee as a fixed-horizon test. AB-Labz computes this boundary continuously and shows you exactly when (if ever) it's valid to stop.

Bayesian A/B testing

Bayesian methods produce a posterior probability distribution over possible effects. This is valid at any sample size — you can inspect it at any time without inflating error rates, because there's no fixed hypothesis test. The tradeoff is that interpretation is different: instead of a p-value, you get a probability that the variant beats the control, and a credible interval for the true effect.

Special case: low-traffic products

For products with fewer than a few thousand monthly active users, reaching standard sample sizes can take six months or more. Practical strategies:

• Test bigger changes — a 30% redesign produces a larger MDE than a button colour tweak, requiring less data.
• Increase your MDE — decide what improvement is minimum-meaningful and test only for that.
• Raise α — for low-stakes, easily reversible decisions, α = 0.10 is defensible.
• Use CUPED — variance reduction using pre-experiment data can cut required sample size by 30–50%.
• Accept uncertainty — for very small products, qualitative user research may provide better signal than underpowered experiments.

Pre-experiment duration checklist

• Calculate required sample size from baseline rate, MDE, α = 0.05, power = 80%.
• Divide by your daily unique visitors to get the minimum days needed.
• Apply the 7-day minimum (or full business cycle if longer).
• Set a calendar reminder for the end date. Do not check results before then.
• If you need to monitor for harm, set up an alert on a guardrail metric — not on your primary outcome metric.
• Document the planned end date in the experiment record before launch.

Summary

Test duration is determined by your sample size calculation and your traffic volume — not by gut feel, sprint length, or "until it looks significant." The minimum is always one full week; the actual minimum is however long it takes to collect the pre-specified sample.

Early stopping (peeking) is not a shortcut — it's a validity problem that produces decisions you can't trust. If you need to stop early legitimately, use sequential testing or Bayesian methods. Otherwise, commit to the duration before you start, and don't touch the experiment until it's done.