Quick definition: The bootstrap estimates uncertainty by repeatedly resampling the observed experimental units with replacement and recalculating a statistic. It is especially useful when a metric’s sampling distribution is awkward to derive analytically.
What is the bootstrap?
The bootstrap is a computational inference method. Given an observed sample of n independent units, it creates many new samples of size n by drawing those units with replacement. Some original units appear more than once in a resample and some do not appear. The chosen statistic—mean difference, median, ratio, percentile, or regression coefficient—is calculated for every resample. The resulting empirical distribution approximates the statistic’s sampling distribution under repeated samples from the population.
It answers a narrower question than “simulate the product”: conditional on the collected data being representative and the resampling unit being correct, how variable would this estimator be? That distinction matters. Bootstrap precision cannot repair selection bias, faulty assignment, missing events, or a metric whose business meaning is unclear.
How bootstrap inference works
For a two-arm test, first construct one row per randomization unit and calculate the planned outcome for each user. Draw users with replacement separately within control and treatment, preserving each arm’s observed sample size. Compute the estimand in bootstrap replicate b, such as Δb = mean(YT,b) − mean(YC,b). Repeat this process B times, commonly 2,000 to 10,000 depending on the stability needed. The standard deviation of the Δ values is a bootstrap standard error; their quantiles can form an interval.
Percentile interval: [q0.025(Δ*), q0.975(Δ*)], where Δ* is the bootstrap distribution. A basic interval reflects these quantiles around the observed estimate, and bias-corrected and accelerated (BCa) intervals also adjust for estimated bias and skewness.
The percentile interval is intuitive but is not universally best. For smooth estimators with large samples, a normal approximation or bootstrap-t interval may be more appropriate. BCa can improve coverage for some skewed statistics but requires an additional jackknife calculation and may be unstable for small or highly discrete samples. Choose and document the interval method before examining results.
Assumptions and valid resampling units
The central assumption is that observed units approximate the relevant population and are independent at the level being resampled. In a user-randomized A/B test, resample users—not page views, orders, or events—after aggregating each user’s outcome. Resampling events gives frequent users extra influence and generally understates uncertainty. If stores, schools, or households were randomized, resample clusters. For a switchback or time-randomized experiment, ordinary independent user bootstrap is usually invalid because outcomes are serially correlated; use a block, cluster, or design-specific method.
Resample within treatment arms when estimating an arm difference. Pooling both arms is useful for a permutation test under a sharp no-effect null, but it is not the ordinary nonparametric bootstrap for uncertainty around an observed difference. Stratify resampling when the analysis is stratified and retain the planned weights. A bootstrap also inherits issues from small samples: with only a few clusters, there are too few distinct resamples to reliably approximate a distribution.
Important: Do not bootstrap a post-treatment subset such as purchasers when the treatment can affect becoming a purchaser. For revenue per randomized user, include every eligible randomized user, with zero revenue where appropriate.
Bootstrap in A/B testing
Bootstrap methods are valuable for product metrics that are non-normal, nonlinear, or difficult to model: median task time, 95th-percentile latency, trimmed mean revenue, ratio metrics, and differences in conversion funnels. A conventional analytic standard error is often faster and more transparent for a simple conversion or mean, so bootstrap is not automatically superior. Its advantage is flexibility when the estimator itself is the business metric.
Plan the estimand and resampling algorithm alongside eligibility, exposure, and maturity rules. Long-tailed revenue, for example, does not require excluding large legitimate purchasers; bootstrap their user-level outcomes and report distribution diagnostics, perhaps alongside a robust estimand chosen beforehand. Use log-transform revenue metrics only when the transformed estimand answers the decision question, not simply to make a result look significant.
Worked example: a median completion-time change
A team randomizes 12,000 exposed users per arm to a new checkout flow. Among users who reach the same pre-treatment checkout trigger, the predeclared outcome is elapsed seconds to complete; non-completers are handled by a separately specified rule. Control’s median is 142 seconds and treatment’s is 136 seconds, so the observed median difference is −6 seconds.
The analyst draws 10,000 bootstrap samples of 12,000 users within each arm and calculates treatment minus control median each time. The 2.5th and 97.5th percentiles are −10.8 and −1.4 seconds. The team reports an estimated six-second reduction with a 95% bootstrap percentile interval of −10.8 to −1.4 seconds. It does not say there is a 95% probability the true effect lies there; under repeated samples and this procedure, intervals of that construction have approximate coverage.
The decision still needs conversion and support-contact guardrails. If the interval excludes zero but the lower plausible speed improvement is only 1.4 seconds, the product owner should compare that with the predefined minimum useful improvement. Statistical compatibility with a nonzero effect is not a shipping criterion by itself.
Interpretation and reporting
Report the unit, eligibility population, outcome construction, number of resamples, resampling scheme, interval type, point estimate, interval, and seed or reproducible implementation. For a ratio metric, state whether the ratio was computed per user then averaged or as a ratio of aggregate totals; these are different estimands. Show arm-level values and practical units, not only a bootstrap p-value.
Risks and limitations
- Bad data are repeated, not corrected: bootstrap replicates preserve tracking bias and contamination.
- Rare events: sparse conversions or rare clusters can make quantiles unstable.
- Dependence: user resampling fails under interference, repeated measures, or clustered assignment unless adapted.
- Computation: complex pipelines can be expensive; verify convergence by increasing B.
- Selection: choosing a metric or resampling variant after seeing the result invalidates a confirmatory interpretation.
Common mistakes
- Resampling event rows rather than the randomized unit.
- Calling a bootstrap interval “distribution-free” without checking independence and representativeness.
- Using a bootstrap p-value as a substitute for a prespecified hypothesis and multiplicity plan.
- Choosing BCa, percentile, or transformations only after inspecting which one crosses zero.
- Ignoring sample ratio mismatch; investigate SRM before interpreting any interval.
Frequently asked questions about the bootstrap
How many bootstrap resamples are enough?
Use enough that interval endpoints are stable; 5,000 or 10,000 is common for a final interval. More may be needed for extreme percentiles or noisy small samples.
Can bootstrap prove causality?
No. Random assignment and sound measurement support causal interpretation. Bootstrap describes sampling uncertainty conditional on the observed design and data.
Should I bootstrap conversion rate?
You can, but a binomial proportion method is usually simpler and well understood. Bootstrap helps more when conversion is embedded in a complex estimator.
Is bootstrap the same as a permutation test?
No. Bootstrap samples with replacement to estimate sampling variability. A permutation test reallocates labels under a no-effect hypothesis to calibrate a null distribution.
Can I use it while monitoring a live test?
Not with ordinary fixed-horizon interpretation. Repeated looks require a sequentially valid design such as one discussed in A/B test duration.
Summary
The bootstrap makes uncertainty estimation practical for complicated A/B-test metrics by resampling the correct experimental unit. It is powerful when the estimand, independence structure, and reporting method are specified in advance, but it cannot compensate for invalid assignment, immature outcomes, or data-driven analysis choices.
Before relying on a bootstrap result, reproduce it with a fixed random seed and inspect whether doubling the number of resamples materially moves the reported interval endpoints. Compare it with a suitable analytic estimate where one exists; a large disagreement is a diagnostic to investigate, not evidence that the more favorable method is preferable. Preserve the raw user-level outcome definition and the exact inclusion query so another analyst can rerun the calculation.
Sources
- Efron and Tibshirani, An Introduction to the Bootstrap
- NIST/SEMATECH e-Handbook: Bootstrap
- Lahiri, Resampling Methods for Dependent Data