Always-Valid Inference

What is always-valid inference?

Always-valid inference is designed for a fact of operational experimentation: someone will look before the planned end. A conventional fixed-horizon p-value of 0.05 has its familiar Type I error interpretation when the analysis is performed at the planned information size under its assumptions. If a team recomputes it every hour and stops at the first favorable value, the chance of eventually seeing a small p-value is higher than 5% even when the treatment has no effect.

An always-valid test controls false-positive risk over the whole permitted monitoring process. Its p-value is often called an anytime-valid p-value: under the null, the probability that it is ever at or below α is at most α. A confidence sequence is the interval analogue. Instead of one interval intended to cover a parameter at one fixed time, it gives a sequence of intervals that covers the true parameter simultaneously over time with a stated probability. Thus a decision can be made when a boundary is crossed without invalidating the stated guarantee merely because the timing depended on the observed data.

“Always valid” does not mean always useful, always precise, or immune to bad experimentation. It addresses optional stopping and repeated monitoring for a defined statistical procedure. It does not fix nonrandom assignment, an incorrect denominator, users exposed to the wrong experience, interference, metric switching, or a causal question that the data do not identify.

The formal idea: validity over time

Let p_t be a p-value computed after information time t. It is anytime-valid for a null hypothesis if, under that null,

P_H0(there exists t such that p_t ≤ α) ≤ α.

This is stronger than saying P(p_T ≤ α) ≤ α for one fixed, preselected T. The time can be a stopping time: a data-dependent moment such as “the first hour the dashboard crosses 0.05.” In practice, a test may be built from a nonnegative martingale or e-process whose expected value is controlled under the null. Evidence grows as data accumulate; a rule rejects when the e-value becomes large enough, commonly at least 1/α after accounting for the method’s precise construction.

A (1 − α) confidence sequence C_t has a related simultaneous-coverage statement:

P(θ belongs to C_t for every permitted t) ≥ 1 − α.

If a confidence sequence for a treatment effect excludes zero, an associated sequential test has crossed its threshold. The interval will usually be wider early than a conventional fixed-time confidence interval because it pays for the freedom to look repeatedly. That conservatism is not a defect; it is the cost of a valid anytime interpretation.

Requirements before using it

First specify the estimand: for example, the difference in seven-day conversion probability between treatment and control among eligible users during the experiment. Then define the analysis population, assignment unit, eligibility event, exposure event, outcome window, missing-data handling, and the monitoring cadence. The exact procedure must match the data-generating structure. A method built for independent Bernoulli outcomes does not automatically apply to repeated user-level events, clustered assignment, heavy-tailed revenue, or an analysis that changes covariates as it runs.

Randomization must be trustworthy. Every eligible unit should have a known assignment probability, and treatment delivery must be recorded. Inspect sample ratio mismatch, exposure logging, identity changes, and event completeness before acting on a boundary. Continuous monitoring can quickly detect harm, but it can also quickly act on a tracking outage if data quality is not gated.

Outcome maturity matters as much as time validity. A confidence sequence for one-day conversion cannot justify a claim about 30-day retention. For delayed outcomes, define when a unit enters the analysis and retain a stable window. If the product needs an emergency safety monitor and a long-term efficacy metric, use separate, clearly labelled procedures instead of silently substituting the quick proxy.

Design choices and their trade-offs

Always-valid inference is not one button. Choose one- or two-sided alternatives, the error level, the effect scale, an optional minimum effect for practical relevance, and whether the goal is monitoring, early stopping, estimation, or all three. A two-sided procedure protects discoveries in either direction but generally spends more evidence than a one-sided procedure; select the direction based on the actual decision before launch, not on the observed sign.

Multiplicity remains. Monitoring ten metrics with a valid procedure for each does not create a valid familywise decision rule for “ship if any metric wins.” Likewise, an always-valid primary endpoint does not cover repeatedly chosen segments, variants, or populations. Define a hierarchy, adjust across the relevant family, or state that secondary findings are exploratory. The same principle applies when an adaptive platform changes allocation or drops arms: the sequential procedure must accommodate that design, not merely the number of looks.

For routine two-arm product work, a bounded schedule of looks using a sequential analysis design can be simpler to validate and communicate. Always-valid methods are particularly useful where monitoring is genuinely continuous, decision timing cannot be scheduled, or safety signals demand prompt action. They should not be chosen only to avoid committing to a decision framework.

Practical scenario: monitoring a checkout error safeguard

An ecommerce team releases a new fraud-screening rule to 10% of eligible checkout attempts. Its primary safety metric is payment failure within the same session; the business outcome is completed purchase within seven days. Before launch, the team defines an anytime-valid one-sided test for an increase in payment failures, a materially harmful absolute threshold, and an automated rollback owner. Assignment and exposure are logged per attempt, and the safety monitor uses only attempts whose session outcome has closed.

At any point, the safety procedure may cross its boundary. That crossing starts a rollback, but it does not grant a broad efficacy conclusion: seven-day purchases remain immature. Conversely, an early apparent purchase gain does not allow the team to ship broadly because the long-term confidence sequence has not yet excluded an unacceptable loss and the guardrail has not completed its monitoring period. At the end, the readout separates the rapid safety decision from the seven-day causal estimate and states which procedure supported each statement.

This scenario also shows why continuous evidence is not continuous product change. A valid monitor may allow the experiment to stop at a data-dependent time; it does not authorize the team to rewrite the fraud rule after every fluctuation and continue treating the combined history as one unchanged experiment.

How to interpret outputs

An anytime-valid p-value is not the probability that the null hypothesis is true. Under the null, it is calibrated so that repeatedly stopping on small values has controlled false-positive frequency. A confidence sequence is not a credible interval unless it was constructed from a Bayesian posterior and interpreted under that model. Its frequentist statement is about simultaneous coverage over the monitoring process.

Read the interval together with practical thresholds. If a sequence excludes zero but still includes effects too small to pay for implementation, the statistical decision may be “evidence of a nonzero effect” while the product decision is “do not prioritize.” If it excludes the minimum acceptable loss, the evidence may support a safety claim even if it does not establish a positive benefit. Report absolute metrics, the effect scale, observation maturity, active users or information, all planned outcomes, and the time or information fraction at which a decision occurred.

Limitations and risks

• Precision cost: confidence sequences can be wider than fixed-time intervals, particularly early and under highly frequent monitoring.
• Model mismatch: dependence, heavy tails, delayed feedback, or changing assignment can invalidate a procedure designed for simpler data.
• Multiple decisions: valid monitoring of one hypothesis does not solve multiple metrics, variants, segments, or repeated product iterations.
• Operational overreaction: a valid statistical signal may still be too small, immature, or unreplicated for an irreversible rollout.
• False reassurance: a non-crossing interval is not evidence that a variant is safe or identical to control.

Decision workflow

1. Separate safety, efficacy, and learning questions; give each its own metric and decision threshold.
2. Confirm that the assignment, exposure, outcome, and unit of analysis satisfy the proposed method’s assumptions.
3. Select a documented anytime-valid test or confidence sequence and define one- or two-sided directionality, multiplicity scope, and monitoring cadence.
4. Simulate or back-test the procedure with plausible traffic, variance, delay, and null scenarios; verify its operating characteristics.
5. Version the protocol and software, validate data-quality gates, and assign authority for stop, rollback, and rollout decisions.
6. When a boundary is crossed, lock the data, verify measurement and guardrails, execute the stated action, and preserve all evidence.
7. Report the full monitoring rule and decision path, then continue post-rollout monitoring if the risk warrants it.

Common mistakes

• Assuming that every Bayesian dashboard, sequential test, or repeated p-value is always-valid.
• Using a method for independent conversions on correlated sessions or clustered users without justification.
• Monitoring a fast proxy while presenting its result as evidence about a delayed primary outcome.
• Changing the metric, segment, or treatment implementation after observing results.
• Ignoring multiplicity across a large dashboard of “valid” metrics.
• Equating a statistically valid stop with an automatic product launch.

Frequently asked questions

Can we really stop whenever we want?

For the specific hypothesis and procedure, yes: an anytime-valid procedure is built to allow data-dependent stopping. You still need a decision protocol, data-quality verification, mature outcomes, and a plan for guardrails. The permission does not extend to unrelated analysis changes.

Is always-valid inference Bayesian?

No. It is commonly formulated with frequentist time-uniform error control, though Bayesian sequential decisions can also be coherent under a stated model and prior. Ask what exact guarantee the tool provides rather than inferring it from the interface.

Does it eliminate the need for a sample-size plan?

No. It changes stopping validity, not business planning. A maximum duration or information target is still needed for resourcing, power, exposure, and a decision if the boundary never crosses.

Can we use it for revenue?

Potentially, but revenue is often skewed, zero-inflated, correlated within users, and affected by censoring. Use a method validated for the chosen estimand and distribution; do not assume a binary-metric implementation transfers automatically.

What is the relationship to optional stopping?

Optional stopping is the practice of choosing when to stop based on accumulated data. Always-valid inference supplies procedures intended to retain their stated error behavior under that practice, within their defined scope.

Summary

Always-valid inference makes repeated monitoring statistically defensible for a specified hypothesis by controlling error over time. It is useful for continuous safety monitoring and unscheduled decisions, but it is not a loophole around experimental discipline. Valid assignment, complete measurement, outcome maturity, multiplicity control, a suitable method, and clear product thresholds remain essential.

Sources

• Howard et al., “Time-uniform, nonparametric, nonasymptotic confidence sequences”
• Ramdas et al., “Game-Theoretic Statistics and Safe Anytime-Valid Inference”
• Bartlett, Sequential Analysis lecture notes

Related terms

• Multivariate Testing
• Multivariate A/B Test
• Factorial Design
• Full Factorial Design
• Fractional Factorial Design