Quick definition: An independent-samples t-test compares the expected values of a continuous outcome in two independent groups. In an A/B test, it estimates whether the mean user-level outcome differs between control and treatment, while accounting for sampling variation.
What is an independent-samples t-test?
An independent-samples t-test, also called a two-sample or unpaired t-test, evaluates a null hypothesis that two population means are equal. It is appropriate when each observation belongs to one group only: for example, a randomized user sees either the existing checkout or a new checkout, not both. The observed difference between sample means is compared with its estimated standard error. A large difference relative to ordinary sampling noise produces a large absolute t statistic and a small p-value.
The method answers an estimand question, not merely a software question. For a product experiment, the outcome could be revenue per randomized user, minutes spent in a product, or number of support contacts per account. Each randomized unit should make one outcome contribution, or the analysis should explicitly model clustering or repeated observations. A t-test on raw events when assignment is at user level treats correlated events as independent and can materially overstate precision.
Test statistic, variants, and interpretation
Let group means be x̄1 and x̄2, sample variances s1² and s2², and sizes n1 and n2. The estimated difference is d = x̄2 − x̄1. Welch’s t statistic is t = d / sqrt(s1²/n1 + s2²/n2). Its degrees of freedom use the Welch–Satterthwaite approximation, which need not be an integer. A two-sided p-value measures how unusual a statistic at least this extreme would be if the equal-means null and the model assumptions held.
Welch’s test does not assume equal variances and is usually the better default. The pooled-variance t-test estimates one common variance and has degrees of freedom n1 + n2 − 2; it can be slightly more efficient when equal variances are genuinely credible, but unequal group spreads combined with unequal group sizes make it unreliable. “Student’s t-test” can refer to either version, so reports should name the variance assumption.
A confidence interval is often more useful than a binary significance label. A 95% interval for the difference gives a range of effects compatible with the data and model at that confidence level. If the interval excludes zero, the corresponding two-sided 5% test rejects; it still does not establish that the effect is large enough to deploy. State whether the difference is treatment minus control, give it in operational units, and show the baseline for a relative interpretation.
Assumptions and diagnostic boundaries
The central design assumption is independence between analysis units. Randomly assigning users does not create independent event rows, orders from the same account, or visits inside a store cluster. For cluster randomization, aggregate to the cluster or use a model with cluster-aware uncertainty. For before-and-after observations from the same user, use a paired design or a repeated-measures model; an independent test discards pairing information and can be wrong.
The classic exact model assumes independent observations with normally distributed outcomes within groups. The test is frequently robust to modest non-normality when groups are large and reasonably balanced because means have approximately normal sampling distributions. It is less reliable for tiny samples with severe skew, influential outliers, or unequal variances. Inspect distributions, group sizes, and plausible data errors instead of applying a normality test mechanically. A transformation changes the question being estimated; it is not a neutral way to make an inconvenient p-value smaller.
The outcome should have a meaningful mean. A binary conversion indicator can be tested with a difference in proportions or modeled with logistic regression; at large samples a t-test on 0/1 outcomes approximates a proportion test, but a binomial analysis is usually clearer. For strongly skewed revenue, the mean may still be the correct business estimand, but robust standard errors, bootstrap methods, or preplanned winsorization may deserve consideration. No statistical method fixes a missing exposure event or inconsistent eligibility rule.
Independent t-tests in A/B testing
The two-sample t-test is useful for a conventional A/B experiment with one continuous, user-level primary metric. It compares average treatment outcome with average control outcome under the assumption that random assignment balances unobserved factors in expectation. Predeclare the primary metric, attribution window, exclusions, analysis population, tail-treatment rule, alpha, and stopping rule. The guide to writing an A/B-test hypothesis explains why this decision structure should precede data collection.
For average revenue per eligible user, include all eligible assigned users and give nonbuyers a zero. Restricting the analysis to purchasers answers average order value instead and can create post-treatment selection bias if the treatment changes who purchases. For time metrics, define censoring and failure behavior. For a product with shared accounts, decide whether account or user is the randomization and analysis unit. These choices determine whether the reported mean is actionable.
A t-test does not handle multiplicity automatically. If a team tries five variants, ten outcomes, and twenty segments, unadjusted 0.05 tests make false discoveries likely. Keep a protected primary outcome, use guardrails appropriately, and define adjusted confirmatory comparison families. See multiple comparisons in A/B testing for the risk created by a long unplanned result table.
Worked example: revenue per visitor
A subscription site randomizes 50,000 eligible visitors to control and 50,000 to a revised pricing page. Seven-day net revenue per assigned visitor averages $4.80 in control and $5.06 in treatment. Sample standard deviations are $18.20 and $19.10. The estimated difference is $0.26 per visitor. Welch’s standard error is approximately sqrt(18.20²/50,000 + 19.10²/50,000) = $0.118, so t ≈ 2.20 and the two-sided p-value is about 0.028.
The team’s 95% confidence interval is roughly $0.03 to $0.49 per visitor. Under its predeclared 5% two-sided test, it rejects the equal-mean null. It should not report “revenue increased by 5.4%” without qualification: 5.4% is the point estimate relative to the $4.80 baseline, and the interval shows the plausible absolute range. At traffic volume of one million similar visitors, even the low end may matter; the team also reviews refunds, chargebacks, conversion, page latency, and revenue-event reconciliation.
Suppose instead the interval were −$0.09 to $0.61. The p-value would not support rejection, but the experiment would not prove there is no worthwhile increase. The interval includes both a modest loss and a potentially valuable gain. The next action depends on the predeclared minimum detectable effect, cost of additional traffic, and whether an improved design or a longer fixed horizon can create enough precision.
Practical analysis workflow
- Define the population, randomization unit, analysis unit, metric formula, observation window, and practical decision threshold.
- Validate allocation, exposure, eligibility, and missingness before calculating outcomes; investigate unexpected imbalance such as sample ratio mismatch.
- Aggregate outcomes to the appropriate independent unit and inspect counts, means, spread, zeros, tails, and impossible values.
- Use Welch’s test unless an equal-variance rationale was specified, then report the signed estimate, interval, t statistic, degrees of freedom, and p-value.
- Apply the planned multiplicity and sequential-monitoring rules, then combine statistical evidence with guardrails and implementation evidence.
Risks and common mistakes
- Using event rows as independent users. Repeated behavior from one assigned unit requires aggregation or clustered methods.
- Choosing pooled variance by default. Welch’s test is safer when variances or group sizes may differ.
- Confusing a p-value with the probability the treatment is ineffective. It is calculated under a null model; it is not a posterior probability.
- Ignoring tails or broken data. Investigate influential observations and report a predeclared handling rule rather than deleting results selectively.
- Analyzing only converters. Conditioning on an outcome affected by treatment changes the estimand and can bias a comparison.
- Peeking until significant. Repeated looks inflate false-positive risk unless a valid sequential design is used.
Frequently asked questions about independent-samples t-tests
When should I use Welch’s t-test?
Use it as the usual default for two independent groups because it does not require equal variances. Its small loss of efficiency under exact equal variances is generally less important than its robustness.
Can I use a t-test for conversion rate?
A proportion test or logistic regression is usually more natural for binary conversion. With large samples, a t-test on user-level 0/1 indicators reaches a similar conclusion, but it should not hide the binary metric definition.
Does a non-significant t-test prove the variants are equal?
No. It means the data did not reject the specified null at the chosen threshold. Use the confidence interval and an equivalence or non-inferiority design if the objective is to demonstrate a practically small difference.
What is the difference between paired and independent t-tests?
A paired test analyzes within-unit differences from matched or repeated measurements. An independent test assumes observations in one group have no one-to-one pairing with those in the other.
How should I report the result?
Report group sizes and means, the signed difference in business units, its confidence interval, the test variant, t statistic, degrees of freedom, p-value, and the predeclared decision context.
Summary
An independent-samples t-test compares two group means using their difference relative to estimated sampling noise. In A/B tests it is reliable only when the metric, independent analysis unit, data quality, variance treatment, and stopping and multiplicity rules match the experimental design. Its estimate and confidence interval should guide the decision more directly than a significance label alone.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, Two-Sample t-Test.
- Welch, B. L. (1947), The Generalization of Student’s Problem When Several Different Population Variances Are Involved.
- Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments.