After your A/B test runs, the platform shows: p = 0.03. Someone on your team says "we're 97% confident the variant wins." Someone else says "there's only a 3% chance this happened by luck." Both are wrong — and this confusion ships bad product decisions every day.
The p-value is probably the most misunderstood number in data-driven product development. This guide explains what it actually means, the mistakes that come from misreading it, and how to use it correctly when running experiments.
What a p-value actually is
The p-value is the probability of observing a result at least as extreme as the one you measured, assuming the null hypothesis is true.
The null hypothesis in an A/B test is that there is no difference between control and variant — that any observed difference is purely due to random sampling variation.
📌 p = 0.03 means: "If there were truly no difference between control and variant, there would be a 3% chance of seeing a difference this large or larger just by chance."
That is all it means. Not "97% confidence the variant is better." Not "only 3% chance we're wrong." The p-value tells you about the data under a hypothetical world — not about the probability that your hypothesis is true.
What a p-value is NOT
These are the most common — and most consequential — misinterpretations:
It is NOT the probability that the null hypothesis is true
"p = 0.03 means there's a 3% chance our variant has no effect." This sounds reasonable but is logically backwards. The p-value assumes the null is true and measures how surprising your data is under that assumption. It says nothing about the probability that the null is actually true.
It is NOT a measure of effect size
A tiny p-value does not mean a large effect. With enough traffic, even a 0.01% difference in conversion rate will produce p = 0.0001. Statistical significance and practical significance are completely separate questions. Always look at the actual effect size alongside the p-value.
It is NOT your confidence level
"95% confidence" comes from the significance threshold α = 0.05, not directly from the p-value itself. Confidence level is a property of your testing procedure, not a transformation of any single p-value.
🚨 Saying "p = 0.04, so we're 96% confident" is wrong. The 95% confidence comes from setting α = 0.05 before the test, not from subtracting the p-value from 1.
How the p-value is computed
In a standard two-proportion z-test (the most common A/B test), the p-value is derived from the test statistic:
where \( \hat p_A, \hat p_B \) are observed conversion rates, \( \hat p \) is the pooled rate, and \( n_A, n_B \) are sample sizes.
$$ \text{p-value} = 2 \times P\!\left(Z > |z|\right) \quad \text{(two-tailed)} $$The resulting p-value is compared to your pre-specified significance threshold \( \alpha \) (usually 0.05). If \( p < \alpha \), you reject the null hypothesis and call the result statistically significant.
Choosing the significance threshold (α)
The threshold α is not given by nature — it's a business decision you make before the experiment begins. Common choices:
- α = 0.05 — industry default. One in twenty experiments will produce a false positive.
- α = 0.01 — for high-stakes decisions (pricing changes, core UX rewrites).
- α = 0.10 — for low-cost, easily reversible tests where speed matters more than certainty.
Lowering α reduces false positives but requires larger sample sizes. Raising it speeds up decisions but accepts more noise. There is no universally correct value — only appropriate values for your context.
⚠️ Never set α after looking at the data. Choosing α = 0.06 because your result is p = 0.055 is p-hacking. It looks like a minor adjustment but inflates your false positive rate systematically.
The peeking problem
One of the most damaging patterns in A/B testing is checking p-values repeatedly during an experiment and stopping as soon as p < 0.05. This is called peeking, and it can push your effective false positive rate from 5% to over 25%.
The reason: every check is an independent opportunity to get a false positive. If you check ten times at α = 0.05, the probability of getting at least one false positive is much higher than 5%.
The two valid approaches to early stopping are:
- Sequential testing (e.g., SPRT, mSPRT) — methods that mathematically account for continuous monitoring while maintaining error control.
- Bayesian A/B testing — provides posterior probability estimates that are valid at any sample size, without the frequentist stopping rule problem.
AB-Labz supports both: sequential testing with valid stopping boundaries, and Bayesian analysis with credible intervals.
Always pair p-values with effect size
A statistically significant result with a tiny effect size is often not worth acting on. A result with a large effect size that doesn't reach significance yet may still be informative. The p-value should always be read alongside:
- Absolute lift — the raw difference in conversion rate (e.g., +0.4 pp)
- Relative lift — the percentage improvement over control (e.g., +12%)
- Confidence interval — the range of plausible true effects at your α level
If your 95% confidence interval for the lift is [+0.1%, +0.7%] and your minimum meaningful lift is 0.5%, your result is statistically significant but practically ambiguous. That's useful information — it means you need more data, not a shipping decision.
Multiple comparisons problem
If you test 20 independent hypotheses at α = 0.05, you expect one false positive by chance alone. This is the multiple comparisons problem, and it's a serious issue for teams that run many simultaneous variants or test many metrics at once.
Common corrections include:
- Bonferroni correction — divide α by the number of tests (conservative, safe)
- Benjamini-Hochberg (FDR) — controls the false discovery rate, less conservative
AB-Labz automatically applies FDR correction when you add multiple metrics to an experiment, with a clear indicator next to each result.
A practical decision framework
When you see an experiment result, ask these questions in order:
- Did I pre-specify α, sample size, and primary metric before the experiment?
- Did the experiment run to completion without peeking or early stopping?
- Is the p-value below my pre-specified α?
- Is the effect size large enough to be business-meaningful?
- Does the confidence interval exclude effects that are too small to matter?
- Are the results consistent across key segments (no SRM, no novelty effect)?
Only if you can answer "yes" to all of the above is it safe to act on the result.
Summary
The p-value is a measure of how surprising your data is under the assumption of no effect. A small p-value means your data would be rare in a world with no true difference — not that the null hypothesis is false, not that the variant is better with high probability.
Use it as one input in your decision. Pair it with effect size, confidence intervals, and sample size adequacy. Set your threshold before the experiment. Don't peek. And if you need to monitor results continuously, use a testing method designed for it.
Run statistically rigorous experiments
AB-Labz computes p-values, effect sizes, and confidence intervals automatically — and warns you when results aren't ready to act on.