Chi-Square Test

What is a chi-square test?

The chi-square test is a family of tests for categorical data. The most common product-experiment form is the Pearson chi-square test of independence. It asks whether membership in one category, such as control or treatment, is independent of membership in another category, such as converted or did not convert. If assignment is valid and the null hypothesis of no effect is true, the conversion mix should be similar across variants apart from ordinary random variation.

For a two-arm experiment, the data can be arranged in a two-by-two table. Rows are variants and columns are outcome categories. The null hypothesis is that the probability of conversion is the same in both arms. The alternative is that the probabilities differ. A statistically significant result is evidence against equal rates under the test assumptions; it is not evidence that the treatment is valuable, correctly implemented, or safe to roll out.

Chi-square methods are useful because they analyze counts directly and extend naturally to several categories or variants. They are not a substitute for a clear estimand. Before choosing a test, define the eligible population, the randomization unit, the outcome window, and whether the metric is conversion per randomized user, conversion per exposed user, or another quantity. Those definitions determine which rows belong in the table.

How the test works

Let O_ij be the observed count in row i and column j. Under independence, the expected count is E_ij = (row total_i × column total_j) / N, where N is the table total. Pearson’s statistic is χ² = Σ (O_ij − E_ij)² / E_ij. It grows when observed counts are farther from their null expectations.

For an r-by-c table, the usual reference distribution has (r − 1)(c − 1) degrees of freedom. The p-value is the probability, assuming the null model and its assumptions, of a statistic at least as large as the observed value. Compare it with an alpha level chosen before looking at the result. A small p-value does not give the probability that the null is true, and a large p-value does not establish that the rates are identical.

For two arms and a binary outcome, the chi-square test is closely related to the two-proportion z-test: without continuity correction, their statistics satisfy χ² = z² and yield the same two-sided p-value. The z-test often makes the signed direction and rate difference easier to communicate. Chi-square remains convenient for a table with several variants or outcomes.

Assumptions and validity conditions

Each analyzed unit should contribute to the table once, or the analysis must account for repeated observations. A person who makes five sessions is not five independent randomization units if assignment occurred by user. Aggregate events to the assigned user when estimating user conversion. If stores, classrooms, households, or regions are randomized, rows within a cluster tend to be correlated. A simple user-row chi-square test then exaggerates the effective sample size; use cluster-level or cluster-robust analysis instead.

Expected counts must be adequate for the chi-square approximation. A common practical rule is that all expected cells should be at least five, although the right threshold depends on the design and software. Rare outcomes, tiny variants, or fine-grained categories may require an exact test, Monte Carlo procedure, category combination justified before analysis, or a regression model. Do not merge categories after seeing a favorable result merely to obtain significance.

Random assignment, consistent eligibility, and comparable measurement are causal prerequisites rather than mathematical details. Check allocation and exposure before interpreting an outcome table. A sample ratio mismatch, blocked tracking event, or treatment-dependent denominator can produce a striking chi-square statistic without a genuine product effect. A test also assumes that the outcome window has matured similarly for every arm.

Using chi-square tests in A/B testing

Use a chi-square test when the primary outcome is categorical and observations are independent at the analysis unit. Typical examples include signup completion, purchase within seven days, error occurrence, consent acceptance, and selected plan type. For a three-variant landing-page experiment, an omnibus three-by-two table tests whether any conversion rate differs. It does not name a winner. Follow-up comparisons should be planned and adjusted for multiplicity.

Analyze every randomized eligible user in the denominator when the estimand is intention-to-treat conversion. Excluding users who were slow to load a page, never saw a downstream component, or made no purchase can change the causal question and introduce bias. Exposure-based analyses may be useful diagnostics, but label them as such and retain the prespecified primary population.

For a large A/B test, accompany the p-value with the control rate, treatment rate, absolute difference in percentage points, relative change if useful, and a confidence interval for the difference or ratio. Product decisions need magnitude and uncertainty. The distinction is explained in absolute difference, while confidence intervals in A/B testing shows why an interval is more informative than a pass/fail label.

Worked scenario: checkout conversion

A retailer randomizes 10,000 eligible visitors per arm to the existing checkout or a simplified checkout. Control records 500 purchases and 9,500 non-purchases; treatment records 560 purchases and 9,440 non-purchases. The observed rates are 5.00% and 5.60%, an absolute lift of 0.60 percentage points and a relative lift of 12%.

Under equal rates, the expected purchase count in each arm is (10,000 × 1,060) / 20,000 = 530. The four contributions to the statistic are based on deviations of plus or minus 30 from their expected counts. The resulting statistic is approximately χ² = 3.49 with one degree of freedom, corresponding to a two-sided p-value near 0.062. At alpha 0.05, the team does not reject equal conversion rates.

The conclusion is not “the redesign failed” or “there is no lift.” The observed effect may still be commercially attractive, but the planned data leave material uncertainty. The team should inspect the interval, outcome maturity, revenue and error guardrails, and the predeclared minimum useful effect. If the test was sized for a larger lift, a follow-up with the correct sample size may be warranted; see how to calculate sample size.

Interpreting results responsibly

State the comparison, denominator, time window, counts, rates, test statistic, degrees of freedom, p-value, effect estimate, and interval. “χ²(1) = 3.49, p = .062” means that the observed departure from equal rates is not below the chosen 0.05 threshold under this analysis. It does not say there is a 6.2% chance the treatment works or that future traffic will show the same rate.

A significant result should be translated into decision units. A 0.05-point increase can be highly significant with millions of users but too small to cover engineering cost, while a larger estimated lift may remain uncertain in a small test. Review primary and guardrail outcomes together; a conversion increase paired with higher cancellations, support contacts, or latency can be a poor product outcome.

Limitations and common mistakes

• Treating sessions as independent users: repeated rows narrow uncertainty incorrectly.
• Ignoring clusters or interference: correlated users need a design-aware estimator.
• Using sparse cells blindly: the asymptotic p-value can be unreliable for rare events.
• Testing many variants, metrics, or segments: an omnibus result and follow-ups require a multiplicity plan; see multiple comparisons.
• Calling significance a winner: report practical effect size, uncertainty, costs, and guardrails.
• Repairing bad data with statistics: no test corrects broken randomization or inconsistent tracking.

Frequently asked questions about chi-square tests

Is a chi-square test the same as a conversion-rate test?

For two independent groups and a binary outcome, it gives the same two-sided conclusion as the uncorrected two-proportion z-test. The latter often presents lift more directly.

When should I use Fisher’s exact test?

It is useful for small or sparse two-by-two tables when the chi-square approximation is doubtful. It can be conservative and does not solve dependence or biased assignment.

Can I use it for three variants?

Yes. A three-by-two omnibus test asks whether any rate differs. Preplanned pairwise control comparisons need suitable adjustment.

Does a non-significant test prove variants are equal?

No. It means the data did not provide enough evidence against the null at the specified threshold. Examine intervals and the minimum detectable effect.

Should I apply Yates continuity correction?

Software choices differ. With large experiment samples it has little effect; with small samples, consider exact or simulation-based methods and report the procedure used.

Summary

A chi-square test compares observed and expected category counts. It is a practical test for independent binary or categorical A/B outcomes, but valid use depends on the randomization unit, adequate expected counts, clean denominators, and a decision that considers effect size and uncertainty rather than a p-value alone.

Sources

• NIST/SEMATECH e-Handbook: Chi-Square Goodness-of-Fit Test
• NIST/SEMATECH e-Handbook: Tests for Two Proportions
• CDC Principles of Epidemiology: Chi-Square Test

Related terms

• Two-Proportion Z-Test
• Z-Test
• T-Test
• Independent Samples T-Test
• Paired T-Test