Quick definition: Fisher’s exact test evaluates association between two categorical variables in a 2×2 table by calculating probabilities from the exact hypergeometric distribution, conditional on the observed row and column totals. It is especially useful when expected cell counts are small.
What is Fisher’s exact test?
Fisher’s exact test asks whether a binary outcome is associated with a binary group indicator. In a simple A/B experiment, the rows are control and treatment and the columns are converted and not converted. Its null hypothesis is that the outcome probability is the same in both groups. Unlike a large-sample chi-square approximation, the classic test obtains a p-value from the probabilities of possible tables with the same margins as the observed table.
“Exact” describes the reference distribution under the conditional null, not the absence of assumptions or measurement error. The test does not prove that two rates are equal, and a small p-value does not supply the effect size, practical importance, or data-quality evidence needed for a product decision. It is a hypothesis test for a well-defined 2×2 question.
Method, formula, and assumptions
Write a table with treatment successes a, treatment failures b, control successes c, and control failures d. Given fixed row totals and column totals, the probability of the observed table under the null is P(A=a)= [C(a+b,a) C(c+d,c)] / C(a+b+c+d,a+c). The two-sided p-value sums probabilities for tables judged at least as incompatible with the null as the observed table; software conventions for “two-sided” can differ, so report the implementation.
The observations should represent independent units under the sampling or randomization design. In a user-randomized test, one user should contribute one binary outcome. Multiple sessions per user, shared household behavior, or store-level assignment create dependence and make a simple user-row Fisher test inappropriate. The test also assumes a meaningful fixed observation window and comparable outcome ascertainment. Exact calculation does not compensate for differential logging or an exposure rule that varies by variant.
Fisher’s test is often chosen when a cell count is small, such as rare purchase, fraud-report, or severe-error outcomes. For large tables it can be computationally heavy and can be more conservative than necessary. A two-proportion test, chi-square test, or logistic regression can be efficient and informative for large samples; choose based on the estimand and design, not a mechanical “expected count under five” rule.
Fisher’s exact test in experiments
Use Fisher’s exact test for a predeclared binary endpoint when traffic is limited or the event is rare. A mobile team might test whether an account-recovery flow changes completion among 600 eligible users; a reliability team might compare an uncommon blocking error between variants. Report both outcome counts and an effect estimate such as risk difference, risk ratio, or odds ratio with an interval. A p-value without a magnitude invites overreaction to a tiny but statistically notable difference or indifference to a material but imprecise one.
For the primary intention-to-treat analysis, retain all eligible assigned users in the denominator. Restricting the table to users who reached a downstream screen can condition on behavior altered by treatment. Validate allocation and exposure first: a rare-event table is particularly sensitive to one missing event or one misclassified user. The diagnostics in sample ratio mismatch are relevant before interpreting any inferential output.
Do not use an exact test to justify unplanned repeated peeking. Each new look is another chance to find a small p-value. Plan a fixed horizon or an appropriate sequential design, and account for multiple variants, metrics, and segments as explained in multiple comparisons for A/B tests.
Worked example: rare checkout failures
In a 2,000-user checkout test, control has 14 payment failures among 1,000 users and treatment has 4 among 1,000. The table is treatment: 4 failures, 996 non-failures; control: 14 failures, 986 non-failures. The observed failure rates are 0.4% and 1.4%, an absolute difference of −1.0 percentage point. Fisher’s exact two-sided test returns a p-value near 0.02 under a conventional implementation.
The appropriate interpretation is that, under the conditional null and assumptions, this table would be relatively unusual. The team should report the estimated one-point reduction, uncertainty interval, payment-provider logs, assignment and exposure counts, and whether the failure definition was fixed before the test. If the new flow also slows legitimate payments, a primary improvement does not settle the release decision; review latency, completion, refunds, and support contacts using the framework for primary and guardrail metrics.
Interpretation workflow
- Define one independent analysis unit and a binary endpoint with a fixed window.
- Show the complete 2×2 table, allocation counts, and outcome rates.
- Validate randomization, event completeness, and the planned stopping rule.
- Compute the chosen one- or two-sided exact p-value and a decision-relevant effect estimate.
- Compare benefit and harm with practical thresholds and other predeclared metrics.
A non-significant result means the planned test did not reject its null at the selected threshold. It does not show equivalence or safety, especially with rare events. A significant result can be operationally trivial if the absolute change is tiny. For rare safety outcomes, the interval width and a maximum acceptable harm threshold are frequently more useful than a binary declaration.
Limitations and common mistakes
- Calling the conclusion assumption-free. Independence, valid measurement, and a coherent sampling or assignment process remain essential.
- Reporting only the p-value. Include counts, absolute rates, effect size, and uncertainty.
- Ignoring two-sided definitions. Different packages may calculate two-sided exact p-values differently.
- Applying it to repeated events. Aggregate to the randomized user or use a model that handles dependence.
- Choosing the test after seeing sparse cells. Predefine the primary method or explain the diagnostic reason for an alternative.
- Overlooking multiplicity. Exact p-values do not protect a family of many endpoints or repeated looks.
Exactness also has a cost in communication. A conditional p-value answers a narrower question than a product team may have in mind, while the odds ratio naturally associated with a 2×2 table can be difficult to translate when the event is rare. Center the decision on an absolute risk difference and the number of affected users, and use the exact test as one component of the evidence. If the table is sparse because the experiment was far smaller than planned, an inconclusive result should motivate an adequately powered follow-up rather than an exaggerated claim of safety.
Frequently asked questions about Fisher’s exact test
When should I use Fisher’s exact test instead of chi-square?
It is a strong option for small expected counts or rare outcomes in a 2×2 table. With large samples, a well-specified proportion test or regression is often simpler and more scalable.
Does Fisher’s test estimate conversion lift?
No. It tests an association. Calculate and report conversion rates and a risk difference or other estimand separately.
Is the test always conservative?
Because it conditions on margins and uses discrete probabilities, the attainable type I error may be below the nominal level. Exact behavior depends on the table and two-sided convention.
Can I use it for more than two variants?
Not as a single ordinary 2×2 test. Use an appropriate r×c exact method, regression, or planned pairwise comparisons with multiplicity control.
Does random assignment make the test causal?
Random assignment supports causal interpretation when execution and measurement are valid. The test itself only analyzes the observed table.
Summary
Fisher’s exact test provides an exact conditional test for a 2×2 association, especially valuable for sparse binary outcomes. Use it with a defined independent unit, transparent counts and effect estimates, valid experiment operations, and a plan for repeated looks and multiple questions.
Sources
- NIST/SEMATECH e-Handbook: Fisher’s Exact Test
- Fisher, On the Interpretation of χ² from Contingency Tables
- Penn State STAT 200: Tests for Two Proportions