Chi-Square or Fisher's Exact Test? The Small-Counts Rule

You have two groups and a yes-or-no outcome, arranged in a two-by-two table. Chi-square or Fisher's exact test? Textbooks mention a rule about small counts, but few explain why it exists or what happens when you ignore it. A single example shows the difference clearly.

Two tests, two different engines

The chi-square test is an approximation. It compares observed counts with expected counts and relies on a smooth distribution that works well when the sample is large. Fisher's exact test calculates the probability directly, using the hypergeometric distribution, so it does not depend on large-sample approximations (Kim, 2017).

The expected-count rule

The usual rule says chi-square is reliable only when no more than 20% of cells have expected counts below 5 and no cell is below 1 (Kim, 2017). In a two-by-two table with four cells, a single small expected count already breaks that rule. Expected counts are row total times column total divided by the grand total, not the observed numbers you see in the table.

An example where they disagree

Suppose a small trial gives these results. In the treatment group, 1 of 12 patients had a complication. In the control group, 6 of 12 did. The smallest expected count is 3.5, below the threshold.

The chi-square test for independence without a continuity correction returns p ≈ 0.025, apparently significant. With Yates' correction it rises to about 0.072, and the Fisher exact test gives p ≈ 0.069.

The "significant" result depended on using an approximation outside its comfort zone.

The odds ratio calculator shows why caution is needed: the estimated odds ratio is 0.09, but with such small counts its interval is very wide.

When the two agree

Now take a larger trial, with 40 of 200 patients affected in one group and 60 of 200 in the other. Every expected count is large. Chi-square gives p ≈ 0.021 and Fisher about 0.028, the same message. With big samples, the choice rarely changes the conclusion, though Fisher remains valid.

This is why many analysts simply use Fisher's exact test for every two-by-two table when software allows it. The cost is small, and it removes the need to check the rule.

Beyond two-by-two tables

For larger tables, exact tests become heavy to compute and software may use simulation or a related method. The chi-square test also assumes independent observations. Paired designs, such as the same patients before and after, need McNemar's test, and the strength of association in larger tables can be measured with Cramér's V.

Practical tips

Many software packages print a warning when expected counts are small, and it is worth heeding. Do not combine categories after seeing the results just to escape small cells, since that invites cherry-picking. Decide any merging of categories in advance, and report both the exact and approximate results if they disagree.

The p-value is also sensitive to how the data were collected. A table built from a randomized trial supports different conclusions from one built from a convenience sample. Describe the design, not only the test.

A last reminder: neither test tells you how big an effect is. Report the counts, an effect measure such as the odds ratio or risk difference, and its interval alongside the p-value.