5  When counts are small: Yates’ correction and Fisher’s exact test

Many people have been taught that “if any cell has an expected frequency below 5, use Fisher’s exact test instead of the χ² test”. chisq.test() also issues the warning Chi-squared approximation may be incorrect when expected frequencies are small. In this chapter, we check the meaning of this rule and what is really happening with exact calculations.

5.1 The lady tasting tea

This is Fisher’s famous example. A lady claimed: “Milk tea tastes different depending on whether the tea or the milk is poured first, so I can tell which.” So 4 cups with tea poured first and 4 cups with milk poured first were served in random order, and she was asked to choose the 4 “tea first” cups. As a result, she got 3 of the 4 right.

All expected frequencies are 2, and chisq.test() issues a warning. With only 8 cups, the χ² approximation cannot be trusted.

5.2 Fisher’s exact test: count every possibility

In this experiment, both the row totals (4 cups each) and the column totals (4 cups are answered “tea first”) are fixed from the start. Then the table is determined solely by how many of the 4 “tea first” cups she got right (0–4). If she chose 4 cups at random, there are 70 ways in total to choose 4 cups from 8, of which…

  • The probability of getting 3 or more right is (16 + 1) / 70 = 0.243 (one-sided)
  • The probability that “the hits are skewed to 3 or more” or “the misses are equally skewed (1 or fewer)” is (1 + 16 + 16 + 1) / 70 = 0.486 (two-sided)

This is Fisher’s exact test. Using the probabilities of tables with fixed marginal totals (the hypergeometric distribution, dhyper()), it sums the probabilities of tables at least as skewed as the observed one.

With 3 of 4 correct, she cannot be distinguished from guessing (if all were correct, one-sided p = 1/70 = 0.014). The p-value from the χ² approximation (0.157) differs greatly from the exact value (0.486).

NoteRelation to the shuffling in Chapter 2

Fisher’s exact test enumerates, with their probabilities, all the tables that can arise from the “label shuffling” in Chapter 2. You can also think of it as the p-value from repeating the shuffling infinitely many times.

5.3 Where did the “expected frequency of 5” rule come from?

The guideline usually cited is the following from Cochran (1954):

  • No cell has an expected frequency below 1
  • No more than 20% of cells have an expected frequency below 5

“At least 5 in every cell” is an even stricter, easier-to-remember version of this. Let us check how important the rule is in terms of type I error (the probability of being “significant” when there is truly no difference).

5.4 Exact calculation: type I error

Suppose there are 10 people in each of 2 groups, and both groups have “an improvement rate of 30%” (truly no difference). The number improved in each group is 0–10, so there are only 11 × 11 = 121 possible tables. If we check, for every table, the probability of that table and whether the test is significant, we can compute the type I error exactly.

For a test at the 5% significance level, the probability of significance when there is no difference should be 5%.

  • χ² test without correction: 3.7%. Despite the small numbers and many cells with expected frequencies below 5, it does not exceed 5%
  • χ² test with Yates’ correction: 1.2%
  • Fisher’s exact test: 1.2%

Let us vary the number of people and the improvement rate (this takes a little time).

ImportantWhat we found
  • Fisher’s exact test and Yates’ correction have a type I error considerably smaller than 5% (conservative)
  • The χ² test without correction is, in many cases, close to 5% even when expected frequencies fall below 5 (occasionally slightly above)

5.4.1 Why it is “exact” yet not 5%

The p-value of Fisher’s exact test is computed exactly, but since the table takes only discrete values, you cannot cut at “exactly 5%”. Significance is decided so as not to exceed 5%, so the actual type I error is smaller than 5%. Yates’ correction makes the χ² value smaller to bring it closer to Fisher’s p-value, so it is similarly conservative.

5.4.2 Why being conservative is a problem: power

A conservative test is more likely to miss real differences. Using the same method, compute the power (the probability of correctly being significant) when comparing two treatments whose true improvement rates are 20% and 60%, with 20 people per group.

Compared with the χ² test without correction, Yates’ correction and Fisher’s exact test have power more than 10 percentage points lower.

5.5 So what should you do?

For small 2×2 tables, here is a summary of the views commonly held today.

  1. Fisher’s exact test is the safest method, guaranteeing that the type I error does not exceed 5%. Use it knowing that it is conservative. It is easy to justify in clinical trials and peer review
  2. Yates’ correction has almost no advantage over Fisher’s exact test, so it is often no longer recommended. R’s chisq.test() applies Yates’ correction by default for 2×2 tables, so always check which one you used
  3. Some research shows that if expected frequencies are roughly 1 or more, the χ² test without correction (or the “N − 1 χ² test”, which multiplies the χ² value by (N − 1)/N) also has a type I error close to 5% (Campbell, 2007)

Decide which to use in the analysis plan, before looking at the data. Choosing whichever gives the smaller p-value after seeing the results increases the type I error.

5.6 When expected frequencies are small in a larger table

Fisher’s exact test (fisher.test()) can also be used for tables larger than 2×2, but with many people the computation takes time. In that case, the simulated p-value from Chapter 2 is convenient.

At the first Framingham examination, look at the relationship between BMI categories and diabetes.

The expected frequency of the cell for BMI 35+ with diabetes is 3.5; 1 of the 8 cells (12.5%) is below 5.

The χ² p-value is 1.1×10⁻⁷, and in the simulation not one of the 20,000 tables reached the observed χ² value (the lower limit of p is 1/20001 ≈ 5×10⁻⁵), so the conclusion is the same either way. If only a few cells have expected frequencies below 5, the χ² approximation usually does not break down badly. When in doubt, check by simulation.

WarningDon’t merge categories after seeing the results

Merging sparse categories with their neighbours to increase expected frequencies is common, but decide how to merge before looking at the data. Changing how you merge while looking at the p-value is itself a form of multiple testing (Chapter 8).

ImportantSummary of this chapter
  • Fisher’s exact test fixes the marginal totals and enumerates the probabilities of all tables
  • “Fisher if an expected frequency is below 5” is a rule on the safe side. Fisher and Yates are conservative (type I error < 5%, lower power)
  • The χ² test without correction often holds up better than expected even for small tables
  • R’s chisq.test() applies Yates’ correction by default for 2×2 tables. Decide which to use before the analysis
  • When expected frequencies are small in a larger table, check with simulate.p.value = TRUE