There’s more than one way to calculate a Fisher’s exact p-value

Steve Simon

2011-07-21

This page is being updated from a version on the original website.

I was trying to check the calculations associated with a two by two table and I noticed an inconsistency in the reporting of results. One program reported a p-value of 0.4588 for the two-tailed Fisher’s exact test, and the other package reported a p-value of 0.308088. The packages otherwise agreed with one another. So which package is right? Well it turns out that both of them are correct because there is more than one way to calculate a Fisher’s exact p-value. To understand this, you need to recall the computational details of Fisher’s exact test.

Here are the two competing outputs from OpenEpi

Figure1. Output from OpenEpi

and the epitools package in R.

Figure2. Output from epitools

The table used in this example,

Figure 3. Two by two table

is one of sixteen possible tables that have the same row totals and column totals.

Figure 4. Sixteen possible two by two tables

Each of these tables has a probability associated with the hypergeometric distribution.

Figure 5. Tables with their hypergeometric probabilities

and you can draw a barchart of these probabilities.

Figure 6. Bar chart of table probabilities

The right-tailed p-value is just the sum of all the probabilities of the observed table (5) and any larger tables (6, 7, …).

Figure 7. Illustration of right-tailed p-value calculation

Figure 8. Illustration of right-tailed p-value calculation

The left-tailed p-value would be the sum of all the probabilities of the observed table (5) and any smaller tables (4, 3, …).

Figure 9. Illustration of left-tailed p-value calculation

Figure 10. Illustration of left-tailed p-value calculation

The problem with this (if you think it is a problem) is that the left-tailed and right-tailed p-values add up to more than 1.0 because the probability associated with the “5” table is counted in both p-values. So an alternative is to split this probability in half and add half to the right-tailed p-value and half to the left-tailed p-value. This is called the mid p correction.

Figure 11. Illustration of the mid p correction

Figure 12. Illustration of mid p correction

Figure 13. Illustration of the mid p correction

Figure 14. Illustration of the mid p correction

We’re not done yet because there is a second source of ambiguity, the calculation of a two-sided p-value. Most people calculate the two-sided p-value by doubling the one-sided p-value. That works perfectly if there is symmetry in the sampling distribution, but that is not the case here. When there is asymmetry, you get a better answer if you sum all the probabilities that are smaller than the probability for the observed table.

Figure 15. Illustration of two-sided p-value calculation

Figure 16. Illustration of two-tailed p-value calculation

So here’s how each of the p-values were computed.

Figure 17. OpenEpi output

The one tailed Fisher’s exact is 0.2294. This corresponds to a right tailed p-value as described above.

The two tailed Fisher’s exact p-value is 0.4588. This is simply twice the one-tailed p-value.

Figure 18. OpenEpi output

The epitools package in R produces two-tailed p-values only.

Figure 19. OpenEpi output

The two tailed p-value associated with Fisher’s exact test is 0.308088. This is computed by looking at the probability of the “5” table and adding in the probabilities of any tables less likely (0, 1, 6, 7, 8,…).

Figure 20. Illustration of two-tailed p-value calculation

Both programs are correct, as verified by my hand calculations shown above. The discrepancy is caused by a different computational algorithm in the two programs.

You can find an earlier version of this page on my original website.