2 Controlling the FWER: Bonferroni, Holm, Hochberg
In Chapter 1 we saw that running many tests increases the probability of at least one false positive (the FWER). This chapter covers methods that keep the FWER at or below 5%.
As an example, suppose the p-values for five outcomes came out as follows.
Without adjustment, all five are significant.
2.1 The Bonferroni method
The simplest method is to divide the significance level by the number of tests k (for five, 0.05 / 5 = 0.01). Equivalently, you can multiply the p-values by k (capping at 1) and compare them with 0.05 (adjusted p-values).
With the Bonferroni method, only blood pressure and glucose remain significant.
The Bonferroni method keeps the FWER at or below α whatever the relationship between the tests. This is because the “probability of at least one false positive” is at most the “sum of the probabilities that each is a false positive” (α/k × k = α).
2.2 The Holm method
Because the Bonferroni method always divides everyone by k, it is conservative (significance is hard to reach). The Holm method reduces the divisor by one at each step, in increasing order of p-values.
- Sort the p-values in increasing order: \(p_{(1)} \le p_{(2)} \le \dots \le p_{(k)}\)
- Compare \(p_{(1)}\) with α/k. If significant, go on; if not, stop here (all remaining ones are non-significant)
- Compare \(p_{(2)}\) with α/(k − 1). If significant, go on…
- Compare \(p_{(i)}\) with α/(k − i + 1)…
- Blood pressure: 0.001 < 0.01 → significant
- Glucose: 0.008 < 0.0125 → significant
- LDL: 0.012 < 0.0167 → significant
- Weight: 0.030 > 0.025 → stop here. Weight and sleep are not significant
With the Holm method, LDL is also significant. In R, p.adjust() computes adjusted p-values.
Like the Bonferroni method, the Holm method keeps the FWER at or below α whatever the relationship between the tests. On top of that, anything significant by Bonferroni is always significant by Holm. If you want to control the FWER, use the Holm method rather than the Bonferroni method.
2.3 The Hochberg and Hommel methods
The Hochberg method checks the same thresholds as Holm, but in decreasing order (a step-up procedure).
- Compare the largest, \(p_{(k)}\), with α. If significant, all are significant and you are done
- If not, compare \(p_{(k-1)}\) with α/2. If significant, everything smaller is significant…
With the Hochberg method, sleep (0.04 < 0.05) is significant, so all five are significant. The Hommel method has slightly higher power still.
The Hochberg and Hommel methods keep the FWER at or below α when the tests are independent or positively correlated (which usually holds for outcomes in clinical trials). When there may be negative correlation, or when the assumption is hard to justify, choose the Holm method, which works under any relationship.
2.4 Comparing by simulation
We run 2000 times a two-group trial with 50 people per group in which, of 10 outcomes, only 3 have a true effect (standardized difference 0.5).
- Without adjustment, the probability that one of the 7 outcomes without an effect is significant is about 31%
- All four adjustment methods keep the FWER at about 5%
- In exchange, the proportion of outcomes with an effect that are detected drops from about 71% unadjusted to about 37–38%. Holm, Hochberg, and Hommel are slightly higher than Bonferroni
Controlling the FWER makes it easier to miss true effects (power goes down). The more outcomes you add, the lower the power for each one. Reducing the number of tests (narrowing down to one primary endpoint, performing only pre-specified comparisons) is the most effective remedy (Chapter 5).
2.5 Adjusting confidence intervals too
The Bonferroni idea also applies to confidence intervals. If you want the probability that the confidence intervals for the differences in five outcomes all simultaneously contain the true values to be at least 95%, compute a 99% (1 − 0.05/5) confidence interval for each.
Showing adjusted (simultaneous) confidence intervals, not just adjusted p-values, lets readers see the size of the effect and its uncertainty.
2.6 Try it
- Adjust the following six p-values with the Bonferroni, Holm, and Hochberg methods, and count the significant ones: 0.002, 0.009, 0.014, 0.020, 0.200, 0.450
- In the simulation above, if you increase the number of outcomes with an effect from 3 to 6, how does the Holm method’s “proportion detected” change?
- Bonferroni: 1 (only 0.002; 0.009 × 6 = 0.054), Holm: 2, Hochberg: 2. With Holm, 0.002 < 0.05/6 and 0.009 < 0.05/5 are significant, and the procedure stops when the third, 0.014, is compared with 0.05/4 = 0.0125. With Hochberg, going from the largest, 0.45 > 0.05, 0.20 > 0.025, 0.020 > 0.0167, and 0.014 > 0.0125 are not significant; then 0.009 < 0.01, so it and the smaller one, two in total, are significant
- It rises slightly, to about 39%. With the Holm method, the divisor shrinks each time a significant result is found, so the more outcomes have an effect, the easier it is for the rest to become significant too (the Bonferroni method has no such bonus)
- Bonferroni: compare with α/k (p × k). Controls the FWER for tests with any relationship, but is conservative
- Holm: compare in increasing order with α/k, α/(k − 1), …, and stop at the first non-significant one. Always better than Bonferroni, with no extra assumptions. The first choice for controlling the FWER
- Hochberg, Hommel: higher power still, but assume the tests are independent or positively correlated
- Adjustment lowers power. Reducing the number of tests is the best remedy
- Confidence intervals can be adjusted too (for Bonferroni, a 1 − α/k confidence interval)