4  Comparing three or more groups

When comparing three or more groups, trying to find out “which group differs from which” requires many two-group comparisons (6 for 4 groups, 10 for 5 groups). This chapter looks at methods that deal with the multiplicity of group comparisons (post hoc comparisons, multiple comparisons).

4.1 Four groups with no true difference

We generate data for 4 groups of 30 people each in which there is truly no difference between any of the groups, and run 6 pairwise t-tests. We repeat this 2000 times.

  • With 6 unadjusted comparisons, about 20% of studies find “a significant difference somewhere”
  • The Holm method and the Tukey method keep this at about 5%
  • The approach “compare pairs without adjustment only when the overall F test is significant” (the protected LSD, Fisher’s least significant difference) is also about 5% in this example. However…

4.2 The pitfall of the protected LSD

Consider the case where, out of 6 groups, only one group truly differs (the other 5 groups do not differ).

Because group 6 differs greatly, the overall F test is almost always significant. The “protection” then does nothing, and the 10 comparisons among the 5 groups with no difference are performed without adjustment, producing a false positive about 27% of the time. The Tukey method keeps the FWER at or below 5% in every situation (control in the strong sense). The protected LSD is safe only with 3 groups.

4.3 Real data: race/ethnicity and BMI

Among NHANES participants aged 20 and over, we compare BMI across race/ethnicity categories.

4.3.1 All pairs: the Tukey method

When you want to compare all pairs of groups (10 pairs), use the Tukey method (Tukey HSD). It gives the estimated differences and adjusted (simultaneous) 95% confidence intervals.

All except the Hispanic–White difference (0.39, confidence interval −0.22 to 0.99) are significant.

4.3.2 Compared with unadjusted and Holm

The difference between Mexican and Hispanic has p = 0.003 unadjusted, but 0.006 with Holm and 0.026 with Tukey. Because NHANES is large, most comparisons stay significant even after adjustment. In small studies, adjustment often changes the conclusion.

4.3.3 Comparing only against a control: the Dunnett method

If all you want to compare is “each group vs. a reference group” (for example, comparing each of three doses of a new treatment with placebo), there are fewer comparisons, so the Dunnett method has higher power.

TipHow to choose a method
What you want to compare Method
All pairs of groups Tukey (Tukey–Kramer, which works with unequal group sizes)
Each group vs. a control Dunnett
A small number of pre-specified comparisons Test only those comparisons and adjust with Holm
Trend across ordered groups (doses, etc.) Perform a single trend test (contrast)
Unequal variances across groups / non-normal data Games–Howell, nonparametric multiple comparisons (Steel–Dwass, etc.)

Avoid “run the overall test first → if significant, compare all pairs without adjustment”. Deciding first what you want to compare reduces the number of comparisons and increases power.

4.4 Try it

NotePractice
  1. Among NHANES participants aged 20 and over, compare systolic blood pressure (BPSysAve) across education levels (Education), and obtain Tukey-adjusted confidence intervals
  2. If the first simulation uses 6 groups instead of 4, what happens to the unadjusted proportion of “a significant difference somewhere”?
  1. Groups with more education tend to have lower systolic blood pressure, and for the differences between college graduates (College Grad) and every other group, the confidence intervals exclude 0 even after adjustment (only the difference between High School and 8th Grade is not significant). Note, however, that the effect of education is confounded by age and other factors (Confounding hands-on, in Japanese)
  2. About 34%. With 6 groups there are 15 comparisons, and false positives increase further
ImportantChapter summary
  • Comparing three or more groups two at a time without adjustment increases false positives (about 20% with 4 groups)
  • “No adjustment if the overall test is significant” (protected LSD) fails to control the FWER with 4 or more groups
  • Tukey for all pairs, Dunnett for comparisons with a control, Holm for a small number of pre-specified comparisons
  • Show the size of differences with adjusted confidence intervals, not just adjusted p-values