9 Exercises and a multiplicity checklist
Let us put the contents of the previous chapters to use with your own hands. The code cells in this chapter do not run automatically. Fill in the blanks (___) and press “Run Code”. Example answers are collapsed below each question.
9.1 Multiplicity checklist
| # | What to check | Chapter |
|---|---|---|
| 1 | Is there one primary endpoint? If several, is it “any one” or “all”? | 5 |
| 2 | Have you decided the family of tests used for claims? | 8 |
| 3 | To control the FWER, use Holm (not Bonferroni). If the assumptions hold, Hochberg or Hommel | 2 |
| 4 | If you claim effects on secondary endpoints, did you plan a fixed-sequence procedure or graphical approach? | 5 |
| 5 | For comparisons of three or more groups, have you decided what to compare (all pairs / comparisons with a control / a small number of pre-specified comparisons)? | 4 |
| 6 | Are you using “no adjustment if the overall test is significant” (protected LSD) with four or more groups? | 4 |
| 7 | For screening with massive numbers of tests, use the FDR (BH). Do not use it for confirmatory conclusions | 3 |
| 8 | Are the subgroups a small, pre-specified set? Did you examine differences in effect with tests for interaction? | 6 |
| 9 | Did you decide analytic choices such as outcomes, covariates, outliers, and participants before looking at the data? | 6 |
| 10 | Did you plan the number, timing, and boundaries (α-spending function) of interim analyses? | 7 |
| 11 | Did you show adjusted confidence intervals and effect sizes? | 2, 4, 8 |
| 12 | Did you label exploratory analyses as exploratory and give the number of tests? | 8 |
| 13 | Did you report deviations from the protocol, analysis plan, and registration? | 6, 8 |
9.2 Exercise 1: eight endpoints
In a trial, the p-values for eight endpoints came out as follows. Compare the number of significant results with no adjustment, Bonferroni, Holm, Hochberg, and BH.
sapply(c("none", "bonferroni", "holm", "hochberg", "BH"), function(m) sum(p.adjust(p, method = m) < 0.05))Unadjusted 6, Bonferroni 2, Holm 3, Hochberg 3, BH 6. Holm finds more than Bonferroni, and BH, which controls the FDR, finds more still. The conclusion changes depending on what you want to control (FWER or FDR), so it must be decided in advance.
9.2.1 1-2. The Holm method by hand
Sort the p-values above in increasing order, compare them with the Holm thresholds (0.05 / (k − i + 1)), and check where the procedure stops.
data.frame(rank = 1:k, p_value = ps, threshold = 0.05 / (k:1), significant = ps < 0.05 / (k:1))1st: 0.001 < 0.00625, 2nd: 0.006 < 0.00714, 3rd: 0.008 < 0.00833 are significant; the procedure stops at the 4th, 0.012 > 0.01. Three are significant.
9.3 Exercise 2: self-rated health and blood pressure
Among NHANES participants aged 20 and over, compare systolic blood pressure across levels of self-rated health (HealthGen, 5 levels). Compare all pairs with the Tukey method.
tk <- TukeyHSD(aov(BPSysAve ~ HealthGen, data = a))Of the 10 pairs, 8 show significant differences — all except Excellent vs. Vgood and Fair vs. Poor. Blood pressure tends to be higher the worse the self-rated health, but there is confounding by age and other factors. Because the groups are ordered, a single trend test may fit the question better than 10 comparisons (Chapter 4).
9.4 Exercise 3: testing many times along the way
In a study planned with 30 people per group, a t-test is run each time 10, 15, 20, 25, and 30 people (per group) have accrued, and the study stops and is declared “significant” as soon as p < 0.05. When there is truly no difference, check by simulation what percentage of studies become “significant”.
any(ps < 0.05)About 12%. Deciding the sample size while watching the data — “maybe it’ll become significant if I add a few more people” — is also a multiplicity problem (Chapter 7).
9.5 Exercise 4: your own study
For a study you are planning, write down the following.
- The primary endpoint and the family of tests used for claims
- Which secondary endpoints you want to claim effects for, and the order in which to test them (fixed sequence)
- The subgroups to pre-specify (ideally 5 or fewer)
- What to treat as exploratory analyses
9.6 Review questions
- With 10 independent tests (each at a 5% significance level), what is the probability of at least one false positive?
- Why can the Holm method be said to be always better than the Bonferroni method?
- Explain the difference between the FWER and the FDR. In what settings is each used?
- In a fixed-sequence procedure (hierarchical testing), what happens if you get the order wrong?
- Why must you not conclude “it does not work in women” from “significant in men, not significant in women”?
- Why are O’Brien–Fleming-type boundaries widely used in interim analyses?
- 1 − 0.95^10 ≈ 40% (Chapter 1)
- Both control the FWER regardless of the relationship between tests, and anything significant by Bonferroni is always significant by Holm (Chapter 2)
- The FWER is “the probability that even one false finding is included”; the FDR is “the mean proportion of false findings among discoveries”. The FWER is used for confirmatory conclusions (primary and secondary endpoints of clinical trials), the FDR for exploratory settings such as screening (Chapter 3)
- If an earlier endpoint is not significant, all later endpoints are declared non-significant (even if they truly have an effect) (Chapter 5)
- Splitting into subgroups reduces the sample size and lowers power, so results often become non-significant even when the effect is the same. Examine differences in effect with tests for interaction, and limit those to a small pre-specified set (Chapter 6)
- Because the early thresholds are very strict and the final-analysis threshold is close to 0.05, the trial can be stopped early only for large effects, with almost no change in power or required sample size (Chapter 7)