6 Subgroup analyses and the “garden of forking paths”
The places where multiplicity is hardest to notice are subgroup analyses and the choice of analysis. In both cases each individual analysis is correct, but “trying many things and reporting what stands out” lets false positives slip in.
6.1 A trial in which everyone has the same effect
In a trial with 400 people per group, suppose the effect of a new treatment is the same for everyone (standardized difference 0.2). We split participants into two subgroups by each of 10 baseline factors (sex, age, diabetes, …) and examine the effect. The baseline factors have nothing to do with the effect.
Overall, the effect is significant (p = 0.0001). Let us estimate the effect in each subgroup.
Although in truth everyone has the same effect,
- 5 of the 20 subgroups show non-significant results
- It is tempting to read this as “it doesn’t work for people with XX”, but the sample size was simply halved and power dropped
6.2 Testing for interaction
Whether the effect differs between subgroups is examined not by comparing the p-values of each subgroup, but by a test for interaction.
Of the 10 interactions, the interaction for “odd birth month” is significant (p < 0.05). Birth month cannot possibly affect the treatment effect. It is chance. With 10 interaction tests, it is not unusual for one of them to be significant by chance.
6.3 Repeating 300 times
- The overall effect is significant in about 81% of trials (a trial with about 80% power)
- But in about 95% of trials, a “subgroup in which the effect is not significant” is found
- And in about 41% of trials, some interaction is significant by chance
- Do not compare p-values across subgroups. “Significant in group A, not significant in group B” is not evidence that the effect differs. Use a test for interaction
- Interaction tests also become significant by chance if you run many of them. Restrict to a small number of pre-specified subgroups, and report how many there were
- Check whether there is a biological reason for the difference in effect, whether it has been replicated in other studies, and whether the difference is large enough to reverse the direction of the overall effect
- A famous example: in a large myocardial infarction trial (ISIS-2), aspirin appeared to have no effect in people born under the astrological signs Gemini or Libra
6.4 Forking paths: the multiplicity of analytic choices
Not only with subgroups: data analysis involves many choices.
- Which outcome to use (one of two measures, or their average)
- Whether to adjust for a covariate
- Whether to remove outliers
- Everyone, men only, or women only
What happens if, on data with no true difference, you try all combinations of these (3 × 2 × 2 × 3 = 36) and pick the smallest p-value?
With one analysis decided in advance, false positives are about 5%. But if you choose the best of the 36, about 33% of studies obtain a “significant result”. Even if each choice is “plausible” on its own, choosing while looking at the results destroys the meaning of the p-value. This is called the “garden of forking paths” or p-hacking.
- Write the primary endpoint, analysis methods, covariates, handling of outliers, and subgroups into the study protocol and statistical analysis plan (SAP) before looking at the data
- Pre-register clinical trials (jRCT, ClinicalTrials.gov, etc.), and if you perform analyses that differ from the plan, report them along with the reasons
- Label exploratory analyses as exploratory, and also report how many were tried
6.5 Try it
- If the number of baseline factors is increased from 10 to 20, what happens to the proportion in which some interaction is significant by chance? (Estimate with the formula \(1 - 0.95^k\))
- In the forking-paths simulation, if you remove the choice “remove outliers or not” (set
trimtoFALSEonly), how does the proportion with smallest p-value below 0.05 change?
- About 40% with 10, about 64% with 20 (interaction tests are close to independent of each other, so the formula fits well)
- It drops to about 22%. The fewer the choices, the fewer the false positives, but with 18 combinations it still far exceeds 5%
- Even when everyone has the same effect, splitting into subgroups makes it easy to find a “subgroup where it doesn’t work” (only because the smaller sample size lowers power)
- Examine differences in effect with tests for interaction. These too become significant by chance if you run many
- Choosing analytic options (outcome, adjustment, outliers, participants) while looking at the results greatly increases false positives (about 33% with 36 combinations)
- The remedy is to decide and write things down before looking at the data