1 Why false positives increase
A test at the 5% significance level keeps the probability of “becoming significant even though there is truly no difference” (a false positive) at 5%. So what happens if you run many tests?
1.1 Twenty outcomes
In a trial investigating the effect of a food, 20 outcomes were measured: blood pressure, blood glucose, cholesterol, body weight, sleep, mood, and so on. In truth, the food has no effect on any of the outcomes. We run a two-group comparison with 50 people per group, and a t-test for each of the 20 outcomes.
Although truly nothing has an effect, the 10th and 14th outcomes came out significant. You could write in a paper that “XX improved significantly”.
1.2 Running the same trial 2000 times
When 20 outcomes are tested, in about 65% of trials at least one is significant. On average, one outcome is significant (20 × 5%).
1.3 How much it grows with the number of tests
For k mutually independent tests, the probability that at least one is a false positive is
\[ 1 - (1 - 0.05)^k \]
It is about 10% with 2 tests, about 23% with 5, about 40% with 10, and about 64% with 20. This “probability of at least one false positive within a collection (family) of tests” is called the FWER (family-wise error rate, the type I error rate per family).
1.4 When the tests are similar
When outcomes are correlated, like systolic and diastolic blood pressure, false positives tend to occur together, and the FWER grows more slowly.
With 10 outcomes, it is about 41% with no correlation, about 24% with correlation 0.6, and about 14% with correlation 0.9. This is because the stronger the correlation, the closer the 10 tests come to “asking the same question 10 times”. Even so, it is well above 5%.
1.5 Multiplicity hides in many places
“Running many tests” does not only happen when there are many outcomes.
| Setting | Example | Chapter |
|---|---|---|
| Multiple outcomes | Testing blood pressure, blood glucose, body weight… all of them | 2, 3, 5 |
| Comparing three or more groups | Comparing 4 groups two at a time, 6 times | 4 |
| Massive numbers of tests | 20,000 genes, image voxels, many biomarkers | 3 |
| Subgroups | Testing separately by sex, age, or severity | 6 |
| Analytic choices | Choosing how to remove outliers, which variables to adjust for, or which time point to evaluate while looking at the results | 6 |
| Interim analyses | Looking at the data repeatedly during a trial and stopping once it becomes significant | 7 |
Even if each individual test is correct, running many tests and reporting or interpreting only the significant ones increases the probability that false-positive results slip in. There are two remedies.
- Adjust: correct the significance level or p-values to control false positives across the whole family (Chapters 2–5, 7)
- Decide in advance: fix the primary endpoint, comparisons, and analysis methods before looking at the data, and limit the number of tests (Chapters 5, 6, 8)
1.6 Try it
- With 5 outcomes, what proportion of trials have at least one significant outcome? (Both by simulation and by formula)
- To keep the “probability of at least one false positive” at 5%, what significance level should each individual test use? (20 independent tests)
- About 23% (22.6% by the formula)
- About 0.00256 (Šidák correction). It is almost the same as the simpler 0.05 / 20 = 0.0025 (Bonferroni correction), covered in Chapter 2
- If you run many tests at the 5% significance level, the probability of at least one false positive (FWER) grows. About 64% for 20 independent tests
- Correlation between tests makes it grow more slowly, but it still exceeds 5%
- Multiplicity hides in many places: outcomes, groups, subgroups, analytic choices, interim analyses, and more
- The remedies are to adjust, and to decide in advance and limit the number of tests