3 Many tests and the FDR
With gene expression, metabolites, images, many biomarkers and so on, you may run hundreds to tens of thousands of tests. In such settings, controlling the FWER (the probability of at least one false positive) at 5% is so strict that almost nothing is found. What is used instead is the idea of controlling the FDR (false discovery rate).
3.1 1000 biomarkers
To search for biomarkers associated with a disease, we compare 1000 markers between patients and healthy people. Suppose that only 100 are truly associated, and the remaining 900 are not.
To keep the computation fast, we generate each marker’s test statistic (z-value) directly. Associated markers follow a normal distribution with mean 3, and non-associated markers one with mean 0.
Let us look at the distribution of p-values.
The p-values of non-associated markers are spread flat over 0–1, while those of associated markers cluster near 0.
3.2 No adjustment vs. Bonferroni
- Without adjustment, 143 are significant, but 52 of them are false (non-associated markers)
- With the Bonferroni method there are no false ones, but of the 100 truly associated markers, only 12 are found
3.3 The idea of the FDR
The purpose of screening is to “select candidates to investigate in detail in the next study”. Even if a few false ones are mixed in among the candidates, they can be checked in the next study, so there is no need to fear that even one false one slips in. Instead, we want to keep the proportion of false ones among the selected candidates below a certain level.
\[ \text{FDR} = E\left[\frac{\text{number of false discoveries}}{\text{number of discoveries}}\right] \]
Controlling the FDR at 5% means “among markers declared significant, false ones make up 5% or less on average”.
3.4 The Benjamini–Hochberg procedure
A representative method for keeping the FDR at or below q (e.g. 5%) is the Benjamini–Hochberg (BH) procedure.
- Sort the m p-values in increasing order: \(p_{(1)} \le \dots \le p_{(m)}\)
- Find the largest i satisfying \(p_{(i)} \le \frac{i}{m} q\)
- Declare \(p_{(1)}, \dots, p_{(i)}\) significant
Everything up to the last point below the red line is significant. The Bonferroni threshold (dotted line) is the same strict value at every rank, so far fewer are significant than with BH.
In R, p.adjust(p, "BH") computes BH-adjusted p-values (sometimes called q-values).
The BH procedure makes 56 discoveries, of which 4 (7%) are false.
3.5 Comparing over 1000 repetitions
| Method | What it controls | Discoveries | Proportion false | Detection rate of the true 100 |
|---|---|---|---|---|
| Unadjusted | ― | about 130 | about 35% | 85% |
| Bonferroni, Holm | FWER (probability of at least one false ≤ 5%) | about 15 | 0.4% | 15% |
| BH | FDR (mean proportion false ≤ 5%) | about 52 | 4.4% | 50% |
The BH procedure keeps the proportion of false discoveries at or below 5% while finding more than three times as many as the Bonferroni method. In exchange, the probability of “at least one false” is high, about 88%. They control different things.
| FWER (Bonferroni, Holm, etc.) | FDR (BH, etc.) | |
|---|---|---|
| Meaning | Probability that even one false finding is included | Proportion of false findings among discoveries |
| Suitable settings | Confirmatory studies: primary endpoints of clinical trials, regulatory submissions, conclusions that change practice | Exploratory studies: screening, narrowing down candidates, generating hypotheses |
| Number of tests | Few (a handful to a dozen or so) | Many (hundreds to tens of thousands) |
3.7 Try it
- If the number of truly associated markers is reduced from 100 to 10, how do the BH procedure’s number of discoveries and proportion of false discoveries change?
- When no marker is associated (m1 = 0), what is the probability of “at least one false” with the BH procedure?
- Discoveries drop to 2–3, and the proportion false stays at about 5% (when true associations are few, the proportion false approaches q × the proportion of non-associated markers ≈ 5%)
- About 5–6% (6% in the simulation). When no marker is associated, all discoveries are false, so FDR = FWER, and the BH procedure also keeps the FWER at 5%
- Controlling the FWER over a massive number of tests finds almost nothing
- FDR = the mean proportion of false findings among discoveries. Used in exploratory settings such as screening
- BH procedure: sort p-values in increasing order and declare significant everything up to the largest i satisfying \(p_{(i)} \le \frac{i}{m} q\).
p.adjust(p, "BH") - FWER for confirmatory conclusions, FDR for exploratory candidate selection. Decide which to use in advance