4 Deciding the effect size
As we saw in Chapter 1, the required sample size is proportional to (σ / Δ)². If you misjudge the difference Δ or the standard deviation σ, the sample size goes badly wrong. This chapter covers how to decide Δ and σ, and common pitfalls.
4.1 There are two kinds of anticipated difference
| Approach | Meaning | Example |
|---|---|---|
| Minimal clinically important difference | If the difference is smaller than this, finding it would not change treatment | Systolic blood pressure 5 mmHg (the smallest difference that can be said to lower stroke risk) |
| Realistically expected difference | The difference that can be realistically expected from the mechanism of action and previous studies | 7 mmHg in a previous study |
- If the expected difference is larger than the minimal important difference, you would usually plan with the minimal important difference (to reliably detect a difference that must not be missed)
- If the expected difference is smaller than the minimal important difference, reconsider whether the study is worth doing at all
- “Only this many people can be recruited, so we back-calculated the detectable difference” tends to be an excuse rather than a plan
4.2 The pitfall of “standardized effect size 0.5”
By Cohen’s rule of thumb, a standardized difference d = Δ / σ of 0.2 is “small”, 0.5 “medium”, and 0.8 “large”. If you calculate with “a medium effect, d = 0.5”, every study gets the same sample size.
64 people per group. But how many mmHg d = 0.5 corresponds to depends on the standard deviation of the participants.
In people with blood pressure of 140 mmHg or more, d = 0.5 corresponds to a difference of 7.1 mmHg. An extra 7 mmHg in a comparison between antihypertensive drugs is quite optimistic. The rule of thumb for d is a last resort when neither Δ nor σ is known, and it has no clinical meaning. Think about differences in the units of the endpoint (mmHg, %, points).
4.3 The standard deviation from a pilot study is unreliable
Suppose the standard deviation is estimated from a pilot study of 20 people. Here, we regard the 962 NHANES participants with blood pressure of 140 mmHg or more as the “true population”, and run a pilot study by drawing 20 people from it 5000 times.
The standard deviation from a 20-person pilot study is 8.9 or less with 10% probability and 19.4 or more with 10% probability. The true value is 14.1. If we decide the sample size with this standard deviation and run the main trial with that sample size, what is the true power?
Although every trial is supposed to have 80% power by plan, the planned sample size varies widely, from 51 to 237 (10th–90th percentile), and in about 57% of trials the true power falls short of 80%. The sample size truly needed is 127.
Why more than half? Because estimates of the standard deviation tend to be biased low (the distribution has a long right tail). Underestimating the standard deviation leads to too few people and insufficient power.
4.3.1 Remedy: use the upper confidence limit of the standard deviation
Instead of using the pilot study’s standard deviation as is, one approach uses its upper confidence limit (e.g., the one-sided 80% upper limit).
The planned sample size increases a little (median 156), but the proportion of underpowered trials drops to about 36%. The confidence interval calculation assumes a normal distribution, but data from people with blood pressure of 140 or more are right-skewed, so it still does not go down to 20% (= 1 − 80%).
- Estimate it from existing data that are as large as possible (previous studies, registries, clinical data). Match the participant criteria
- If only a small pilot study is available, use the upper confidence limit, or calculate the sample size for several values and show the range
- The main purpose of a pilot study is to check procedures, recruitment, and dropout. Do not estimate the effect size from a pilot study (next section)
4.4 Effects in previous studies are too large
Suppose a previous study reported a “significant effect”. Can we use that effect size?
Consider a world where the true effect is d = 0.3, many small studies with 30 people per group are conducted, and only significant ones get published.
These studies have only about 21% power, and only studies that happened to find a large difference become significant. The mean effect of published studies is 0.65, more than twice the true effect of 0.3 (the winner’s curse).
If you believe this 0.65 and plan the next study:
A study planned with 39 people per group has a true power of about 26%. What was truly needed was 176.
- The smaller the study, the earlier the study, or the more only significant results were reported, the more likely the effect is overestimated
- Prefer values from meta-analyses that pool multiple studies, or from large studies
- Do not use the effect from previous studies as is; anticipate it conservatively (e.g., toward the lower confidence limit)
- Planning with “the minimal clinically important difference” avoids this problem
4.5 Calculate with several values and show the range
Since the anticipated Δ and σ are uncertain, present them as a table rather than a single value.
Over a difference of 4–6 mmHg and a standard deviation of 12–18 mmHg, the required sample size ranges from 64 to 319 per group, a fivefold difference. Including such a sensitivity analysis in the protocol, along with the rationale for the adopted values, makes it easier for reviewers and ethics committees to judge.
4.6 Try it
- If the pilot study size is increased from 20 to 60, how does the proportion of trials whose true power falls short of 80% change? (Change the 20 in
pilot_sdto 60 and do the same calculation) - If the sample size per group of previous studies is increased from 30 to 100, how does the mean effect of published studies change?
- Around 55%. The proportion falling short does not drop much, but the spread of planned sample sizes (especially trials that miss badly) becomes smaller. “About half fall short” is a property that does not change as long as the median of the estimate is close to the true value
- About 0.40. With 100 people per group the power is about 56%, and the overestimation of effects in published studies becomes smaller
- Base the anticipated difference on the minimal clinically important difference. The rule of thumb for standardized effect sizes (d = 0.5) is a last resort
- The standard deviation from a small pilot study fluctuates widely and tends to come out small. Use the upper confidence limit, or estimate it from large existing data
- Effects in previous studies where only significant results were reported are overestimated (the winner’s curse)
- Include a sensitivity analysis table varying Δ and σ in the protocol