1 What is power?
Suppose you are planning a clinical trial of a new antihypertensive drug. You expect it to lower systolic blood pressure by a further 5 mmHg compared with the conventional drug. The reduction in blood pressure varies between people, with a standard deviation of about 10 mmHg.
If you recruit 20 people per group and run a t-test, will you find the difference?
1.1 Running one study
We create a world where “the true difference is 5 mmHg” ourselves, and run one study with 20 people per group.
The p-value is 0.40, not significant — even though there truly is a difference. If you change the number in set.seed() to 3 or 5, it becomes significant this time. Even with the same study plan, the result varies greatly by luck.
1.2 Running the same study 2000 times
We repeat a study under the same conditions 2000 times and count how many times it is “significant”.
Only about 33% were significant. This “probability of being significant when there truly is a difference” is called power. A study with 20 people per group was a plan that misses the difference two times out of three.
R has a function that calculates this.
power = 0.338. It almost matches the simulation.
1.3 When there truly is no difference
Next, in a world where there is truly no difference (both means are 0), we run the same study 2000 times.
About 5% were “significant”. The probability of being significant when there is no difference is the significance level α (usually 5%).
Let us put the distributions of p-values in the two worlds side by side.
- In the world with no difference, p-values are spread evenly from 0 to 1. 5% fall below 0.05
- In the world with a difference, p-values cluster near 0. The stronger the clustering, the higher the power
1.4 Two types of error
| Truly no difference | Truly a difference | |
|---|---|---|
| Significant | Type I error (probability α) | Correct detection (probability = power 1 − β) |
| Not significant | Correct | Type II error (probability β) |
- α (usually 5%) is a value the researcher decides
- β is a value determined by the sample size, the size of the difference, and the variability. Usually the sample size is chosen so that β is 20% or less (power 80% or more), or 10% or less (90% or more)
1.5 Increasing the sample size
We see how power changes as we vary the sample size per group.
How many people per group are needed for 80% power? If you specify power instead of n, you can back-calculate the sample size.
63.8 people per group, rounded up to 64 (128 in the two groups). Let us check by simulation.
It came out at about 80%.
1.6 The relationship among five values
Sample size calculation is a relationship among the following five values. Fixing four determines the remaining one.
| Value | Meaning | This example | Making it larger, the required sample size |
|---|---|---|---|
| Significance level α | Probability of being significant when there is no difference | 0.05 (two-sided) | decreases (but false positives increase) |
| Power 1 − β | Probability of being significant when there is a difference | 0.80 | increases |
| Difference Δ | Size of the difference you want to detect | 5 mmHg | decreases (acts as a square) |
| Standard deviation σ | Individual variability | 10 mmHg | increases (acts as a square) |
| Sample size n | Number per group | 64 | — |
- Raising power from 80% to 90% multiplies the sample size by about 1.3 (64 → 86)
- Halving the difference or doubling the standard deviation multiplies it by about 4 (64 → 253)
What determines the sample size is the ratio of Δ to σ (Δ / σ, the standardized difference). Δ = 5, σ = 10 and Δ = 0.5, σ = 1 give the same sample size.
The required sample size is proportional to (σ / Δ)². A small shift in the expected difference or variability changes the required sample size a lot. How to estimate the difference and the standard deviation (Chapters 2 and 4) is at the heart of sample size design.
1.7 Try it
- Obtain the power with 40 people per group using both
power.t.test()and simulation (sim_power_t()) - If the true difference were 3 mmHg instead of 5 mmHg, what would the power of a study with 64 people per group be?
- About 60% (0.598 with
power.t.test()). The simulation also gives around 60% - About 39%. A study planned with 64 people assuming 5 mmHg misses the difference with a probability of about 60% if the true difference is 3 mmHg
- Power = the probability of being significant when there truly is a difference. Usually set to 80% or 90%
- Significance level α = the probability of being significant when there truly is no difference. Usually 5% (two-sided)
- Of the five values — α, power, difference, standard deviation, and sample size — fixing four determines the rest
- The required sample size is proportional to (standard deviation ÷ difference)². Half the difference means 4 times the sample size
- Power can be checked by simulation (running the same study thousands of times)