1  What is power?

Suppose you are planning a clinical trial of a new antihypertensive drug. You expect it to lower systolic blood pressure by a further 5 mmHg compared with the conventional drug. The reduction in blood pressure varies between people, with a standard deviation of about 10 mmHg.

If you recruit 20 people per group and run a t-test, will you find the difference?

1.1 Running one study

We create a world where “the true difference is 5 mmHg” ourselves, and run one study with 20 people per group.

The p-value is 0.40, not significant — even though there truly is a difference. If you change the number in set.seed() to 3 or 5, it becomes significant this time. Even with the same study plan, the result varies greatly by luck.

1.2 Running the same study 2000 times

We repeat a study under the same conditions 2000 times and count how many times it is “significant”.

Only about 33% were significant. This “probability of being significant when there truly is a difference” is called power. A study with 20 people per group was a plan that misses the difference two times out of three.

R has a function that calculates this.

power = 0.338. It almost matches the simulation.

1.3 When there truly is no difference

Next, in a world where there is truly no difference (both means are 0), we run the same study 2000 times.

About 5% were “significant”. The probability of being significant when there is no difference is the significance level α (usually 5%).

Let us put the distributions of p-values in the two worlds side by side.

  • In the world with no difference, p-values are spread evenly from 0 to 1. 5% fall below 0.05
  • In the world with a difference, p-values cluster near 0. The stronger the clustering, the higher the power

1.4 Two types of error

Truly no difference Truly a difference
Significant Type I error (probability α) Correct detection (probability = power 1 − β)
Not significant Correct Type II error (probability β)
  • α (usually 5%) is a value the researcher decides
  • β is a value determined by the sample size, the size of the difference, and the variability. Usually the sample size is chosen so that β is 20% or less (power 80% or more), or 10% or less (90% or more)

1.5 Increasing the sample size

We see how power changes as we vary the sample size per group.

How many people per group are needed for 80% power? If you specify power instead of n, you can back-calculate the sample size.

63.8 people per group, rounded up to 64 (128 in the two groups). Let us check by simulation.

It came out at about 80%.

1.6 The relationship among five values

Sample size calculation is a relationship among the following five values. Fixing four determines the remaining one.

Value Meaning This example Making it larger, the required sample size
Significance level α Probability of being significant when there is no difference 0.05 (two-sided) decreases (but false positives increase)
Power 1 − β Probability of being significant when there is a difference 0.80 increases
Difference Δ Size of the difference you want to detect 5 mmHg decreases (acts as a square)
Standard deviation σ Individual variability 10 mmHg increases (acts as a square)
Sample size n Number per group 64 —
  • Raising power from 80% to 90% multiplies the sample size by about 1.3 (64 → 86)
  • Halving the difference or doubling the standard deviation multiplies it by about 4 (64 → 253)

What determines the sample size is the ratio of Δ to σ (Δ / σ, the standardized difference). Δ = 5, σ = 10 and Δ = 0.5, σ = 1 give the same sample size.

ImportantWhat drives the sample size is the square of “difference ÷ variability”

The required sample size is proportional to (σ / Δ)². A small shift in the expected difference or variability changes the required sample size a lot. How to estimate the difference and the standard deviation (Chapters 2 and 4) is at the heart of sample size design.

1.7 Try it

NotePractice
  1. Obtain the power with 40 people per group using both power.t.test() and simulation (sim_power_t())
  2. If the true difference were 3 mmHg instead of 5 mmHg, what would the power of a study with 64 people per group be?
  1. About 60% (0.598 with power.t.test()). The simulation also gives around 60%
  2. About 39%. A study planned with 64 people assuming 5 mmHg misses the difference with a probability of about 60% if the true difference is 3 mmHg
ImportantSummary of this chapter
  • Power = the probability of being significant when there truly is a difference. Usually set to 80% or 90%
  • Significance level α = the probability of being significant when there truly is no difference. Usually 5% (two-sided)
  • Of the five values — α, power, difference, standard deviation, and sample size — fixing four determines the rest
  • The required sample size is proportional to (standard deviation ÷ difference)². Half the difference means 4 times the sample size
  • Power can be checked by simulation (running the same study thousands of times)