2  Difference in means

In Chapter 1, we used power.t.test() to find that “with a difference of 5 mmHg, a standard deviation of 10 mmHg, and 80% power, 64 people per group are needed”. In this chapter, we check what is inside with the formula and consider, using real data, what value to enter for the standard deviation.

2.1 Deriving the formula

The difference in the means of two groups, \(\bar{y}_1 - \bar{y}_0\), follows a normal distribution when the sample size is large enough, and its standard error is

\[ SE = \sigma \sqrt{\frac{1}{n} + \frac{1}{n}} = \sigma \sqrt{\frac{2}{n}} \]

(n people per group, standard deviation σ).

  • In the world with no difference, the difference scatters around 0. It is significant at two-sided 5% when the difference exceeds \(1.96 \times SE\)
  • In the world where the true difference is Δ, the difference scatters around Δ. We want 80% of it to exceed \(1.96 \times SE\)

The shaded area (the proportion exceeding the threshold in the world where the true difference is 5) is the power. The threshold lies “\(1.96 \times SE\) above 0” and also “\(0.84 \times SE\) below Δ” (0.84 is the lower 20% point of the normal distribution), so

\[ \Delta = (1.96 + 0.84) \times \sigma \sqrt{\frac{2}{n}} \quad\Longrightarrow\quad n = \frac{2 (z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2}{\Delta^2} \]

The formula gives 62.8 people per group, about 1 fewer than the 63.8 from power.t.test(). The formula uses the normal distribution, but the actual t-test also estimates the standard deviation from the data, so it uses the t-distribution and needs slightly more people. The smaller the sample, the larger this difference.

TipWhat the formula tells us
  • The sample size is proportional to (z_α + z_β)²: for 80% power, (1.96 + 0.84)² ≈ 7.85; for 90%, (1.96 + 1.28)² ≈ 10.5. So going to 90% multiplies it by about 1.34
  • The sample size is proportional to σ² / Δ²: half the difference means 4 times the sample size
  • The total for both groups is twice this

2.2 Where do you get the standard deviation from?

In actual planning, σ is estimated from previous studies, pilot studies, or existing data. Treating NHANES participants aged 40–69 as “existing data”, let us look at the standard deviation of systolic blood pressure.

The standard deviation is 18.0 mmHg, much larger than the 10 mmHg we tentatively assumed in Chapter 1. Let us recalculate with this value.

205 people per group — more than 3 times the 64 with σ = 10.

2.2.1 The standard deviation changes with the eligibility criteria

For an antihypertensive trial, participants would be limited to people with high blood pressure. Looking only at people with systolic blood pressure of 140 mmHg or more:

The standard deviation is 14.1 mmHg, and the required sample size is 127 per group. Restricting the participants narrows the range of values and makes the standard deviation smaller. It is important to use the standard deviation from a population close to the participants of the planned study.

The standard deviation also varies across age groups, from 15 to 19 mmHg.

WarningThe standard deviation “of what”?
  • The standard deviation to use differs depending on whether the trial endpoint is “blood pressure at 12 weeks” or “the change over 12 weeks”
  • The standard deviation of the change is \(\sigma\sqrt{2(1-\rho)}\), depending on the correlation ρ between before and after, and if ρ is greater than 0.5, it is smaller than the standard deviation of a single measurement
  • An analysis adjusting for the baseline value (ANCOVA) can reduce the sample size further (Chapter 5)

2.3 When the data are not normal

What if the endpoint has a skewed distribution and you plan to use the Wilcoxon rank-sum test (Mann-Whitney U test)? If the data are normal, the Wilcoxon test has slightly lower power than the t-test.

With the 64 people that give 80% for the t-test, it is about 78%. Increasing the sample size by about 5% (t-test sample size ÷ 0.955 ≈ 67) reaches 80%. Conversely, if the data have a heavy-tailed distribution, the Wilcoxon test has higher power. When you do not know the shape of the distribution, check the power under the assumed distribution with the simulation in Chapter 7.

2.4 Using a one-sided test

With one-sided 5%, it is 51 people per group, fewer than the 64 for two-sided. However, one-sided 5% is the same criterion as “two-sided 10%”; it just makes significance easier. In verifying superiority in clinical trials, two-sided 5% (or one-sided 2.5%) is usually used. One-sided tests are used when only one direction is asked from the outset, as in non-inferiority trials (Chapter 6).

2.5 Try it

NotePractice
  1. For NHANES participants aged 40–69, find the standard deviation of total cholesterol (TotChol, mmol/L), and calculate the number per group needed to detect a difference of 0.3 mmol/L with 90% power
  2. The difference between the sample size from the formula (normal distribution) and from power.t.test() is larger when the sample size is small. Check this with Δ = 10, σ = 10 (standardized difference 1.0)
  1. The standard deviation is about 1.09 mmol/L, and the required sample size is 277 per group
  2. 15.7 with the formula, 16.7 with power.t.test(). Both differ by about 1 person, but because the sample size is small, this is a difference of about 6% in relative terms
ImportantSummary of this chapter
  • Formula: \(n = 2(z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2 / \Delta^2\) (per group). power.t.test() uses the t-distribution and so gives slightly more
  • Estimate the standard deviation from a population close to the participants of the planned study. It changes a lot with eligibility criteria and age
  • The standard deviation to use differs depending on whether the endpoint is a “value” or a “change”
  • If you use the Wilcoxon test, add about 5% for normally distributed data
  • A one-sided test is not a tool for reducing the sample size