3  Difference in proportions

When the endpoint is binary, such as “did an event occur?” or “did the patient improve?”, we compare proportions between two groups. Suppose you expect the 5-year incidence of cardiovascular events to be 20% with the conventional treatment and 15% with the new treatment.

3.1 Calculating with power.prop.test()

906 people per group (905.4 rounded up), 1812 in the two groups. For a difference in means we had to set two values, the “difference” and the “standard deviation”, but for proportions, setting the two proportions also determines the variability, because the variance of binary data with proportion p is p(1 − p).

3.1.1 Formula

\[ n = \frac{\left( z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)} \right)^2}{(p_1 - p_2)^2}, \quad \bar{p} = \frac{p_1 + p_2}{2} \]

It matches power.prop.test(). The variability is computed with the common proportion \(\bar{p}\) of the two groups in the world with no difference, and with each group’s own proportion in the world with a difference.

3.1.2 Checking by simulation

With the uncorrected χ² test (the z-test for a difference in proportions), it is about 80%. With Yates’ correction, the default in R’s prop.test(), it drops to about 77%. power.prop.test() assumes the uncorrected test. If you plan to apply the correction in the analysis, increase the sample size accordingly or check by simulation (Yates’ correction is covered in Chapter 5 of the chi-squared test hands-on).

3.2 The same “5-point difference”, but…

Fixing the difference at 5 percentage points, we vary the reference proportion (the proportion with the conventional treatment).

Even for the same 5-point difference, the closer the proportion is to 50%, the more people are needed (1565 for 50% → 45%, 435 for 10% → 5%). This is because the variance p(1 − p) is largest at p = 0.5.

3.3 Thinking in terms of relative reduction

In clinical trials, the effect is often anticipated in relative terms, such as “reduce events by 25%” (risk ratio 0.75). Fixing the risk ratio at 0.75, we vary the reference proportion.

With few events (a reference proportion of 2%), more than 10,000 people per group are needed. However, looking at the number of events that occur, it is about 250–380 in every case and does not change much.

ImportantFor endpoints with low incidence, “number of events” matters more than “number of people”

The power to detect a relative effect is determined mainly by the number of events that occurred. This is why prevention trials for low-incidence diseases involve tens of thousands of people. Efforts are made to increase the number of events, such as extending follow-up or targeting high-risk people (calculating the number of events for survival analysis is covered in Chapter 7).

3.4 When anticipating an odds ratio

Previous studies may report an odds ratio from logistic regression. Since power.prop.test() takes proportions, convert the odds ratio into a proportion.

An odds ratio of 0.7 corresponds to 14.9% when the reference is 20%, requiring 866 people per group. With the same odds ratio of 0.7, if the reference is 50% it becomes 41.2% (a difference of 8.8 points), and the sample size is 499. The odds ratio alone does not determine the sample size; the reference proportion is needed.

3.5 When the sample is small: Fisher’s exact test

When the difference is large, as with adverse event rates of 5% and 20%, the required sample size is small.

76 people per group. But with so few, expected counts become small, and the analysis may end up using Fisher’s exact test. Fisher’s test is conservative (harder to reach significance).

With 76 people, Fisher’s exact test has only about 75% power. About 80% with 81 people, and about 83% with 86. About 10% more people are needed.

TipCheck power with the test you will actually use in the analysis

power.prop.test() gives the power of “the uncorrected χ² test”. If the analysis plan differs — Yates’ correction, Fisher’s exact test, adjustment with logistic regression, etc. — the power differs too. Simulating with the test exactly as in the analysis plan is the reliable way (Chapter 7).

3.6 Try it

NotePractice
  1. Find the number per group needed to compare two groups with response rates of 60% and 70% with 90% power
  2. With a reference proportion of 30% and a risk ratio of 0.8, find the number per group and the number of events that occur
  1. 477 people per group
  2. 859 people per group, about 464 events. A risk ratio of 0.8 is a smaller effect than 0.75, so more events are needed
ImportantSummary of this chapter
  • When comparing proportions, setting the two proportions determines the sample size (variability is p(1 − p))
  • For the same difference, the closer the proportions are to 50%, the more people are needed
  • When anticipating a relative effect (risk ratio), the number of events determines the power. With low incidence, the sample size becomes huge
  • When anticipating an odds ratio, convert it to a proportion together with the reference proportion
  • power.prop.test() is for the uncorrected χ² test. If you use Yates’ correction or Fisher’s exact test, increase the sample size