9 Exercises and a checklist for sample size design
Let us put the contents of the previous chapters to use with your own hands. The code cells in this chapter do not run automatically. Fill in the blanks (___) and press “Run Code”. Example answers are folded under each question.
9.1 Checklist for sample size design
| # | What to check | Chapter |
|---|---|---|
| 1 | Is there a single primary endpoint? If there are several, how will the significance level be adjusted? | 8 |
| 2 | Is the question one of superiority, non-inferiority, equivalence, or estimation? | 6, 8 |
| 3 | What is the test / model of the main analysis? (t-test, ANCOVA, logistic regression, log-rank test, …) | 2, 3, 5, 7 |
| 4 | Is the anticipated difference the minimal clinically important difference? Are you not relying on rules of thumb for standardized effect sizes? | 4 |
| 5 | If you use the effect from previous studies, did you consider the possibility of overestimation? | 4 |
| 6 | Is the source of the standard deviation, proportion, or event rate close to the participants of the planned study? | 2, 4 |
| 7 | If the standard deviation comes from a small pilot study, did you use the upper limit or do a sensitivity analysis? | 4 |
| 8 | For proportions, did you check that the reference proportion changes the sample size a lot? If incidence is low, did you think in terms of the number of events? | 3, 7 |
| 9 | If you measure baseline, did you plan with ANCOVA? Is the correlation ρ conservative? | 5 |
| 10 | Did you adjust for allocation ratio, dropout (by division), and clusters (design effect)? | 5 |
| 11 | Did you check by simulation the power with the test used in the analysis (Yates, Fisher, adjustment, etc.)? | 3, 7 |
| 12 | Did you prepare a sensitivity analysis table varying Δ and σ? | 4 |
| 13 | Did you document values, sources, and software so that a third party can reproduce it? | 8 |
9.2 Exercise 1: a trial of a weight-loss program
You are planning a trial comparing a weight-loss program with usual advice in people aged 30–59 with BMI 25 or more. The primary endpoint is BMI at 6 months, and the minimal clinically important difference is 1 kg/m².
9.2.1 1-1. Estimate the standard deviation and find the sample size
From NHANES, find the standard deviation of BMI in a population close to the participants, and calculate the number per group needed at two-sided 5% and 80% power.
s <- sd(target$BMI)
n <- ceiling(power.t.test(delta = 1, sd = s, power = 0.80)$n)The standard deviation is about 6.4 kg/m², and the required sample size is 652 per group.
9.2.2 1-2. Dropout and baseline
You expect 20% to drop out over 6 months. Find the sample size allowing for dropout. Next, with ANCOVA adjusting for baseline BMI (correlation between baseline and 6 months ρ = 0.8), what is the sample size before allowing for dropout?
ceiling(n / (1 - 0.2))
ceiling(power.t.test(delta = 1, sd = s * sqrt(1 - rho^2), power = 0.80)$n)Allowing for dropout gives 815 per group. With ANCOVA, it drops to 236 per group before dropout (295 allowing for dropout). For values that are stable within a person, like BMI, the benefit of adjusting for baseline is large.
9.3 Exercise 2: a trial of a smoking cessation program
You expect the 6-month quit rate to be 10% with conventional advice and 18% with the new program.
9.3.1 2-1. Calculate the sample size and check it by simulation
Find the number per group needed at two-sided 5% and 90% power, and check it by simulation with the uncorrected χ² test.
n2 <- ceiling(power.prop.test(p1 = 0.10, p2 = 0.18, power = 0.90)$n)
prop.test(c(x, y), c(n2, n2), correct = FALSE)$p.value < 0.05394 per group. The simulation also gives about 91%.
9.3.2 2-2. Allocating by school
It was decided to allocate the same intervention by high school (25 people per school, ICC = 0.02). If the BMI trial of Exercise 1 were also run by school, what would the design effect and the number of schools needed per group be (based on the 652 people of Exercise 1)?
de <- 1 + (25 - 1) * 0.02The design effect is 1.48, and 39 schools (975 people) per group are needed. Even with a small ICC of 0.02, the sample size increases about 1.5 times.
9.4 Exercise 3: a non-inferiority trial
You expect the cure rate of a new antibiotic to be 90%, the same as the conventional drug, with a non-inferiority margin of 5 points. Find the number per group needed at one-sided 2.5% and 80% power.
p <- 0.90; M <- 0.05
ceiling((qnorm(0.975) + qnorm(0.80))^2 * (p * (1 - p) * 2) / M^2)566 per group.
9.5 Exercise 4: your own study
For a study you are planning (or one from a paper you have read), enter the following values, calculate the sample size, and make a sensitivity analysis table varying Δ and σ.
9.6 Review questions
- What is it that happens with 80% probability when we say “80% power”?
- If the anticipated standard deviation doubles, by what factor does the required sample size increase?
- Allowing for a 25% dropout rate, what should a required sample size of 100 become?
- What is the problem with reporting “post-hoc power was 35%” in a study that found no significant difference?
- What is the problem with using, as is, the effect size that was significant in a small previous study with 30 people per group to plan the next study?
- Why can a sample size planned for the main analysis not necessarily detect differences in effect between subgroups (interaction)?
- That the study result is significant when there truly is a difference of the anticipated size (Chapter 1)
- 4 times. The sample size is proportional to σ² (Chapter 1)
- 134 people (100 ÷ 0.75 = 133.3, rounded up). 100 × 1.25 = 125 is not enough (Chapter 5)
- Post-hoc power calculated from the observed effect is determined by the p-value alone and is always below 50% if not significant. It provides no new information; show the effect estimate and confidence interval instead (Chapter 8)
- A small study becomes significant only when the effect happens to come out large, so reported effects tend to be overestimated (the winner’s curse). Planning with that effect leads to too few people and insufficient power (Chapter 4)
- Because interaction is “the difference of differences between subgroups”, it has large variability, and detecting an effect of the same size requires about 4 times as many people (Chapter 7)