7 When there is no formula: obtaining power by simulation
power.t.test() and power.prop.test() are functions for simple two-group comparisons. Actual analysis plans are almost always more complex: logistic regression adjusted for covariates, survival analysis, tests of interaction, and so on.
What you can use in such cases is simulation, which we have used many times in the previous chapters. Even without a formula, you can obtain the power of any analysis.
7.1 The four steps of simulation
- Decide the data-generating mechanism: write the anticipated effect, variability, covariate distributions, dropout, etc. in R code
- Analyze as in the analysis plan: use the model and test you plan to write in the protocol, exactly as is
- Repeat 1–2 thousands of times
- The proportion that is significant is the power. Vary the sample size and find the smallest one exceeding 80%
Power obtained by simulation also has error. With 2000 repetitions, it fluctuates by about ±1.8 points (95% range) around 80% power. When deciding the final sample size, use 5000–10000 repetitions or allow some margin.
7.2 Example 1: logistic regression adjusted for age
We compare events within 5 years (yes/no) between the new and the conventional treatment. The analysis plan is “logistic regression adjusted for age”.
Assumptions:
- With the conventional treatment, the event rate for a person of average age is 20%
- The odds ratio for the new treatment is 0.6
- The odds double for each 1-standard-deviation increase in age
Adjusting for age gives about 3 points higher power than not adjusting, reaching about 81% with 450 people per group (without adjustment, around 500 would be needed).
In logistic regression, the odds ratio is a ratio “among people of the same age”. In a comparison not adjusted for age, the event proportions are about 22% and 15%, and the effect appears somewhat diluted (non-collapsibility of the odds ratio). If you plan with an adjusted model, simulating with the adjusted model is the reliable way.
In linear regression (ANCOVA in Chapter 5), adjusting for covariates related to the endpoint reduced the error and greatly increased power. In logistic regression, the gain in power from adjustment is smaller. Even so, it is important to check power with the planned analysis.
7.3 Example 2: survival analysis
We compare the time until an event occurs using the log-rank test. The hazard ratio for the new treatment is anticipated to be 0.75.
7.3.1 Required number of events
The power of the log-rank test is determined mainly by the number of events that occurred (Chapter 3). By Schoenfeld’s formula, the number of events needed with 1:1 allocation is
\[ d = \frac{4 (z_{1-\alpha/2} + z_{1-\beta})^2}{(\log HR)^2} \]
380 events are needed.
7.3.2 How many people to recruit
To convert the number of events into a number of people, you need the probability that an event occurs during follow-up. Suppose the median survival time with the conventional treatment is 5 years (exponential distribution), and everyone is followed for 3 years.
The probabilities of an event within 3 years are 34% and 27%, so 624 people per group (1248 in the two groups) are needed. Let us check by simulation.
About 79%, almost as planned. In an actual trial, you would add things like gradual enrollment of participants (so follow-up differs between people) and dropout to the simulation.
7.4 Example 3: does the effect differ by subgroup? (interaction)
Suppose you want to investigate whether “the new treatment works only in women”. How much power does a test of interaction have?
- Half men, half women; a difference of 5 mmHg in women and 0 in men (interaction size 5 mmHg); standard deviation 10 mmHg
If everyone had a 5 mmHg effect, 64 people per group would give 80% power, but detecting an interaction of the same size requires about 4 times as many, 256 people.
This is because the interaction is “the difference in women − the difference in men”, each subgroup has half the people, and so the variance of the difference of differences is 4 times the variance of the overall difference.
With the sample size planned for the main analysis, the power to find an interaction (a difference in effect between subgroups) is usually low. “No significant difference in a subgroup” does not mean “it does not work in that subgroup”. The more subgroups there are, the more likely chance significant differences become (multiple testing hands-on, Chapter 6).
7.5 Try it
- In Example 2, if the hazard ratio is 0.80, what are the required number of events and the number of people per group?
- In Example 1, if the odds ratio for age is changed from 2 to 3 (age strongly related to events), how does the difference in power between adjusted and unadjusted analyses change? (450 people per group)
- 631 events, 1012 people per group. Just moving the hazard ratio from 0.75 to 0.80 multiplies the number of events by about 1.7
- About 75% unadjusted and about 79% adjusted. When the effect of age is strong, both powers drop (events are largely determined by age, burying the treatment effect), and the gain from adjustment is about 4 points
- Even without a formula, power can be obtained by generate data → analyze as planned → repeat → proportion significant
- The error with 2000 repetitions is about ±1.8 points. Increase the number for the final decision
- Think of survival analysis in terms of the number of events (Schoenfeld’s formula). Convert to people using the probability of an event
- Detecting an interaction of the same size requires about 4 times as many people