8 Post-hoc power, precision, and how to report
Finally, we cover the misuse of “power” after a study is over, how to decide the sample size from the precision of estimation rather than testing, and how to write it in protocols and papers.
8.1 Do not calculate “post-hoc power”
Papers that found no significant difference sometimes write: “power calculated from the observed difference and standard deviation was 30%, so the study was underpowered and no difference was found”. This post-hoc power using the observed effect (observed power) is meaningless.
This is because power calculated from the observed difference is determined by the p-value alone.
- If p = 0.05 exactly, the post-hoc power is always about 50%
- If p > 0.05 (not significant), the post-hoc power is always below 50%
In other words, “not significant → post-hoc power was low” just restates the same thing and provides no new information. Changing the sample size (here 64 per group) hardly changes this relationship.
- The effect estimate and its confidence interval. Whether the interval includes “a clinically meaningful difference” or not
- The power at the planning stage (power for the difference anticipated in advance), and how the actual sample size and standard deviation differed from the plan
8.2 Deciding the sample size from precision
Some studies aim at estimation rather than testing, such as prevalence surveys or evaluations of measurement performance. In that case, decide the sample size from “how wide you want the confidence interval to be”.
8.2.1 Estimating a proportion
Suppose you want to estimate the prevalence of hypertension in a region with a precision of ±3 percentage points (half the width of the 95% confidence interval).
\[ n = \frac{z_{1-\alpha/2}^2 \, p(1-p)}{E^2} \]
When you have no idea of the expected prevalence, calculate with p = 0.5, which gives the largest sample size (1068 people).
8.2.2 Estimating a mean
If you want to estimate the mean systolic blood pressure with a precision of ±2 mmHg, with a standard deviation of 18 mmHg (Chapter 2):
312 people.
Precision-based calculations do not include power (1 − β). The confidence interval width obtained with the calculated sample size only meets the target on average, and in about half of studies it will be wider than the target. If you want to be sure of staying within the target width, use the upper estimate of the standard deviation (Chapter 4), add a few people, and so on.
8.3 When the sample size is fixed
In rare diseases or studies using existing cohorts, the sample size may be fixed in advance. In that case, calculate and show at the planning stage “the difference detectable with 80% power at this sample size”. Unlike post-hoc power, this is a meaningful calculation.
With 30 people per group, what can be detected with 80% power is a difference of 7.4 mmHg or more. If this is larger than the clinically meaningful difference, reconsider the purpose of the study (position it as exploratory, change the endpoint, make it multicentre, …).
8.4 How to write it in protocols and papers
Write the justification for the sample size so that a third party can reproduce the same calculation (CONSORT statement item 7a, SPIRIT statement item 14).
- The primary endpoint and the test / analysis method to be used
- The anticipated difference (and its rationale: the minimal clinically important difference, or previous studies)
- The variability (standard deviation, control-group proportion, event rate) and its source
- The significance level (two-sided or one-sided) and power
- The calculated sample size, and adjustments for dropout, allocation ratio, clusters, etc.
- The software / functions used (
power.t.test(); for simulation, its conditions and number of repetitions) - For non-inferiority / equivalence, the margin and its rationale
- If there are multiple primary endpoints or interim analyses, the method of adjusting the significance level (multiple testing hands-on)
8.5 Try it
- How many people are needed to estimate the prevalence of a disease expected to be 20% with a precision of ±2 points?
- In a study that can recruit only 50 people per group, with a standard deviation of 14.1 mmHg, what difference can be detected with 80% power?
- 1537 people. Tightening the precision from ±3 to ±2 points multiplies the sample size by (3/2)² = 2.25
- About 8.0 mmHg. Not enough to detect a 5 mmHg difference
- Post-hoc power calculated from the observed effect is a restatement of the p-value and is meaningless. Show the confidence interval
- If estimation is the goal, decide the sample size from the width of the confidence interval. If you have no idea of the proportion, use p = 0.5
- If the sample size is fixed, calculate and show the detectable difference at the planning stage
- In the protocol, write the endpoint, analysis, difference, variability and its source, α, power, adjustments, and software