8 Power, multiple comparisons, and how to report
Finally, we cover using the χ² test in the planning and reporting of a study.
- How many people do you need to find a difference (power and sample size)?
- When examining “which groups differ from which” among 3 or more groups (multiple comparisons)
- How to write up the results
8.1 How many people do you need to find a difference?
Suppose you expect a new treatment to raise the improvement rate from 25% to 35%. How many people per group are needed to test at a 5% significance level (two-sided) with 80% power (an 80% probability of significance when there truly is a difference)?
About 329 people per group (rounded up, 330). Let us check by simulation.
It was significant with a probability of about 80%. power.prop.test() corresponds to the χ² test without correction (= the z-test for a difference in proportions, Chapter 2).
8.1.1 When the difference is small, the required number rises sharply
When the difference halves, the required number becomes about 4 times larger (the number is inversely proportional to the square of the difference).
8.1.2 Power for r×c tables
For 3 or more groups or tables with 3 or more categories, power can be computed with the noncentral χ² distribution. From the assumed proportions of the table, compute Cohen’s effect size w, and use N × w² as the noncentrality parameter.
With 300 people across the 3 groups, power is about 41%; with 600, about 71%; with 900, about 88%. You can also confirm this by simulation (try it).
8.2 Where is the difference among 3 or more groups? Post-hoc multiple comparisons
Among NHANES participants aged 20 or over, compare the proportion with diabetes by race/ethnicity category.
We know there is a difference among the 5 groups (χ² = 78.1, 4 degrees of freedom). Then which groups differ from which?
8.2.1 Comparing every pair of groups
There are 10 ways to choose 2 groups out of 5. If you test each at a 5% significance level, the probability that at least one of the 10 is significant, even if there are no differences anywhere, is much larger than 5%.
About 40% (the 10 actual comparisons are not independent of each other, so the exact value is a little smaller than this).
8.2.2 Simulation: when there is truly no difference
We create data with truly no difference — all 5 groups have a diabetes proportion of 14%, with 300 people per group — and run 10 pairwise tests.
Without correction, about 30% of studies find “a significant difference somewhere”, even though there is truly no difference. Correcting the p-values with Holm’s method keeps this at 5% or below.
8.2.3 Correcting with real data
Even after Holm’s correction, we can say that Black has a higher proportion than all other groups, and White a lower proportion than Black, Hispanic, and Mexican.
- Refine the question in the order overall χ² test → standardized residuals (Chapter 4) → pairwise comparisons (with correction)
- If the comparisons you want are fixed from the start (e.g., only comparing each group with White), test only those and correct for that number
- Correction methods include Bonferroni (p-value × number of comparisons; simple and conservative) and Holm (always more powerful than Bonferroni; use this if unsure)
- Subgroup analyses (testing repeatedly by sex, by age, etc.) have the same multiple-testing problem
8.3 How to write up the results
When reporting the result of a χ² test, write it so that readers can reproduce the table and know which test was used.
- The number and proportion in each group (always give the denominator)
- The effect size, such as a difference or ratio, with its confidence interval
- The name of the test (Pearson’s χ² test / Fisher’s exact test / McNemar’s test / MH test / trend test) and whether a correction was applied (Yates)
- The χ² value, degrees of freedom, and p-value (write the actual p-value, not “p < 0.05”; if very small, p < 0.001)
- If you made multiple comparisons, the correction method
- If you considered confounding, what you stratified or adjusted for (in this example, adjusting for age changes the conclusion; Chapter 7)
When p = 0.30, all you can say is that “a difference could not be detected with this number of people”. If the confidence interval is wide and includes clinically large differences, you cannot say “there is no difference”. If you want to show equivalence, plan it from the start as an equivalence or non-inferiority test.
8.4 Try it
- Using
power.prop.test(), find the number per group needed to detect a difference in improvement rate from 25% to 30% with 90% power - Confirm by simulation the “about 71% with 600 people across 3 groups” computed with
power_chisq()
- 1674 per group (1673.9 rounded up). With a 5-point difference, you need about 5 times as many as with a 10-point difference (about 330)
- It comes out at around 71% (being a simulation, it varies by a few points)
- Use
power.prop.test()for the required number. When the difference halves, the number becomes about 4 times larger - Power for r×c tables can be computed with the effect size w and the noncentral χ² distribution
- If you compare pairs post hoc, correct with Holm or similar. Without correction, false significant differences arise easily
- Report “numbers and proportions, effect size and confidence interval, name of the test and correction, χ²(df), p-value”