8  When to adjust, and how to report

So far we have looked at various methods for adjusting for multiplicity. Finally, we consider “when you should adjust and when you need not”, and how to write up the results.

8.1 Deciding the “family”

Adjustment is done within a “collection of tests addressing the same question” (a family). What counts as a family is decided not by statistics but by the research question.

Setting Family Adjustment
One primary endpoint That one Not needed
Claiming efficacy “if any one of A, B, C is significant” A, B, C Needed (Chapters 2, 5)
Efficacy if all of A, B, C are significant (co-primary) ― Not needed (power drops) (Chapter 5)
Comparing all pairs of 4 groups to find which pairs differ 6 comparisons Needed (Chapter 4)
Searching for candidates among 1000 markers 1000 markers With the FDR (Chapter 3)
Separate questions (efficacy and safety, separate hypotheses of a paper) Each separate No adjustment across families
Exploratory analyses / secondary endpoints (not used for claims) ― Often not adjusted, but label them exploratory and give the number of tests
NoteOver-adjustment is also a problem

“Adjusting every p-value in the paper with Bonferroni” lumps separate questions together and lowers power more than necessary. The purpose of adjustment is to “control the probability that the conclusion you want to claim is due to chance”. Decide in advance the collection of tests used for your claims, and adjust within it.

8.2 Adjusted p-values and nominal p-values

Here is an example that shows the results for secondary endpoints with both unadjusted nominal p-values and Holm-adjusted p-values.

  • With nominal p-values, three appear significant (p < 0.05), but after Holm adjustment only systolic blood pressure is significant
  • Showing the differences and 95% confidence intervals together lets readers see how large an effect each endpoint is likely to have. It is important not to judge solely by whether p-values are significant or not

8.3 Example write-up

TipExample Methods text

The primary endpoint was the change in systolic blood pressure at 12 weeks, tested at a two-sided significance level of 5%. Among the secondary endpoints, diastolic blood pressure, LDL cholesterol, and HbA1c were tested, only if the primary endpoint was significant, in this order using a fixed-sequence procedure, each at 5%. The other secondary endpoints and the analyses of four pre-specified subgroups (sex, age 65 years or older, diabetes, reduced kidney function) were considered exploratory, and nominal p-values and 95% confidence intervals are presented (no adjustment for multiplicity was made).

The key points of this write-up are:

  1. The tests used for claims (primary endpoint → the three in fixed sequence) and their method of adjustment are clear
  2. For exploratory analyses, their number and the fact that they were not adjusted are stated
  3. Subgroups are limited to pre-specified ones, and their number is given

8.4 Checklist for reporting multiplicity

ImportantWhat to include in the report
  1. The primary endpoint and the family of tests used for claims
  2. The adjustment method (Holm, fixed sequence, graphical approach, Tukey, BH, etc.) and the fact that it was decided in advance
  3. Adjusted p-values or adjusted significance levels; if possible, adjusted (simultaneous) confidence intervals
  4. The number of all tests performed (endpoints, group comparisons, subgroups, sensitivity analyses)
  5. Label exploratory analyses as exploratory, and indicate that their p-values are nominal
  6. The number, timing, and boundaries (α-spending function) of interim analyses, and whether the trial was stopped early
  7. Deviations from the protocol, statistical analysis plan, and trial registration, and the reasons for them

8.5 Approaches other than adjustment

Dealing with multiplicity is not only about adjusting p-values.

  • Report primarily effect sizes and confidence intervals, and do not rely too heavily on the 0.05 threshold for p-values
  • Reproducibility: treat findings discovered in exploratory analyses as hypotheses until confirmed in other data
  • Hierarchical Bayes / shrinkage estimation: there are also methods that soften extreme chance results by estimating the effects of many subgroups or endpoints “shrunk” toward the overall mean

8.6 Try it

NotePractice
  1. If the six secondary endpoints in the table above are adjusted with the BH procedure (FDR), how many are significant? Is it appropriate to use the FDR in this setting?
  2. Choose a paper in your own field and check how much of the checklist above it reports regarding the primary endpoint, the number of tests performed, and the adjustment method
  1. After BH adjustment, only systolic blood pressure (0.003) is below 0.05 (diastolic blood pressure and LDL are 0.08). However, if you want to claim an effect on secondary endpoints of a clinical trial, a method that controls the FWER rather than the FDR (fixed sequence, Holm, etc.) is required. The FDR suits exploratory settings for finding candidates (Chapter 3)
  2. Many papers state the primary endpoint but not the total number of tests performed or whether the subgroups were pre-specified
ImportantChapter summary
  • Adjust within the “collection (family) of tests used for claims”. The family is determined by the research question
  • Lumping separate questions together and over-adjusting loses more power than necessary
  • For exploratory analyses, instead of adjusting, label them exploratory and give the number of tests
  • Show effect sizes and confidence intervals, not only adjusted p-values
  • Write the adjustment method and family into the protocol before looking at the data