5  Multiplicity in clinical trials: multiple endpoints

In clinical trials, the principle is to fix a single primary endpoint and judge efficacy by that test alone. With a single primary endpoint, no multiplicity adjustment is needed. In practice, however,

This chapter compares methods that keep the FWER at 5% in such settings, using a trial with two endpoints, A and B.

5.1 Setup

In a two-group comparison with 100 people per group, endpoints A and B (correlation 0.5) are tested with t-tests. We vary the true effect (standardized difference) by scenario.

For each scenario, we compare “the probability that A is significant”, “the probability that B is significant”, and “the probability that an endpoint with truly no effect is significant (FWER)”.

5.2 Scenario 1: no effect on either

  • If, without adjustment, you declare efficacy “when either A or B is significant”, the FWER is about 7% (a little lower than the 9.75% for independent endpoints, because the two endpoints are correlated)
  • Fixed sequence, Holm, and weighted Holm are all at or below 5%

5.3 Scenario 2: an effect on A only

  • Fixed sequence: A can be tested at an unadjusted 5%, so A’s power is the same as unadjusted, about 82%. B has no effect, so it is significant about 3% of the time
  • Holm: A ends up tested at 2.5%, so A’s power drops to about 74%
  • Weighted Holm: 4% was allocated to A, so A’s power is about 80%

5.4 Scenario 3: an effect on B only

  • Fixed sequence cannot test B unless A is significant, so it almost always misses the effect on B (B is significant only about 3% of the time)
  • Holm also tests B at 2.5%, so it detects it about 73% of the time
  • Weighted Holm allocates only 1% to B, so it is about 61%

5.5 Scenario 4: effects on both A and B

Every method has a chance to detect both A and B. Fixed sequence keeps A’s power, and when A is significant, it can also test B at 5%.

5.6 Summary of methods

Method Procedure When it suits
Fixed sequence (hierarchical testing) Test A at 5% → if significant, test B at 5% → if significant, test C… stop at the first non-significant one There is a clear order of importance or of confidence in the effect among endpoints (primary → secondary)
Holm Split the significance level equally; when one is significant, pass its level on to the rest The endpoints are about equally important
Weighted Holm, graphical approaches Split the significance level with weights, and pass the level of a significant endpoint on to the others in pre-specified proportions Importance differs, but you do not want to miss the second and later endpoints either
TipThe idea behind the fixed sequence

Fixed-sequence (hierarchical) testing lets every endpoint be tested at 5% by deciding before seeing the data the rule “if A is not significant, B and everything after it are declared non-significant”. If you get the order wrong (putting an endpoint whose effect is uncertain first), you lose all the endpoints behind it. Order them starting from those with large, reliable effects and those that are clinically important.

5.7 When “both” must be significant: co-primary endpoints

Sometimes all primary endpoints are required to be significant (co-primary endpoints), as when a drug for Alzheimer’s disease must show improvement in both “cognitive function” and “activities of daily living”. In this case, one endpoint becoming significant by chance does not establish efficacy, so no adjustment of the significance level is needed. Instead, power goes down.

The power for each of A and B is about 82%, but both are significant only about 71% of the time. Co-primary trials are planned with more participants so that “the probability that both are significant” reaches the target power (80%, etc.) (see the simulation in Chapter 7 of the Sample size hands-on, in Japanese).

ImportantWhat matters for multiplicity in clinical trials
  • In principle, there is one primary endpoint. If there are several, decide whether it is “any one” or “all”
  • If you want to claim effects on secondary endpoints, write a fixed-sequence procedure, graphical approach, etc. into the protocol and statistical analysis plan before the trial starts
  • Decide the adjustment method before seeing the data. Choosing a convenient method after seeing the results does not protect the FWER
  • Report secondary endpoints and exploratory analyses that are not covered by the adjustment as nominal p-values, stating this explicitly

5.8 Try it

NotePractice
  1. If the correlation between endpoints is changed from 0.5 to 0.9, how does the “unadjusted” FWER in Scenario 1 change? (Change rho = 0.5 inside run_scenario())
  2. Confirm in Scenario 2 that weighted Holm with weights (0.5, 0.5) is the same as Holm
  1. It drops to about 6%. When the endpoints are strongly correlated, the two tests come close to “nearly the same thing twice”. Even so, it exceeds 5%
  2. Weighted Holm with weights (0.5, 0.5) gives the same results as Holm. Weighted Holm is a generalization of Holm
ImportantChapter summary
  • With a single primary endpoint, no adjustment is needed. If efficacy is declared “when any one is significant”, adjustment is needed
  • Fixed sequence: test in order at 5% and stop at the first non-significant one. Powerful if the order is right; if it is wrong, you lose everything behind
  • Holm, weighted Holm (graphical approaches): split the significance level and pass it on to the rest when one is significant
  • With co-primary endpoints (all must be significant), no adjustment is needed, but power drops, so increase the sample size
  • Write the method into the protocol before seeing the data