7  Interim analyses

In long clinical trials, you may look at the data partway through and stop early if the effect is clear (switching participants to the effective treatment sooner). This is called an interim analysis. But testing every time you look at the data is also a multiplicity problem.

7.1 Looking at the data many times

Consider a trial that tests each time 1/5, 2/5, …, 5/5 of the planned sample has accrued, and stops at that point declaring “effective” as soon as p < 0.05. If there is truly no effect, how likely is it to be declared “effective”?

To keep the computation fast, we generate the test statistic (z-value) of the accumulated data directly. The z-value at each time point is computed from all data collected up to then, so they are strongly correlated with each other.

Looking at the data 5 times, about 14% of trials are declared “effective” even though there is no effect. Because of the correlation, it is not as bad as 5 independent tests (23%), but it is well above 5%.

Even without an effect, the z-value wanders up and down during the trial. With the rule “stop if it ever crosses ±1.96 along the way”, false positives increase.

7.2 Making the boundaries stricter

To keep the FWER (the overall type I error) at 5% even with interim analyses, the boundary at each time point is made stricter than ±1.96. There are two representative methods.

  • Pocock boundary: the same boundary at every time point
  • O’Brien–Fleming boundary: very strict at first, gradually relaxed. The boundary at the final analysis is close to 1.96

Both are chosen so that “the probability of crossing the boundary at some point among the 5” is 5%. This can be computed with the multivariate normal distribution from the correlation of the z-values across time points (\(\sqrt{i/j}\) between times i and j).

Method p-value threshold at each time point Characteristics
Pocock 0.016 at all Easier to stop early, but the final analysis is also strict at 0.016
O’Brien–Fleming 0.000005 → 0.0013 → 0.008 → 0.023 → 0.041 Early on, it will not stop without a very large difference. The final analysis is nearly 0.05
  • The naive method (1.96 every time) has a type I error of about 14%
  • Pocock and O’Brien–Fleming both keep it at about 5%
  • When there is an effect, Pocock ends on average at about 62% of the planned sample, but power drops to 83% (because the final-analysis threshold is strict)
  • O’Brien–Fleming hardly loses any power (89%) and ends on average at about 74% of the planned sample
TipWhy O’Brien–Fleming-type boundaries are widely used
  • The final-analysis threshold is close to 0.05, so interim analyses hardly change the power or required sample size
  • It stops early only when the effect is very large. With few participants, you are not thrown around by large differences arising by chance
  • In real trials, the looks at the data do not happen exactly as planned, so boundaries matching the actual times are computed with an α-spending function (Lan–DeMets) (R packages gsDesign and rpact)

7.3 Trials stopped early overestimate the effect

Let us look at the effect estimates of trials that stopped early under the O’Brien–Fleming boundary, expressed as a ratio to the true effect (true effect = 1).

Trials that stopped at 2/5 have estimates that are on average about 1.8 times the true effect (the 5/5 row also includes trials that never crossed the boundary, so it is, conversely, on the small side). This is because crossing the boundary early happens when a large difference happens to appear by chance (the same mechanism as the “winner’s curse” in Chapter 4 of the Sample size hands-on, in Japanese).

WarningCautions for interim analyses
  • Write the number and timing of interim analyses and the boundaries (α-spending function) into the protocol before the trial starts
  • Only the independent data monitoring committee sees interim results. If those running the trial learn the results, the conduct of the trial changes
  • The effect in trials stopped early tends to be overestimated. Interpret results of trials stopped with few participants especially cautiously
  • Interim analyses for “futility” (stopping because an effect seems unlikely) do not increase the type I error, but they lower power

7.4 Try it

NotePractice
  1. If the number of looks at the data is increased from 5 to 10, what happens to the type I error of the naive method (p < 0.05 every time)?
  2. With only one interim analysis (at half the planned sample, K = 2), compute the O’Brien–Fleming boundaries
  1. It increases to about 19%. The more often you look at the data, the more false positives keep increasing (if you kept looking forever, it would eventually always become significant)
  2. The boundaries are 2.797 (interim) and 1.977 (final), and the p-value thresholds are 0.0052 and 0.048. With one interim analysis, the final analysis can be tested at nearly 0.05
ImportantChapter summary
  • Stopping at p < 0.05 every time you look at the data increases the type I error (about 14% with 5 looks)
  • Pocock (the same boundary) and O’Brien–Fleming (strict at first, close to 0.05 at the end) keep the overall level at 5%
  • O’Brien–Fleming-type boundaries hardly lose power and reduce the average sample size. In practice they are computed with an α-spending function
  • The effect in trials stopped early tends to be overestimated