8  Sensitivity analysis for MNAR and reporting

Multiple imputation assumes that missingness is MAR (the missing-data mechanism can be explained by the variables in the imputation model). But as seen in Chapter 1, MAR and MNAR cannot be distinguished from the data. So we perform a sensitivity analysis to check “how much would the conclusions change if the data were MNAR?”

8.1 Example: a clinical trial of an antihypertensive drug

Consider a trial comparing a new drug with a standard drug (200 people per group). The primary endpoint is systolic blood pressure at 12 weeks, analysed by regression adjusted for baseline blood pressure (ANCOVA). Blood pressure at 12 weeks is more likely to be missing for people with higher baseline blood pressure, and about 1/4 is missing overall.

8.1.1 Multiple imputation assuming MAR

The effect of the new drug is −5.7 mmHg (95% confidence interval −8.6 to −2.9), which is significant. However, this result rests on the assumption (MAR) that “the 12-week blood pressure of those who dropped out is distributed like that of those who did not drop out, in the same group and with the same baseline”.

8.2 The delta-adjustment method

People who dropped out of the new-drug group may have done so because of side effects or insufficient efficacy. If so, their 12-week blood pressure would have been higher than the values imputed under MAR. So we add δ mmHg to the values imputed under MAR (only for dropouts in the new-drug group) and examine how the results change.

In mice, the post argument lets you modify values right after they are imputed.

  • The larger δ, the smaller the effect of the new drug
  • At δ = 13 mmHg the confidence interval crosses 0, and the result is no longer significant. This δ is called the tipping point

To overturn the conclusion, “the blood pressure of dropouts in the new-drug group would need to be 13 mmHg higher on average than predicted under MAR”. Whether this is clinically plausible (would dropouts alone be 13 mmHg higher when the drug’s effect itself is about 5 mmHg?) is judged by subject-matter experts. If it is implausible, the conclusion can be said to be robust to MNAR.

TipKey points of delta adjustment and tipping-point analysis
  • δ is an easily interpreted parameter expressing “how far things depart from MAR”
  • Choose the group and direction for δ that work against the conclusion (in this example, making the new-drug group worse)
  • Separate δ values can be added to the two groups to make a two-dimensional table
  • Other methods assume that dropouts “follow the same course as the standard-drug group” (jump to reference, copy reference), and are chosen together with the trial’s estimand (ICH E9(R1))

8.3 With data whose true values are known

Under the MNAR mechanism of Chapter 1 (the higher the blood pressure, the more missing), bias remains even with multiple imputation. What value of δ brings us close to the true mean?

Multiple imputation assuming MAR (δ = 0) is less biased than complete case, but does not reach the pre-missingness mean (it improves only to the extent that age, BMI, and sex explain part of blood pressure). In this example, δ around 15 mmHg brings it close to the pre-missingness mean. In real data the true values are unknown, so the aim is to show whether the conclusions change over a plausible range of δ.

8.4 What to include in reports

ImportantReporting missing data (based on Sterne et al., BMJ 2009, and others)
  1. The number and proportion missing for each variable, and the number of complete cases
  2. The reasons for missingness (as far as known), and a comparison of the characteristics of those missing and those observed
  3. The missing-data mechanism assumed in the primary analysis (MAR, etc.) and its justification
  4. For multiple imputation:
    • The software and method used (mice, pmm, etc.)
    • The variables included in the imputation model (outcome, auxiliary variables, interactions)
    • m, the number of iterations, and the random seed
    • That convergence and imputed values were checked
  5. Comparison with the results of complete-case analysis
  6. Sensitivity analyses for MNAR (delta adjustment, tipping point, etc.) and their results

8.5 Try it

NotePractice
  1. In the trial example, what happens to the effect of the new drug if the same δ = 10 is added to dropouts in both groups?
  2. Also analyse the trial example with complete-case analysis (lm(week12 ~ group + base, data = trial)) and compare with MAR multiple imputation
  1. The effect of the new drug hardly changes (making both groups worse by the same amount does not change the difference between groups; it changes only slightly because the dropout proportions differ between groups). In sensitivity analyses, it is important to consider δ values that differ between groups
  2. Complete-case analysis gives −5.9 mmHg, almost the same as multiple imputation (−5.7). Because missingness is determined by baseline (an explanatory variable in the model), complete-case analysis is also unbiased (Chapter 2)
ImportantChapter summary
  • Multiple imputation assumes MAR. Prepare for the possibility of MNAR with sensitivity analyses
  • Delta adjustment: add δ to the imputed values and see how the conclusions change. It can be written with mice’s post argument
  • Tipping point: the δ at which the conclusion changes. Experts judge whether it is clinically plausible
  • Reports should include the amount of and reasons for missingness, the assumptions, the imputation model, m, the comparison with complete-case analysis, and sensitivity analyses