9 Exercises and a missing-data checklist
Let us put the contents of the previous chapters to use with your own hands. The code cells in this chapter do not run automatically. Fill in the blanks (___) and press “Run Code”. Example answers are collapsed below each question.
9.1 Missing-data checklist
| # | What to check | Chapter |
|---|---|---|
| 1 | Did you check the proportion missing for each variable and the number of complete cases? | 1, 2 |
| 2 | Did you compare the characteristics of those missing and those observed? Did you investigate the reasons for missingness? | 1 |
| 3 | Did you think about the missing-data mechanism (MCAR, MAR, MNAR) using knowledge rather than data? | 1 |
| 4 | Is the condition for unbiased complete-case analysis (missingness does not depend on the outcome) met? | 2 |
| 5 | Are you avoiding single imputation such as mean imputation, LOCF, or the missing-indicator method? | 3 |
| 6 | Did you pool the multiple-imputation results with Rubin’s rules (pool())? |
4, 5 |
| 7 | Did you include the outcome in the imputation model? | 6 |
| 8 | Did you include auxiliary variables (related to missingness, or to the missing values)? | 6 |
| 9 | Did you include the interactions and non-linear relationships used in the analysis in the imputation model too? | 6 |
| 10 | Are categorical variables factors? Did you use pmm for variables with a restricted range? | 5 |
| 11 | Did you check convergence (plot(imp)) and the imputed values (densityplot())? |
7 |
| 12 | Is m about the percentage of people with missing values? Are the results stable when the random seed is changed? | 7 |
| 13 | Did you compare with the results of complete-case analysis? | 5, 8 |
| 14 | Did you perform a sensitivity analysis for MNAR (delta adjustment, tipping point)? | 8 |
| 15 | Did you report the amount of and reasons for missingness, the assumptions, the imputation model, m, and the sensitivity analyses? | 8 |
9.2 Exercise 1: cholesterol and lifestyle
Select 3000 people from adults, and regress total cholesterol on age, sex, BMI, and physical activity.
9.2.1 1-1. Check the missingness
Check the proportion missing for each variable and the number of people actually used in the regression.
round(100 * colMeans(is.na(a)), 1)
nobs(fit_cc)The regression uses 2684 of the 3000 people.
9.2.2 1-2. Multiply impute
Perform multiple imputation with mice with m = 20, analyse and pool the same regression, and compare with complete-case analysis.
imp <- mice(a, m = 20, seed = 1, printFlag = FALSE)
summary(pool(fit_mi))[, c("term", "estimate", "std.error")]The age coefficient is 0.0091 for complete case and 0.0092 for multiple imputation — almost the same. In this example, the two methods lead to the same conclusions.
9.3 Exercise 2: computing Rubin’s rules by hand
With multiple imputation (m = 5), the estimates and standard errors of a mean came out as follows. Compute the pooled estimate, standard error, and degrees of freedom.
W <- mean(se^2); B <- var(est); T <- W + (1 + 1 / m) * BEstimate 5.17, standard error 0.417, degrees of freedom about 1070. Because the between-imputation variance B (0.009) is very small compared with the within-imputation variance W (0.163), the standard error is barely different from that of a single dataset (about 0.40), and the degrees of freedom are large. This is an example with little loss of information due to missingness.
9.4 Exercise 3: when an explanatory variable is missing depending on the outcome
Make 40% of BMI in full missing with a mechanism in which higher blood pressure makes missingness more likely. Compare the BMI coefficient from complete-case analysis and from multiple imputation.
d$BMI <- make_missing(full$BMI, miss_prob(full$BPSysAve, 0.4))
s[s$term == "BMI", c("estimate", "std.error")]Compared with 0.242 before missingness, complete case gives a smaller 0.215 (biased because missingness is determined by the outcome; Chapter 2), and multiple imputation gives 0.240. Because blood pressure is in the imputation model, the missing-data mechanism for BMI is MAR, and multiple imputation removes the bias.
9.5 Exercise 4: your own data
With your own research data (or public data), go through items 1–15 of the checklist in order. In particular, write down before looking at the data:
- Possible reasons for missingness
- Variables that can serve as auxiliary variables
- The δ in the direction that works against the conclusion, and its clinical plausibility
9.6 Review questions
- What does “random” mean in MAR?
- When missingness is determined only by explanatory variables, are the complete-case mean and regression coefficients each biased?
- With stochastic regression imputation, why are confidence intervals too narrow even though the bias of the estimates is small?
- In Rubin’s rules, what does the total variance T consist of?
- When imputing an explanatory variable, what happens if the outcome is not included in the imputation model?
- What is the problem with multiple imputation using m = 5?
- It means “conditional on the observed variables, whether a value is missing is random”. Being random regardless of anything is MCAR (Chapter 1)
- The mean is biased, but the regression coefficients are not (if the model is correct) (Chapter 2)
- Because the uncertainty of the imputed values (and of the regression coefficients used for imputation) is ignored and the imputed values are treated like measured values, the standard errors are too small (Chapter 3)
- The within-imputation variance W (uncertainty within one dataset) and the between-imputation variance B (how much the results change with imputation). \(T = W + (1 + 1/m) B\) (Chapter 4)
- The relationship between the imputed explanatory variable and the outcome is weakened toward 0, and the coefficient is underestimated (Chapter 6)
- Although the estimates tend to be stable, the estimate of the between-imputation variance B is unstable, so standard errors and p-values change greatly depending on the random seed (Chapter 7)