7 Checking the imputations and choosing m
Running mice() always produces some result. To decide whether to trust it, check the following three things.
- Have the iterations converged?
- Are the imputed values plausible?
- Is the number of imputations m sufficient?
7.1 Checking convergence
mice runs the iterations that re-impute each variable with missing values in turn separately in m chains (Chapter 5). We plot how the mean and standard deviation of the imputed values changed with each iteration.
Each line is one chain. If they have converged,
- the lines show no tendency to keep rising or falling
- the m lines are intermingled, with no single line standing apart
In this example, after the first few iterations the lines are flat and well mixed. It seems safe to consider them converged.
- If lines keep rising, or chains settle at different levels, increase
maxitand watch - If it still does not converge, revisit the imputation model. Common causes are imputing separately strongly correlated variables (the same thing measured in different units, a total and its components) or variables computed from other variables (passive imputation in Chapter 6)
7.2 Checking the imputed values
Place the distribution of the imputed values next to that of the observed values.
The blue line is the distribution of the observed values, and the red lines are those of the imputed values (m lines).
Column 0 shows the observed values, and columns 1–5 the imputed values (red points).
Under MAR, the people with missing values may well have different characteristics from those observed, so it is natural for the distribution of imputed values to differ somewhat from the observed values (in Chapter 1, the people missing under MAR were skewed toward the young). What you want to check is
- whether impossible values have been imputed (negative values, out-of-range values)
- whether the distribution is extremely shifted (and if so, whether you can explain why)
7.3 How large should m be?
It used to be said that “m = 3–5 is enough”, because the estimates themselves tend to be stable even with small m. However, stabilizing standard errors and p-values requires more imputations.
Using the data with 40% of blood pressure in full made missing, we rerun mice 8 times changing only the random seed, and compare how much the estimate and standard error of the mean blood pressure fluctuate for m = 5, 20, and 50.
| m | Estimate (range over 8 runs) | Standard error (range over 8 runs) |
|---|---|---|
| 5 | 121.52–121.84 | 0.42–0.78 |
| 20 | 121.59–121.86 | 0.54–0.60 |
| 50 | 121.64–121.81 | 0.54–0.57 |
With m = 5, the estimates do not fluctuate much, but the standard errors range from 0.42 to 0.78, nearly a factor of 2. With the same data and the same method, the width of the confidence interval and the p-value change greatly depending on the random seed. This is because the between-imputation variance B is estimated from the spread of only 5 estimates (Practice 2 in Chapter 4).
- Set m to about the percentage of people with missing values (the rule of thumb of White et al.). In this example that is 40%, so m ≈ 40
- If unsure, use m = 20–50. On today’s computers this finishes quickly in most cases
- In the final analysis, check that the results hardly change when the random seed is changed
For the Chapter 5 data it is 21%, so m ≈ 20 is sufficient.
7.4 Try it
- Use
plot(imp)to check the chains forPoverty(poverty ratio) as well - Use
densityplot()to compare the distributions of observed and imputed values forPoverty. Since it is imputed with pmm, do you notice any characteristic feature?
- As with the other variables, it flattens out after a few iterations, and the chains are well mixed
- The poverty ratio is capped at 5, so there is a peak at 5. pmm selects from observed values, so the peak at 5 is reproduced in the imputed values too (imputing with normal linear regression does not reproduce this peak and also produces values above 5). The peak of imputed values is lower than for the observed values because people with missing income are predicted, from the other variables, to have somewhat lower income — consistent with the tendency seen in Chapter 1 that “people with missing blood pressure have slightly lower income”
- Check convergence with
plot(imp): no trend, and chains intermingled - Check the imputed values with
densityplot()andstripplot(): no impossible values, and any shift can be explained - With small m, standard errors are unstable. Use about the percentage of people with missing values, or 20–50 if unsure