7 Propensity scores: weighting, matching, and their limits
The propensity score is “the probability that a person receives the treatment (exposure), given their background”. It is a tool for summarizing many confounders into a single number in order to compare people who did and did not receive treatment so that their backgrounds are balanced. We will try it on antihypertensive medication and 24-year death from Chapter 1.
This is data from the 1950s, and the antihypertensive drugs of that time are completely different from today’s. The results in this chapter are for learning how to use propensity scores, not conclusions about the effect of antihypertensive drugs.
7.1 First: are there people to compare with? (positivity)
All 144 people taking antihypertensive medication had a history of hypertension. People without hypertension have zero chance of taking antihypertensives, so they cannot serve as a comparison. Propensity score methods assume that for every background, there are both people who receive treatment and people who do not (positivity). So we restrict to people with a history of hypertension, and compare “people with hypertension who were and were not taking antihypertensive medication”.
7.2 Balance before adjustment: SMD
We look at the difference in background between the two groups with the standardized mean difference (SMD) (Chapter 7 of the descriptive statistics hands-on). As a rule of thumb, an absolute value below 0.1 is considered well balanced.
People taking antihypertensive medication were older (0.35), more often male, had higher blood pressure (0.47), and more often had a history of heart disease or stroke; their backgrounds differ considerably.
7.3 Estimating the propensity score
We fit a logistic regression with the treatment (antihypertensive medication) as the outcome and the confounders as predictors. The predicted probability is the propensity score.
Comparison is only possible in the range where the propensity score distributions of the two groups overlap. In this example, there are also untreated people within the range of propensity scores of the treated (0.03–0.78).
The goal of the propensity score model is not to predict treatment accurately but to balance the confounders. A higher AUC is not necessarily better (including variables that only determine treatment and are unrelated to the outcome actually makes it less stable; Chapter 2).
7.4 Method 1: inverse probability weighting (IPW)
Give treated people a weight of 1 / propensity score, and untreated people a weight of 1 / (1 − propensity score). “People who were treated despite a background unlikely to be treated” and “people who were not treated despite a background likely to be treated” get large weights, and in the weighted population, treatment becomes unrelated to background (as if randomly assigned).
After weighting, all SMDs are below 0.1. We compare the weighted death proportions.
The weighted death proportions are 61.5% and 51.8%, the risk difference is 9.7 percentage points, and the risk ratio is 1.19. The confidence interval is obtained by the bootstrap, including the estimation of the propensity score (it takes a little time).
The 95% confidence interval for the risk difference is 0.6 to 19.2 percentage points, quite wide.
7.4.1 Beware of extreme weights
A 40-year-old with a blood pressure of 122 who was nonetheless taking antihypertensive medication has a low propensity score of 0.03 and carries the weight of 36 people on their own. When a few people have extremely large weights, the estimate becomes unstable. Remedies include stabilized weights (multiply the weights by the proportion treated), weight truncation (cap at the top 1%), and excluding people with little overlap.
7.5 Method 2: matching
For each person who was taking antihypertensive medication, we choose one untreated person with the closest propensity score. If there is no one close enough, no pair is formed (caliper: within 0.2 × the SD of the logit of the propensity score).
Pairs were found for 138 of the 140 people, and age and blood pressure were balanced, but some imbalance remained for smoking (0.16), history of heart disease (0.11), and diabetes (0.10). With matching, the remaining imbalance is sometimes further adjusted for by regression after matching. The death proportions are 65.2% and 52.9%.
- The IPW above (1/ps, 1/(1−ps)) estimates the effect if antihypertensives were used in all people with hypertension (ATE)
- Matching estimates the effect in people like those who actually took antihypertensives (ATT)
If the effect differs between people likely and unlikely to be treated, the two values will differ. Decide first which question you want to answer. Matched pairs are paired data, so use tests such as McNemar’s test (Chapter 6 of the chi-squared test hands-on).
7.6 Differences between methods, and the remaining problem
The difference, 14.8 percentage points without adjustment, shrank to 6–12 points with every method, but it does not reach 0.
Propensity scores only balance the confounders put into the model (that were measured). Antihypertensives may have been prescribed because of differences in organ damage, severity, or physician judgment that the measured variables cannot capture. Propensity scores cannot adjust for these either. Propensity scores are not a “more powerful” method than regression; they are a different way of adjusting under the same assumption (no unmeasured confounding). Their advantages are that balance can be checked without looking at the outcome, that many confounders can be summarized into one (when events are few; Chapter 6), and that positivity problems become easier to see.
7.6.1 How much unmeasured confounding would explain the result? The E-value
The E-value expresses how strong a confounder would be needed to explain the IPW risk ratio (about 1.19) by unmeasured confounding alone.
An E-value of about 1.66 means that “if there were an unmeasured confounder associated with both taking antihypertensives and death with a risk ratio of 1.66 or more, it could explain this association entirely”. For a confounder like severity, an association of that size is quite plausible.
- Propensity score = “the probability of receiving treatment” given background. First check positivity (whether there are people to compare with)
- IPW (effect in the whole population) and matching (effect in the treated) estimate effects in different populations
- Check balance with SMD after adjustment. Beware of extreme weights
- Unmeasured confounding cannot be removed. Use the E-value to show how much confounding could explain the result