8  Exercises and a checklist for adjusting for confounding

Let us put the contents of the previous chapters to use with your own hands. The code cells in this chapter do not run automatically. Fill in the blanks (___) and press “Run Code”. Example answers are folded under each question.

8.1 Checklist

# What to check Chapter
1 Did you write down which effect (total or direct) of which exposure on which outcome you want to know? 2
2 Did you draw a DAG and distinguish confounders, intermediates, and colliders? Did you choose adjustment variables before looking at the data? 2
3 Are you not adjusting for things that happened after the exposure (intermediates, things affected by the outcome)? 2
4 Is the way participants were selected (inpatients, volunteers, people who were tested) not a collider? 2
5 When the outcome is common (10% or more), are you not interpreting odds ratios as if they were risk ratios? Did you also show standardized risk differences and risk ratios? 3
6 Are you not coarsely categorizing continuous confounders? Are you not categorizing the exposure? Did you check the linearity assumption with splines? 4
7 Did you assess differences in effect with interaction, not with p-values by subgroup? Did you state the scale (multiplicative or additive)? 5
8 Are there not too many variables relative to the number of events? (10 events per variable as a rule of thumb) 6
9 Are you not selecting confounders by p-values or stepwise methods? 6
10 If you use propensity scores, did you check positivity (overlap), balance (SMD), and extreme weights? Did you state the target population (ATE / ATT)? 7
11 Did you discuss unmeasured confounding (E-value, etc.)? 1, 7
12 Are you not reporting coefficients of adjustment variables from one model as “effects” (the Table 2 fallacy)? 2

8.2 Exercise 1: diabetes and 24-year cardiovascular disease

8.2.1 1-1. Unadjusted, and adjusted for age and sex

fit1 <- glm(CVD ~ DIABETES + AGE + SEX, family = binomial, data = fram)

The odds ratio drops from 4.70 to 3.63. People with diabetes have a mean age of 55.4 years, older than those without (49.8 years), and that confounding has been removed.

8.2.2 1-2. Standardized risk difference and risk ratio

24-year CVD is 26%, a “common outcome”. Obtain the risk difference and risk ratio adjusted for age and sex by standardization.

p1 <- mean(predict(fit1, newdata = mutate(fram, DIABETES = 1), type = "response"))
p0 <- mean(predict(fit1, newdata = mutate(fram, DIABETES = 0), type = "response"))

The risk difference is 26.4 percentage points and the risk ratio is 2.04. The risk ratio is much closer to 1 than the odds ratio of 3.63 (Chapter 3).

8.3 Exercise 2: what to adjust for

We investigate “does smoking increase 24-year CVD? (total effect)”. Sort the following variables into those that should be adjusted for and those that should not.

  1. Age
  2. Sex
  3. Cigarettes per day (CIGPDAY)
  4. Antihypertensive treatment started after the first examination
  5. Education
  6. Hospitalization for myocardial infarction during the study period
  • Adjust: 1 age, 2 sex, 5 education (common causes that can affect both smoking and CVD)
  • Do not adjust:
    • 3 cigarettes per day: part of the exposure itself (only smokers have a count). If you want to see a dose–response, treat it separately as the exposure
    • 4 treatment after the exposure: an intermediate / consequence of the exposure, as in smoking → higher blood pressure → treatment
    • 6 part of the outcome / something that happened after the outcome: adjusting creates collider bias
  • Blood pressure and BMI at the first examination could be both a consequence of smoking (intermediate) and a cause (confounder), so you need to draw a DAG and discuss (Chapter 2)

8.4 Exercise 3: residual confounding from coarse categories

For smoking and CVD, compare entering age as “under 50 / 50 and over” with entering it as continuous.

age_2cat       = ... + age_2cat
age_continuous = ... + AGE

It goes 1.17 → 1.23 → 1.30. With 2 categories, confounding due to smokers being younger remains, and the association is underestimated (Chapter 4).

8.5 Exercise 4: does the impact of smoking differ between men and women?

Check the odds ratios by sex and the interaction term.

fit_int <- glm(CVD ~ CURSMOKE * factor(SEX) + AGE, family = binomial, data = fram)

Men 1.23 (1.01–1.50), women 1.44 (1.15–1.81); the confidence intervals overlap substantially. The interaction term has p = 0.85, and there is no evidence that the effect differs between men and women. Rather than writing “the association is stronger in women”, report that “there was no clear difference between men and women” (Chapter 5).

8.6 Review questions

NoteQ1

In a study of “exercise habits and dementia”, adjusting for BMI, blood pressure, and HbA1c made the effect of exercise smaller. Can you say “most of the effect of exercise was confounding”?

No. Exercise improves BMI, blood pressure, and blood glucose, so these are likely intermediates. Adjusting for them subtracts the part of the effect of exercise that works through them (overadjustment, Chapter 2).

NoteQ2

A study of hospitalized patients only found that “smokers less often develop severe COVID-19”. What is a possible explanation?

Hospitalization (the way participants were selected) may be a collider. When people hospitalized for smoking-related diseases are mixed with people hospitalized for COVID-19, a negative association arises between smoking and severe COVID-19 among inpatients (Berkson’s bias, Chapter 2).

NoteQ3

For 24-year death (35%), an adjusted odds ratio of 1.63 was written as “smokers have a 63% higher risk of death”. What is the problem?

Because the outcome is common, the odds ratio is further from 1 than the risk ratio. The standardized risk ratio was 1.28 (28% higher), and the risk difference was 8.6 percentage points (Chapter 3).

NoteQ4

In a study with 40 events, a logistic regression adjusted for 15 confounders gave an exposure odds ratio of 3.5. How do you view this result?

There are only about 2.5 events per variable, so the odds ratio is likely biased upward and unstable (Chapter 6). Consider narrowing down confounders with a DAG, summarizing them into a propensity score, or using penalized regression.

NoteQ5

In propensity score matching, the SMDs of all covariates fell below 0.1. Can you now say the result is as trustworthy as a randomized controlled trial?

No. Balance was achieved only for the measured covariates. Randomization also balances unmeasured factors, but propensity scores cannot (Chapter 7). Show robustness to unmeasured confounding with the E-value or similar.

NoteQ6

A study found the treatment effect significant in people under 50 (p = 0.01) and not significant in those 50 and over (p = 0.20), and concluded that “it works only in younger people”. What should be checked?

Directly test and estimate the interaction between age and treatment. If the estimates and confidence intervals of the two strata overlap, it is not evidence of a difference in effect. Also check whether that subgroup analysis was planned in advance (Chapter 5).

8.7 Closing

Adjusting for confounding is not “the task of putting variables into a model”, but a sequence of steps:

  1. Decide the question (which effect you want to know)
  2. Choose adjustment variables from the causal structure (DAG)
  3. Build the model without discarding information (continuous variables, splines) and without putting in too many variables
  4. Present the results on easily interpretable scales (standardized risk differences and risk ratios)
  5. Honestly discuss the possibility of unmeasured confounding

Whichever method you use, do not forget that unmeasured confounding remains.