6  Number of variables, overfitting, and variable selection by p-values

It is tempting to think “adjusting for many confounders is safer”, but if there are too many variables relative to the number of events (outcomes), the estimate itself becomes unstable and biased. Variable selection that “keeps only variables with small p-values” also has problems that are easy to overlook. Everything in this chapter is checked with simulations where the answer is known.

6.1 Number of events and number of variables

In a logistic regression, let the true odds ratio of exposure X be 2, and adjust for 10 other covariates. We vary the sample size (= the number of events) and repeat the study 300 times for each.

  • With about 30 events (about 3 events per variable), the mean odds ratio is an overestimated 2.7, and it varies from 1.2 to 4.3 across studies
  • With 10 or more events per variable, it gets close to 2

Logistic regression is known to be biased away from 1 when there are few events. A commonly used rule of thumb is “10 events per variable (EPV ≥ 10)”, though in some situations more are said to be needed.

TipWhen events are few
  • Narrow the adjustment variables down to the confounders that are truly needed according to the DAG
  • Summarize many confounders into a single propensity score (Chapter 7)
  • Use penalized regression (e.g., Firth’s method)

6.2 What happens when you select variables by p-values?

Suppose that out of 20 candidate variables, 5 have a truly weak effect (log-odds 0.15) and 15 have no effect at all. We build a model on data from 300 people and select only the variables with p < 0.05 as “meaningful”, repeating this 300 times.

  • Of the 5 variables with a real effect, on average only about 1 is selected
  • Of the 15 with no effect, on average about 0.8 are selected
  • The mean estimate of the selected “real” variables is 0.37 (truly 0.15). Overestimated by more than twofold

Selecting by p-values leaves only the estimates that happened to come out large (the winner’s curse). The coefficients, p-values, and confidence intervals of the selected variables ignore the uncertainty of the selection itself, so they are too optimistic.

6.3 Do not select confounders by p-values

What happens if you select confounders by the rule “adjust if significantly associated with the outcome”? Z is a confounder strongly related to the exposure X and moderately related to the outcome Y. The true effect of X is 0.

  • Z is significant in only about 54% of studies
  • Always adjust for Z: the mean coefficient of X is nearly 0 (correct)
  • Adjust only when significant: mean 0.11 (bias remains)
  • Unadjusted: 0.42

Because Z is strongly related to X, the effect of Z is shared with X inside the model, and the p-value for Z tends to be large. But dropping Z brings the bias back. Select confounders by the causal structure (DAG) and prior knowledge, not by their p-values with the outcome.

6.4 Real data: stepwise selection

R’s step() searches for a “good” model by adding and removing variables based on AIC. Let us try it on Framingham for 24-year death.

step() selects variables that help predict death. That is a different goal from the set of confounders needed to learn “the effect of smoking” (the difference between prediction and causal inference). In this example, BMI, diastolic blood pressure, and total cholesterol were dropped. BMI may be a confounder of smoking and death, but it was dropped on the grounds that it “does not help prediction”. Even when building a prediction model, unless the whole selection procedure is evaluated by cross-validation or similar, performance will be overestimated (the agreement hands-on, Chapter 6).

ImportantSummary of this chapter
  • With too many variables relative to the number of events (fewer than 10 events per variable), odds ratios are biased upward and unstable
  • Selecting variables by p-values overestimates the selected estimates (the winner’s curse) and also selects variables with no effect
  • Do not select confounders by p-values. Dropping them brings the bias back. Decide by the DAG and prior knowledge
  • Stepwise selection is a method for selecting “variables that help prediction”, not a method for adjusting for confounding