6  ROC curves and AUC

In Chapter 5 we saw that moving the cut-off creates a trade-off between sensitivity and specificity. Connecting sensitivity and (1 − specificity) at every cut-off gives the ROC curve, and the area under it is the AUC (c-statistic). In this chapter we consider the example of predicting, from the 1st Framingham examination, whether a person will develop cardiovascular disease (CVD) within 24 years.

6.1 Predicting with systolic blood pressure alone

The closer the curve is to the top-left corner, the better the test discriminates. The diagonal (dotted line) is the same as deciding by tossing a coin (no discriminative ability).

6.2 What AUC means

AUC is the area under the ROC curve. It also has another, very intuitive meaning:

If you randomly pick one person who developed CVD and one who did not, the probability that the person who developed CVD has the higher value

This is the same as the rank-based Mann-Whitney statistic.

The AUC of systolic blood pressure is 0.66. Only 66% of pairs have the higher blood pressure in the person who developed CVD.

6.3 Combining several variables

We combine age, sex, blood pressure, cholesterol, smoking, BMI and diabetes with logistic regression to build a “risk score”.

The AUC of the risk score is 0.74. Disease onset 24 years ahead cannot be predicted perfectly from a single examination.

6.4 The optimism of the “optimal cut-off”

A commonly used approach is to call the cut-off that maximizes the Youden index (sensitivity + specificity − 1) “optimal”. The problem here is reporting its performance on the same data used to choose the cut-off.

We split the data in half, choose the cut-off on one half (training) and check its performance on the other half (validation).

The cut-off chosen on the training data (133 mmHg) has a Youden index of 0.27, but on new data it drops to 0.22. This is because it adapts to the chance bumps and dips of the data it was chosen on (optimism).

6.5 Optimism of prediction models: many variables, few people

The same thing happens with the AUC of a prediction model. We take only 200 people from Framingham and build a model with the 7 true predictors plus 30 random-noise variables that have nothing to do with CVD.

On the data used to build it, the AUC looks high at 0.85, but on new data it drops to 0.62 (lower than the 0.74 obtained when the 7-variable model was built on everyone). For a model with many variables relative to the number of people (especially the number of events), you cannot trust its performance on the data it was built on. Report performance on separate data (external validation), or correct for optimism with cross-validation or the bootstrap.

6.6 AUC is not enough: calibration

AUC looks only at ranks. Even if you halve all predicted probabilities, the ranks do not change, so the AUC stays the same.

However, if you tell someone “your 24-year CVD risk is 15%”, what matters is whether the predicted probabilities match the actual incidence (calibration). We split people into 10 groups by predicted risk and compare predicted with observed.

The original model lies along the dotted line, but the model with halved probabilities, despite having the same AUC, systematically underestimates risk. When a model is applied to a population from another country or another era, this kind of miscalibration is common (e.g., applying the Framingham risk equation as is to a population with less heart disease overestimates risk).

ImportantChapter summary
  • The ROC curve shows sensitivity and false positive rate at every cut-off. AUC = the probability that the person who develops the disease has the higher value
  • Reporting an “optimal cut-off” or a prediction model’s performance on the same data used to build it is optimistic. Check with separate data, cross-validation or the bootstrap
  • Optimism is especially large for models with many variables relative to the number of people (events)
  • AUC looks only at ranks. If you use the risk values themselves, also check calibration