---
webr:
packages: [dplyr, ggplot2]
resources: [R/setup.R, data/nhanes_bp.rds, data/framingham.rds]
cell-options:
autorun: true
---
{{< include ./_extensions/r-wasm/live/_knitr.qmd >}}
# ROC curves and AUC {#sec-roc}
```{webr}
#| include: false
source("R/setup.R")
d <- fram |> filter(!is.na(SYSBP), !is.na(TOTCHOL), !is.na(BMI), !is.na(GLUCOSE))
```
In Chapter 5 we saw that moving the cut-off creates a trade-off between sensitivity and specificity.
Connecting sensitivity and (1 − specificity) at every cut-off gives the **ROC curve**, and the area under it is the **AUC** (c-statistic).
In this chapter we consider the example of predicting, from the 1st Framingham examination, **whether a person will develop cardiovascular disease (CVD) within 24 years**.
```{webr}
nrow(d)
mean(d$CVD) # proportion who developed CVD within 24 years
```
## Predicting with systolic blood pressure alone
```{webr}
roc_points <- function(score, y) {
cuts <- sort(unique(score), decreasing = TRUE)
data.frame(cutoff = cuts,
sensitivity = sapply(cuts, function(k) mean(score[y == 1] >= k)),
fpr = sapply(cuts, function(k) mean(score[y == 0] >= k))) # false positive rate = 1 - specificity
}
roc_sbp <- roc_points(d$SYSBP, d$CVD)
head(roc_sbp[roc_sbp$cutoff %in% c(180, 160, 140, 130, 120), ], 5)
```
```{webr}
#| fig-height: 4.5
ggplot(roc_sbp, aes(fpr, sensitivity)) +
geom_step(direction = "vh", linewidth = 1, colour = "steelblue") +
geom_abline(linetype = 2, colour = "grey50") +
coord_equal() +
labs(x = "False positive rate (1 − specificity)", y = "Sensitivity", title = "ROC curve: predicting 24-year CVD with systolic BP")
```
The closer the curve is to the top-left corner, the better the test discriminates. The diagonal (dotted line) is the same as deciding by tossing a coin (no discriminative ability).
## What AUC means
AUC is the area under the ROC curve. It also has another, very intuitive meaning:
> **If you randomly pick one person who developed CVD and one who did not, the probability that the person who developed CVD has the higher value**
This is the same as the rank-based Mann-Whitney statistic.
```{webr}
auc <- function(score, y) { # ties are counted as 0.5
r <- rank(score); n1 <- sum(y == 1); n0 <- sum(y == 0)
(sum(r[y == 1]) - n1 * (n1 + 1) / 2) / (n1 * n0)
}
auc(d$SYSBP, d$CVD)
# check it by actually making pairs
set.seed(1)
case <- sample(d$SYSBP[d$CVD == 1], 1e5, replace = TRUE)
ctrl <- sample(d$SYSBP[d$CVD == 0], 1e5, replace = TRUE)
mean(case > ctrl) + 0.5 * mean(case == ctrl)
```
The AUC of systolic blood pressure is **0.66**. Only 66% of pairs have the higher blood pressure in the person who developed CVD.
## Combining several variables
We combine age, sex, blood pressure, cholesterol, smoking, BMI and diabetes with logistic regression to build a "risk score".
```{webr}
fit <- glm(CVD ~ AGE + SEX + SYSBP + TOTCHOL + CURSMOKE + BMI + DIABETES, family = binomial, data = d)
d$risk <- fitted(fit)
c(systolic_BP = auc(d$SYSBP, d$CVD), age = auc(d$AGE, d$CVD), risk_score = auc(d$risk, d$CVD))
```
```{webr}
#| fig-height: 4.5
bind_rows(cbind(roc_points(d$SYSBP, d$CVD), measure = "Systolic BP"),
cbind(roc_points(d$risk, d$CVD), measure = "Risk score")) |>
ggplot(aes(fpr, sensitivity, colour = measure)) +
geom_step(direction = "vh", linewidth = 1) +
geom_abline(linetype = 2, colour = "grey50") +
coord_equal() + labs(x = "False positive rate (1 − specificity)", y = "Sensitivity", colour = NULL)
```
The AUC of the risk score is **0.74**. Disease onset 24 years ahead cannot be predicted perfectly from a single examination.
## The optimism of the "optimal cut-off"
A commonly used approach is to call the cut-off that maximizes the **Youden index** (sensitivity + specificity − 1) "optimal".
The problem here is **reporting its performance on the same data used to choose the cut-off**.
We split the data in half, choose the cut-off on one half (training) and check its performance on the other half (validation).
```{webr}
set.seed(1)
idx <- sample(nrow(d), nrow(d) / 2)
train <- d[idx, ]; test <- d[-idx, ]
youden <- function(score, y, k) mean(score[y == 1] >= k) + mean(score[y == 0] < k) - 1
cands <- sort(unique(train$SYSBP))
J <- sapply(cands, function(k) youden(train$SYSBP, train$CVD, k))
best <- cands[which.max(J)]
c(chosen_cutoff = best,
Youden_training = max(J),
Youden_validation = youden(test$SYSBP, test$CVD, best))
```
The cut-off chosen on the training data (133 mmHg) has a Youden index of 0.27, but on new data it drops to 0.22.
This is because **it adapts to the chance bumps and dips of the data it was chosen on** (optimism).
## Optimism of prediction models: many variables, few people
The same thing happens with the AUC of a prediction model.
We take only **200 people** from Framingham and build a model with the 7 true predictors plus **30 random-noise variables that have nothing to do with CVD**.
```{webr}
set.seed(4)
small <- d |> slice_sample(n = 200)
noise <- as.data.frame(matrix(rnorm(200 * 30), 200, dimnames = list(NULL, paste0("noise", 1:30))))
small2 <- cbind(small[, c("CVD", "AGE", "SEX", "SYSBP", "TOTCHOL", "CURSMOKE", "BMI", "DIABETES")], noise)
fit_s <- suppressWarnings(glm(CVD ~ ., family = binomial, data = small2))
# validate on the remaining people (the noise variables are generated randomly in the same way)
rest <- d |> anti_join(small, by = "RANDID")
noise_r <- as.data.frame(matrix(rnorm(nrow(rest) * 30), nrow(rest), dimnames = list(NULL, paste0("noise", 1:30))))
rest2 <- cbind(rest, noise_r)
c(AUC_development_data = auc(fitted(fit_s), small2$CVD),
AUC_new_data = auc(predict(fit_s, rest2), rest2$CVD))
```
On the data used to build it, the AUC looks high at **0.85**, but on new data it drops to **0.62** (lower than the 0.74 obtained when the 7-variable model was built on everyone).
For a model with many variables relative to the number of people (especially the number of events), **you cannot trust its performance on the data it was built on**.
Report performance on **separate data (external validation)**, or correct for optimism with **cross-validation or the bootstrap**.
## AUC is not enough: calibration
AUC looks only at **ranks**. Even if you halve all predicted probabilities, the ranks do not change, so the AUC stays the same.
```{webr}
c(original_score = auc(d$risk, d$CVD), halved_probability_score = auc(d$risk / 2, d$CVD))
```
However, if you tell someone "your 24-year CVD risk is 15%", what matters is **whether the predicted probabilities match the actual incidence** (**calibration**).
We split people into 10 groups by predicted risk and compare predicted with observed.
```{webr}
#| fig-height: 4.2
calib <- function(p, y) {
tibble(p, y) |>
mutate(group = ntile(p, 10)) |>
group_by(group) |>
summarise(predicted = mean(p), observed = mean(y))
}
bind_rows(cbind(calib(d$risk, d$CVD), score = "Original model"),
cbind(calib(d$risk / 2, d$CVD), score = "Model with halved probabilities")) |>
ggplot(aes(100 * predicted, 100 * observed, colour = score)) +
geom_abline(linetype = 2) + geom_point(size = 2) + geom_line() +
coord_equal(xlim = c(0, 80), ylim = c(0, 80)) +
labs(x = "Predicted probability of onset (%)", y = "Observed proportion with onset (%)", colour = NULL,
title = "Calibration plot (dotted line: predicted = observed)")
```
The original model lies along the dotted line, but the model with halved probabilities, despite having the same AUC, **systematically underestimates risk**.
When a model is applied to a population from another country or another era, this kind of miscalibration is common (e.g., applying the Framingham risk equation as is to a population with less heart disease overestimates risk).
::: {.callout-important}
## Chapter summary
- The ROC curve shows sensitivity and false positive rate at every cut-off. **AUC = the probability that the person who develops the disease has the higher value**
- Reporting an "optimal cut-off" or a prediction model's performance **on the same data used to build it is optimistic**. Check with separate data, cross-validation or the bootstrap
- Optimism is especially large for models with many variables relative to the number of people (events)
- AUC looks only at ranks. If you use the risk values themselves, also check **calibration**
:::