5 Diagnostic accuracy: sensitivity, specificity, predictive values
From here on, we deal with diagnostic accuracy: how correctly a test can discriminate whether a disease (condition) is present or not. In the evaluation of agreement (Chapters 1–4), neither of the two measurements was the “correct answer”, but in evaluating diagnostic accuracy we assume there is a correct answer (reference standard, gold standard).
5.1 Data: casual blood glucose and diabetes
At the 1st Framingham examination, we use casual blood glucose (blood glucose measured regardless of the time of the last meal) and the record of diabetes.
People with diabetes tend to have higher blood glucose, but the two distributions overlap. Wherever you draw a line and call it “positive”, there will always be misclassifications.
5.2 The 2×2 table and four measures
Let us call 126 mg/dL or higher “positive”.
- Sensitivity 59.5%: 4 in 10 people with diabetes are missed by this test
- Specificity 99.4%: almost all people without diabetes test negative
- Positive predictive value 74.2%: 1 in 4 people who test positive do not have diabetes
- Sensitivity and specificity: split by the truth (disease present or not), and see how the test comes out (look column by column in the table)
- Predictive values: split by the test result, and see whether the disease is truly present (look row by row in the table)
What a patient wants to know is “the probability that I, having been told I am positive, really have the disease” = the positive predictive value.
5.2.1 Also show confidence intervals
Sensitivity is computed only from the 116 people with diabetes, so its uncertainty is large.
The 95% confidence interval for sensitivity is wide at 50–68%, much more uncertain than specificity (99.1–99.6%).
5.3 Changing the cut-off
Lowering the cut-off raises sensitivity and lowers specificity (a trade-off). Where to cut depends on which is worse: missed cases (false negatives) or unnecessary further testing (false positives). The ROC curve in Chapter 6 shows this relationship at a glance.
“Diabetes” in this data is defined as people who, at the 1st examination, were under treatment or had casual blood glucose of 200 mg/dL or more. In other words, the test being evaluated (blood glucose) is included in the definition of the truth. At 200 or more one is always “diabetic”, so specificity is always 100%. This is called incorporation bias, and it makes a test look more accurate than it really is. The truth must be determined independently of the test being evaluated.
5.4 The positive predictive value changes with prevalence
Sensitivity and specificity are (to some extent) properties of the test itself, but predictive values change greatly with the prevalence in the population tested. They can be computed with Bayes’ theorem.
\[ \text{PPV} = \frac{\text{sensitivity} \times \text{prevalence}}{\text{sensitivity} \times \text{prevalence} + (1 - \text{specificity}) \times (1 - \text{prevalence})} \]
Even with a “good-looking” test of 90% sensitivity and 95% specificity, in a population with 1% prevalence (screening the general public), 85% of the people who test positive do not have the disease. If the same test is used on people with symptoms in whom the disease is suspected (prevalence 30%), the positive predictive value becomes 89%.
5.4.1 Likelihood ratios: updating the pre-test probability to the post-test probability
The likelihood ratio expresses, independently of prevalence, how much a test result shifts the diagnosis.
- Positive likelihood ratio LR+ = sensitivity / (1 − specificity): how many times the odds become when positive
- Negative likelihood ratio LR− = (1 − sensitivity) / specificity: how many times the odds become when negative
If a person with a pre-test probability of 3% tests positive, the post-test probability becomes 75%. LR+ ≈ 99 is very large, so a positive result greatly increases the probability of diabetes. On the other hand, LR− ≈ 0.41, so even a negative result has little power to rule out diabetes.
5.5 Spectrum bias: whom did you evaluate?
Sensitivity and specificity also actually change with the people used in the evaluation. In studies that recruit severely ill patients and healthy volunteers (case-control-type diagnostic studies), the difference between the two groups is clear, so accuracy comes out high.
Let us check by simulation. Suppose test values vary with the severity of the disease.
With the same test and the same cut-off, evaluating in severe cases and healthy people makes both sensitivity and specificity come out high. Unless you evaluate in the people who will actually use the test (attendees whose diagnosis is uncertain), you overestimate its accuracy in practice.
- Look at sensitivity and specificity split by the truth, and predictive values split by the test result. Give a confidence interval for sensitivity
- Moving the cut-off creates a trade-off between sensitivity and specificity
- The positive predictive value changes greatly with prevalence. In screening low-prevalence populations, most positives may be false positives
- Likelihood ratios update the pre-test probability to the post-test probability
- Do not include the test in the definition of the truth (incorporation bias); evaluate in the people who will actually use the test (spectrum bias)