7 Exercises and a checklist for agreement and diagnostic accuracy
Now let’s apply the contents of the previous chapters with your own hands. The code cells in this chapter are not run automatically. Fill in the blanks (___) and press “Run Code”. A sample solution is collapsed under each problem.
7.1 Checklist
7.1.1 Agreement (two measurement methods, raters, or repeated measurements)
| # | What to check | Chapter |
|---|---|---|
| 1 | Have you avoided concluding agreement from a correlation coefficient or a “non-significant” t-test alone? | 1 |
| 2 | Have you shown the mean difference (systematic error) and the 95% limits of agreement with a Bland-Altman plot? With few people, also their confidence intervals | 2 |
| 3 | Have you checked in the plot whether the differences show a slope (proportional error) or widening (error proportional to the value → log / ratio)? | 2 |
| 4 | Did you decide the clinically acceptable range for calling it “agreement” before looking at the data? | 2 |
| 5 | Do you know the repeatability of the conventional method itself (within-person SD, repeatability coefficient)? | 3 |
| 6 | When reporting an ICC, did you state its type (one-way, consistency, absolute agreement) and the range of the population? | 3 |
| 7 | For categorical agreement, did you show the percentage agreement and the proportion in each category along with κ? If ordered, weighted κ (and state the weights) | 4 |
7.1.2 Diagnostic accuracy and prediction
| # | What to check | Chapter |
|---|---|---|
| 8 | Was the correct answer (reference standard) determined independently of the test being evaluated (incorporation bias)? | 5 |
| 9 | Are the people evaluated a population similar to those on whom the test will actually be used (spectrum bias)? | 5 |
| 10 | Did you attach confidence intervals to sensitivity and specificity? | 5 |
| 11 | When reporting predictive values, did you state the prevalence of the population? | 5 |
| 12 | Did you check the performance of a cut-off or prediction model on data other than the data used to choose/build it (or with cross-validation / the bootstrap)? | 6 |
| 13 | If you use risk values, did you check not only the AUC but also calibration? | 6 |
7.2 Exercise 1: Repeatability of diastolic blood pressure
We examine the agreement between the 2nd and 3rd NHANES readings of diastolic blood pressure (BPDia2, BPDia3).
7.2.1 1-1. Bland-Altman plot
On the left side of the plot, is there a row of points that looks different from the rest? What is it?
diff = BPDia2 - BPDia3The mean difference is 0.4 and the limits of agreement are −12.1 to +12.9 mmHg. On the left side of the plot, points with large differences and low means line up diagonally. These are people whose diastolic blood pressure was recorded as 0 (118 people). With the auscultatory method, the sounds sometimes do not disappear all the way down to 0 mmHg, in which case 0 is recorded. As a measurement, it is not the true blood pressure.
sum(dia$BPDia2 == 0 | dia$BPDia3 == 0)
dia2 <- dia |> filter(BPDia2 > 0, BPDia3 > 0)
round(ba(dia2$BPDia2, dia2$BPDia3), 1)Excluding the zeros, the limits of agreement narrow to −9.7 to +10.4 mmHg. A Bland-Altman plot is also a tool for finding anomalies in the data (report the reason for exclusion and the number excluded).
7.2.2 1-2. ICC
Using the data with the zeros removed, compute the ICC and the within-person SD.
icc_oneway(cbind(dia2$BPDia2, dia2$BPDia3))The ICC is 0.91 and the within-person SD is 3.6 mmHg. Including the zeros gives an ICC of 0.89 and a within-person SD of 4.5 mmHg; the anomalous values were making the precision look worse.
7.3 Exercise 2: Agreement of hypertension classification
For the 2nd and 3rd readings, classify as hypertensive if “systolic ≥ 140 or diastolic ≥ 90”, and compute the percentage agreement and κ of the classifications.
h3 <- factor(e$BPSys3 >= 140 | e$BPDia3 >= 90, c(FALSE, TRUE))The percentage agreement is 94.6%, chance agreement 70.2%, and κ = 0.82. When reporting, also give the proportion classified as hypertensive (about 19%).
7.4 Exercise 3: Using it in health screening
Suppose a test has a sensitivity of 85% and a specificity of 90%.
- When used on people attending a health screening with a prevalence of 2%, what is the positive predictive value?
- If 10,000 people are screened, how many test positive, and how many of them truly have the disease?
FP <- N * (1 - prev) * (1 - spec)1150 people test positive, of whom only 170 truly have the disease, so the positive predictive value is 14.8%. 85% of those who test positive will undergo a detailed work-up even though they do not have the disease. There are also 30 people who are missed (false negatives). Whether to use the test in screening is decided by weighing the burden of work-ups against the benefit of detection.
7.5 Exercise 4: Predicting CVD with total cholesterol
In Framingham, compute the AUC of total cholesterol (TOTCHOL) at the 1st examination for 24-year CVD.
auc(fc$TOTCHOL, fc$CVD)The AUC is 0.58. Total cholesterol is an established risk factor for CVD, but on its own it has little ability to discriminate who will develop the disease. Being a risk factor (being associated) and being useful for prediction (being able to discriminate) are different things. Even a risk factor with an odds ratio of 2 often does not raise the AUC much.
7.6 Review questions
The correlation coefficient between a new thermometer and a conventional thermometer was 0.98. Can you say that “the new thermometer agrees with the conventional one”?
No. Even if it always reads 0.5 °C higher, the correlation coefficient remains high. Also, if you collect a wide range of people, from febrile patients to people with normal temperature, the correlation coefficient becomes high. Use a Bland-Altman plot and compare the mean difference and limits of agreement with a pre-specified acceptable range (e.g., ±0.3 °C) (Chapters 1 and 2).
The ICC measured in 30 healthy university students was 0.55, and the ICC measured in 30 inpatients was 0.90. Can you say that measurement precision is higher in patients?
No. The ICC depends on the variability between people, and it becomes high in a patient population with a wide range of values. Measurement precision itself should be compared with the within-person SD or the repeatability coefficient (in the same units as the measurement) (Chapter 3).
People whose blood pressure was 160 mmHg or higher at the first measurement were gathered and given salt-reduction counselling; one month later their mean blood pressure had fallen by 8 mmHg. Can you call this an effect of the counselling?
No. People who were high on a single measurement tend to be lower at the next measurement even if nothing is done (regression to the mean). In NHANES, the reading fell by about 4 mmHg just at the 2nd measurement on the same day. You need a control group for comparison (Chapter 3).
The percentage agreement between two pathologists’ assessments was 96%, but κ was 0.40. How do you interpret this?
Because “positive” (or “negative”) is very rare (common), the percentage agreement is likely high even by chance. The low κ may indicate poor agreement in the rarer category. Check the proportion in each category and the positive and negative agreement (Chapter 4).
A new blood test for cancer was evaluated in 100 patients with advanced cancer and 100 healthy young blood donors, giving a sensitivity of 95% and a specificity of 98%. What would you expect if this test were introduced into general health screening?
Among actual screening attendees, many of whom have early cancer (small changes in test values) or diseases other than cancer (which can raise test values), both sensitivity and specificity are likely to be lower (spectrum bias). Moreover, because prevalence is low in screening, the positive predictive value will be considerably lower (Chapter 5).
Using data on 1000 people, a prediction model was built by selecting variables from 50 candidates, and its AUC on the same data was 0.88. How should this be written up in a paper?
The AUC on the same data is optimistic. Either correct for optimism by repeating the whole procedure, including variable selection, with cross-validation or the bootstrap, or report the AUC from external validation on separate data. Also show calibration (agreement between predicted and observed). Refer to reporting guidelines such as TRIPOD (Chapter 6).
7.7 Closing remarks
In evaluating agreement and diagnostic accuracy, the results for the same test can differ greatly depending on
- what is being compared with what (two measurement methods, or is there a correct answer?)
- in whom it was evaluated (range of values, prevalence, severity)
- on which data it was checked (the same data used to build it?)
Rather than a single number (correlation coefficient, κ, AUC), report a combination of plots, multiple measures, and a description of the population evaluated.