Measurement Agreement and Diagnostic Accuracy: A Hands-on Course
From “high correlation = agreement” to Bland-Altman, ICC, κ, sensitivity/specificity, and ROC — with real data
Introduction
English version — machine-assisted translation (by generative AI, Anthropic Claude) of the original Japanese hands-on. 日本語版はこちら.
Can a new blood pressure monitor be used in place of the current one? How well do two doctors’ diagnoses agree? How well can a blood glucose value from a health check-up detect diabetes?
Evaluations of measurement agreement and diagnostic accuracy like these are carried out almost daily in clinical research and device development. At the same time, we often see plausible-looking but wrong evaluations, such as:
- Concluding that two measurement methods “agree well because their correlation coefficient was 0.95”
- Reporting that raters’ judgements agree well because the percentage agreement is 95%
- Thinking that with a highly sensitive and specific test, people who test positive almost certainly have the disease
- Reporting the performance of an “optimal cut-off” chosen from the data on the same data
In this hands-on, using real data in which the same people’s blood pressure was measured 3 times (NHANES) and the teaching dataset of the Framingham Heart Study, and working with R running in your browser, you will experience how to evaluate agreement and diagnostic accuracy correctly.
Goals of this hands-on
- Explain why the correlation coefficient is not a measure of agreement
- Evaluate systematic error, limits of agreement, and proportional error with a Bland-Altman plot
- Obtain within-person variability and the ICC from repeated measurements, and explain that the ICC depends on the population
- Evaluate categorical agreement with the κ coefficient, and explain its pitfalls
- Explain the relationship between sensitivity, specificity, and positive predictive value, and the influence of prevalence
- Evaluate ROC curves, AUC, and cut-offs without being over-optimistic
Who this is for
- People involved in analysing clinical research, device development, or healthcare data
- People who understand means, standard deviations, correlation coefficients, and proportions (roughly the level of Japan’s Statistics Certificate Grade 3)
- For R, it is enough if you can “read code, modify it a little, and run it”
R runs in your browser
The code cells in each chapter are executed by R inside your browser (a technology called webR). Edit the code and press “Run Code” and the result changes immediately. No installation is needed.
- Right after opening each chapter, loading R and the packages takes from several tens of seconds to about a minute
- If you move the tab to the background while loading, processing slows down, so please wait until it is displayed
- Viewing on a PC is recommended
ICC, κ, ROC, and so on are computed by ourselves from their definitions, without dedicated packages (small functions are collected in R/setup.R).
Data used
| Data | Contents | Main chapters |
|---|---|---|
NHANES 2009–2012 (bp) |
11778 people aged 20+ from a US health and nutrition survey. Blood pressure measured 3 times at one examination (systolic and diastolic) | 1–4 |
Framingham (fram) |
4434 people at the 1st examination of a US heart study. Casual blood glucose, diabetes, cardiovascular disease over 24 years, etc. | 5, 6 |
- NHANES is a survey that oversampled certain racial groups and age groups. Here we tabulate without weights, so the results are not values for the US population
- Part of the Framingham teaching dataset has been modified for anonymisation. It cannot be cited as research findings
- The readings of the “new blood pressure monitors” are created by simulation for teaching purposes (this is stated explicitly in the chapters)
Chapter overview
| Chapter | Contents |
|---|---|
| 1 | High correlation = agreement? (what the correlation coefficient does not look at) |
| 2 | Bland-Altman plots (systematic error, limits of agreement, proportional error) |
| 3 | Repeated measurements and reliability (within-person SD, ICC, regression to the mean) |
| 4 | Categorical agreement: the κ coefficient (chance agreement, influence of prevalence, weighted κ) |
| 5 | Diagnostic accuracy: sensitivity, specificity, predictive values (prevalence, spectrum bias) |
| 6 | ROC curves and AUC (optimism in choosing cut-offs, calibration of prediction models) |
| 7 | Exercises and a checklist for agreement and diagnostic accuracy |