Measurement Agreement and Diagnostic Accuracy: A Hands-on Course

From “high correlation = agreement” to Bland-Altman, ICC, κ, sensitivity/specificity, and ROC — with real data

Published

October 18, 2026

Introduction

English version — machine-assisted translation (by generative AI, Anthropic Claude) of the original Japanese hands-on. 日本語版はこちら.

Can a new blood pressure monitor be used in place of the current one? How well do two doctors’ diagnoses agree? How well can a blood glucose value from a health check-up detect diabetes?

Evaluations of measurement agreement and diagnostic accuracy like these are carried out almost daily in clinical research and device development. At the same time, we often see plausible-looking but wrong evaluations, such as:

  • Concluding that two measurement methods “agree well because their correlation coefficient was 0.95”
  • Reporting that raters’ judgements agree well because the percentage agreement is 95%
  • Thinking that with a highly sensitive and specific test, people who test positive almost certainly have the disease
  • Reporting the performance of an “optimal cut-off” chosen from the data on the same data

In this hands-on, using real data in which the same people’s blood pressure was measured 3 times (NHANES) and the teaching dataset of the Framingham Heart Study, and working with R running in your browser, you will experience how to evaluate agreement and diagnostic accuracy correctly.

Goals of this hands-on

  1. Explain why the correlation coefficient is not a measure of agreement
  2. Evaluate systematic error, limits of agreement, and proportional error with a Bland-Altman plot
  3. Obtain within-person variability and the ICC from repeated measurements, and explain that the ICC depends on the population
  4. Evaluate categorical agreement with the κ coefficient, and explain its pitfalls
  5. Explain the relationship between sensitivity, specificity, and positive predictive value, and the influence of prevalence
  6. Evaluate ROC curves, AUC, and cut-offs without being over-optimistic

Who this is for

  • People involved in analysing clinical research, device development, or healthcare data
  • People who understand means, standard deviations, correlation coefficients, and proportions (roughly the level of Japan’s Statistics Certificate Grade 3)
  • For R, it is enough if you can “read code, modify it a little, and run it”

R runs in your browser

The code cells in each chapter are executed by R inside your browser (a technology called webR). Edit the code and press “Run Code” and the result changes immediately. No installation is needed.

Note
  • Right after opening each chapter, loading R and the packages takes from several tens of seconds to about a minute
  • If you move the tab to the background while loading, processing slows down, so please wait until it is displayed
  • Viewing on a PC is recommended

ICC, κ, ROC, and so on are computed by ourselves from their definitions, without dedicated packages (small functions are collected in R/setup.R).

Data used

Data Contents Main chapters
NHANES 2009–2012 (bp) 11778 people aged 20+ from a US health and nutrition survey. Blood pressure measured 3 times at one examination (systolic and diastolic) 1–4
Framingham (fram) 4434 people at the 1st examination of a US heart study. Casual blood glucose, diabetes, cardiovascular disease over 24 years, etc. 5, 6
WarningNotes on the data
  • NHANES is a survey that oversampled certain racial groups and age groups. Here we tabulate without weights, so the results are not values for the US population
  • Part of the Framingham teaching dataset has been modified for anonymisation. It cannot be cited as research findings
  • The readings of the “new blood pressure monitors” are created by simulation for teaching purposes (this is stated explicitly in the chapters)

Chapter overview

Chapter Contents
1 High correlation = agreement? (what the correlation coefficient does not look at)
2 Bland-Altman plots (systematic error, limits of agreement, proportional error)
3 Repeated measurements and reliability (within-person SD, ICC, regression to the mean)
4 Categorical agreement: the κ coefficient (chance agreement, influence of prevalence, weighted κ)
5 Diagnostic accuracy: sensitivity, specificity, predictive values (prevalence, spectrum bias)
6 ROC curves and AUC (optimism in choosing cut-offs, calibration of prediction models)
7 Exercises and a checklist for agreement and diagnostic accuracy