viewof nSub = Inputs.range([10, 300], { value: 100, step: 10, label: "Number of measurements" })
viewof hrLo = Inputs.range([40, 100], { value: 50, step: 5, label: "Heart-rate range: lower limit (bpm)" })
viewof hrHi = Inputs.range([60, 200], { value: 170, step: 5, label: "Heart-rate range: upper limit (bpm)" })
viewof bias = Inputs.range([-10, 10], { value: 0, step: 0.5, label: "Systematic error (bpm)" })
viewof prop = Inputs.range([-20, 20], { value: 0, step: 1, label: "Proportional error (%)" })
viewof sdErr = Inputs.range([0, 15], { value: 3, step: 0.5, label: "Random error SD (bpm)" })
viewof hiNoise = Inputs.toggle({ label: "Larger error at higher heart rates (during exercise)", value: false })8 Evaluating Measurement Accuracy
Heart rate from a new wearable, a home-made ECG circuit, a new algorithm —— to show “how correct this value is”, you need to measure simultaneously with a reference method (the gold standard, or reference method) and compare. This chapter covers Bland-Altman analysis, the method most commonly used for this, and common mistakes.
8.1 The correlation coefficient does not express “agreement”
When comparing two methods, showing only a scatter plot and the correlation coefficient r is a common mistake.
- The correlation coefficient only expresses “the strength of a linear relationship”, not whether the values agree
- Even if one method is always 10 bpm higher, or always 1.2 times the other, the correlation coefficient can be close to 1
- The wider the measurement range, the larger the correlation coefficient (if heart rates from rest to maximal exercise are included, r is high even when accuracy is poor)
8.2 Bland-Altman analysis
In the method proposed by Bland and Altman (1986), for each subject you plot
- horizontal axis: the mean of the two methods (A + B) / 2
- vertical axis: the difference between the two methods A - B
and compute the following values.
| Metric | Calculation | Meaning |
|---|---|---|
| Systematic error (bias) | Mean of the differences \bar{d} | How much higher (lower) one method is on average |
| Limits of agreement (LoA) | \bar{d} \pm 1.96\, s_d | The range containing about 95% of the differences between individual measurements |
Whether the LoA fall within a clinically or practically acceptable range is judged against criteria decided in advance (it is not something decided by a statistical test).
8.2.1 Demo
We simulate the values of the device under evaluation (e.g. wrist PPG) against heart rates from the reference method (e.g. chest ECG).
The orange dotted line is the regression of the difference on the mean (it tilts when there is proportional error).
TipTry it
- Set the systematic error to +8 bpm. How does the correlation coefficient r change? What changes in the Bland-Altman plot?
- Reset the systematic error to 0 and set the proportional error to +10%. What pattern appears in the Bland-Altman plot?
- Keeping the random error SD at 5 bpm, compare the heart-rate range “60–80 bpm (rest only)” with “50–190 bpm (including exercise)”. How does r change for a device of the same accuracy?
- Turn on “Larger error at higher heart rates”. Are the LoA representative of the whole range?
NoteExplanation
- Systematic error does not change the correlation coefficient at all (everything is just shifted). In the Bland-Altman plot, the bias line moves away from 0.
- With proportional error, an upward-sloping pattern appears in which the difference grows with the mean. In that case it is inappropriate to summarize accuracy with the overall bias and LoA alone; consider correction by regression, or analysis on a log scale or as ratios.
- The wider the measurement range, the closer the correlation coefficient gets to 1 for the same error. This is a classic example of a high r not proving accuracy.
- When the size of the error depends on the magnitude of the measurement (heteroscedasticity), the LoA are too wide at low heart rates and too narrow at high heart rates. It is preferable to report separately for each condition (rest, exercise, etc.).
8.3 Points to watch when designing a validation study
- Choice of reference method: for heart rate, an ECG (preferably a medical-grade ECG rather than a chest strap); for SpO₂, arterial blood gas analysis
- Time synchronization: devices differ in averaging windows and display latency. Compare values over the same time intervals
- Conditions: include the actual conditions of use, such as rest, daily activities, exercise (at different intensities) and sleep
- Diversity of participants: age, sex, skin colour, body build, diseases (arrhythmias, etc.)
- Repeated measurements: when multiple values are taken from the same person, treating them as independent data underestimates the width of the LoA → use a Bland-Altman method that accounts for repeated measurements
- Pre-specified acceptance criteria: decide before the analysis “how much error is acceptable for the intended use”
8.3.1 Other metrics
| Metric | Use | Caveats |
|---|---|---|
| MAE / MAPE | Summarize the size of the error in a single number. Widely used in wearable evaluation (some propose MAPE below 10% as a rule of thumb) | Cannot distinguish systematic from random error |
| Intraclass correlation coefficient (ICC) | Agreement between methods or raters; reproducibility (reliability) | Affected by the width of the measurement range |
| Coefficient of variation (CV) | Variability of repeated measurements under the same conditions (reproducibility) | |
| Se / PPV | Evaluating detection (R waves, arrhythmias) (Chapter 7) | Depend on the tolerance window and the definition of the reference |
8.4 Summary
- The correlation coefficient does not express agreement. Systematic error does not show up in r, and a wide range makes r high
- Bland-Altman analysis: plot difference vs. mean and compute the bias and the limits of agreement (±1.96 SD)
- Watch for patterns of proportional error and heteroscedasticity
- Decide the acceptable error criteria in advance, based on the intended use