1 High correlation = agreement?
Code cells can be edited and re-run by pressing “Run Code” (R is running inside your browser). The first time, loading the packages takes several tens of seconds.
When comparing two measurement methods, the most commonly seen (and most misunderstood) statistic is the correlation coefficient. In this chapter, we check what the correlation coefficient says, and does not say, about “agreement”.
1.1 Data: the same person’s blood pressure, 3 times
In NHANES, the same person’s blood pressure is measured 3 times, about 30 seconds apart, at the examination.
Since the same person is measured in succession with the same device, the 1st and 2nd readings should be “almost the same”.
The correlation coefficient is 0.95. So far, everything looks fine.
1.2 Evaluating new blood pressure monitors (simulation)
Suppose a company has developed “new blood pressure monitors” and runs a study comparing them with the current monitor. Here, we take the mean of the 3 NHANES readings as “the value of the current monitor”, and create the readings of the new monitors by simulation.
- Device A: low blood pressures read high and high blood pressures read low (slope 0.8)
- Device B: always reads 10 mmHg high
- Device C: correct on average, but with large variability
Devices A and B have correlation coefficients as high as 0.96–0.98. Can they be used in place of the conventional monitor?
- Device A: measures a person with a systolic pressure of 180 mmHg as 174 mmHg. It underestimates the severity of hypertension
- Device B: measures everyone 10 mmHg high. Many people with normal blood pressure are classified as “hypertensive”
- Device C: the correlation coefficient is 0.77, but each individual’s value is off by about ±30 mmHg
None of these can be used in place of the conventional monitor. Yet the correlation coefficients of Device A (0.96) and Device B (0.98) are higher than that between the 1st and 2nd readings of the same monitor (0.95).
1.3 What does the correlation coefficient measure?
The correlation coefficient measures whether two values “lie on a straight line”. It does not look at whether that line is y = x (agreement).
All of these have a correlation coefficient of 1 (perfect correlation), but they do not agree.
1.4 The correlation coefficient changes with the range of people included
The correlation coefficient has another major property: the wider the range of values among the people measured, the larger it becomes. Using the same 1st and 2nd readings, restrict to people whose mean is 110–130 mmHg.
The measurement discrepancy (SD of the differences) hardly changes, yet the correlation coefficient fell from 0.95 to 0.60. The correlation coefficient reflects not “the quality of the measurement” but the ratio of “variability between people” to “measurement discrepancy”. It is low if evaluated only in healthy people, and high if severely ill patients are included.
1.5 “No significant difference” is not agreement either
Another common practice is to compare the values of two measurement methods with a paired t-test and conclude that “they agree because there was no significant difference”.
For Device C, the mean difference is almost 0, so the t-test shows no significant difference. Yet each individual’s value is off by as much as ±30 mmHg. The t-test only looks at whether the means differ, not at how far each individual’s value is off. Conversely, with many people, even a clinically meaningless 1 mmHg discrepancy becomes “significant”.
1.6 So what should we look at?
To evaluate agreement, look directly at each individual’s difference (new − conventional).
- Mean difference: systematic discrepancy (+10 mmHg for Device B)
- SD of the differences and limits of agreement: how far each individual’s value can be off (±30 mmHg for Device C)
The figure of this is the Bland-Altman plot in the next chapter.
- The correlation coefficient looks at “whether values lie on a line”, not at whether it is y = x (agreement). Always 10 higher or twice as large still gives a correlation of 1
- The correlation coefficient changes greatly with the range of people measured. Collect people over a wide range and it can be made as high as you like
- “No significant difference” in a paired t-test says only that the means do not differ
- Evaluate agreement by the mean (systematic error) and spread (limits of agreement) of each individual’s difference