3  Repeated measurements and reliability: within-person SD, ICC, regression to the mean

Chapter 2 asked “do two measurement methods agree?”. This chapter covers how much measurements vary when the same person is measured repeatedly with the same method (repeatability, reliability). Even when evaluating a new measurement method, unless you first know how much the conventional method itself varies, you cannot judge whether “the discrepancy of the new method” is large or small.

3.1 Within-person SD: variability when measuring the same person

Using the 3 NHANES systolic readings, compute the variability (variance) of each person’s 3 values, and average over everyone.

The within-person SD is 4.3 mmHg. Even measuring the same person with the same monitor 30 seconds apart, the readings vary this much.

3.1.1 Repeatability coefficient: how different can two measurements be?

The SD of the difference between two measurements is √2 × within-person SD, so for 95% of people the difference between two measurements falls within 1.96 × √2 × within-person SD ≈ 2.77 × within-person SD. This is called the repeatability coefficient.

“If the same person is measured twice, the difference is within 12 mmHg with 95% probability”. Even if blood pressure is 10 mmHg lower than last time, it may be within the range of measurement variability.

3.1.2 Does variability differ with the level of blood pressure?

The higher the blood pressure, the slightly larger the within-person SD (3.9 → 5.6 mmHg).

3.2 ICC: the intraclass correlation coefficient

The ICC (intraclass correlation coefficient) is the proportion of the total variability of the measurements that is due to differences between people.

\[ \mathrm{ICC} = \frac{\sigma^2_{\text{between people}}}{\sigma^2_{\text{between people}} + \sigma^2_{\text{within people}}} \]

The closer to 1, the more “differences in the measured values are almost all due to differences between people”, meaning a high ability to distinguish people (reliability).

The ICC is 0.945.

3.2.1 The ICC changes with the people included

Like the correlation coefficient in Chapter 1, the ICC also depends strongly on variability between people. Let us compute it only for people whose mean of the 3 readings is below 120 mmHg.

The within-person SD (the variability of the measurement itself) changes little, at 3.9–4.7 mmHg, yet the ICC fell from 0.95 to 0.76. This is because in a population with little variability between people, the same measurement “has a harder time distinguishing people”.

WarningWhen reporting the ICC
  • The ICC changes with the characteristics of the population. Also report whom you measured (range and SD of the values)
  • To show the precision of the measurement itself, also present the within-person SD or repeatability coefficient (in the same units as the measurement)
  • Rules of thumb such as “ICC > 0.75 is good” cannot be compared across different populations

3.2.2 There are several kinds of ICC

The ICC has multiple definitions to suit the measurement situation (Shrout & Fleiss, McGraw & Wong).

Type Situation Systematic differences between occasions
One-way No distinction of rater, device, or occasion Included in error
Two-way, consistency The same raters measure everyone. It is enough if the ranking is preserved Not included in error
Two-way, absolute agreement We want the values themselves to agree Included in error

For blood pressure, the differences in means between readings (the 1st is about 2 mmHg higher) are small, so both are almost the same. When one of two raters always scores higher, it can happen that consistency is high but absolute agreement is low. Always state which ICC you used.

3.3 Measure several times and average

Within-person variability becomes smaller if you measure several times and average. The within-person SD of the mean of k measurements is within-person SD / √k.

This is why office blood pressure is measured 2–3 times and averaged.

3.4 Regression to the mean

Let us look at the 2nd and 3rd readings of people whose 1st blood pressure was 160 mmHg or higher.

People whose 1st reading was high dropped by 3.8–6.4 mmHg at the 2nd and 3rd readings without doing anything, and people whose 1st reading was low rose by 1.5 mmHg. This is because the high group contains many people whose 1st reading “happened to come out high” (regression to the mean).

WarningImplications for clinical research

If you recruit “people whose blood pressure was high on a single measurement” and treat them, their blood pressure will be lower at the next measurement even without treatment. Concluding “treatment lowered blood pressure” from a before–after comparison without a control group mistakes regression to the mean for a treatment effect. Countermeasures include having a control group, using different measurements for eligibility and for outcome assessment, and enrolling based on the mean of several measurements.

3.5 Measurement error weakens associations (regression dilution)

Measurement variability weakens the strength of associations (regression coefficients) with other variables towards 0. Let us check by simulation. Suppose that when “true blood pressure” is 10 mmHg higher, some indicator rises by 1.

With a single measurement, the slope is weakened to about 0.94 times the true value (= the reliability). With a measurement of reliability 0.5, the association appears halved. For measurements with large within-person variability, such as nutrient intake or physical activity, this effect is large.

ImportantSummary of this chapter
  • The within-person SD and the repeatability coefficient (2.77 × within-person SD) show the precision of the measurement itself in the same units as the measurement
  • The ICC is the proportion due to “differences between people”. It changes with the variability of the population, so report it together with the population’s characteristics. Also state the type (one-way, consistency, absolute agreement)
  • People selected for one high value will be lower at the next measurement (regression to the mean). Before–after comparisons need a control group
  • Measurement error weakens associations (regression dilution). Averaging several measurements reduces it