4  Categorical agreement: the κ coefficient

We now consider agreement when the result of a measurement is a category, such as “hypertensive or not” or “an imaging finding present or absent”. Examples include the judgements of two doctors, two judgements of the same person, or the results of an old and a new test.

4.1 Percentage agreement

For the 1st and 2nd systolic readings, classify each as “140 mmHg or higher (high)” or not.

Of the 10091 people, the classifications agreed for 94.7%.

4.1.1 The problem with percentage agreement: agreement also happens by chance

Even if the 2nd classification were decided at random, completely independently of the 1st, with the same proportion (about 17% “high”),

  • the probability of “<140” both times: 0.82 × 0.83
  • the probability of “140+” both times: 0.18 × 0.17

add up to about 71% agreement by chance alone.

In a population where most people are “<140”, classifying everyone as “<140” without thinking would still give high percentage agreement. Percentage agreement alone does not tell you how good the classification is.

4.2 Cohen’s κ coefficient

The κ coefficient subtracts the agreement expected by chance and expresses “how much of the maximum possible beyond-chance agreement was actually achieved”.

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]

(\(p_o\): observed agreement, \(p_e\): agreement expected by chance)

κ = 0.82. 1 means perfect agreement; 0 means agreement no better than chance. According to a commonly used guideline (Landis & Koch), 0.61–0.80 is “substantial agreement” and 0.81 or more “almost perfect agreement” (this guideline has little basis, so treat it only as a reference).

4.2.1 Different percentage agreement, same κ

Let us split by age.

Percentage agreement differs greatly — 98.5% under 40 and 90.0% at 60 and over — but κ is 0.78 in both. This is because under 40 there are hardly any people at “140+”, so 93% agreement occurs even by chance.

4.3 A pitfall of κ: the prevalence paradox

Conversely, κ can differ greatly even when percentage agreement is the same. The following two tables both have 90% agreement.

In table A, where “present” is common (85%), the chance agreement is high (74.5%), so κ stays at 0.61. In table B, where “present” and “absent” are half and half, κ = 0.80.

Warningκ depends on prevalence (the proportion in each category)
  • In populations with extreme prevalence (almost all “present” or “absent”), κ is low even if raters agree well
  • Do not directly compare κ from different populations or different studies
  • Along with κ, also report percentage agreement, the proportion in each category, and if possible agreement on “present” and on “absent” separately (positive and negative agreement)

In table A, we see that agreement on “present” is high at 94%, and agreement on “absent” is low at 67%. This conveys more information than a single κ.

4.4 Ordered categories: weighted κ

If blood pressure is divided into 4 levels, “off by one level” and “off by three levels” are not the same disagreement.

Most disagreements are shifts to the adjacent category. Weighted κ gives “partial agreement” according to the size of the discrepancy.

  • Unweighted (even a shift to the adjacent category is complete disagreement): κ = 0.67
  • Linear weights (off by one level = 2/3 agreement): κ = 0.80
  • Quadratic weights (off by one level = 8/9 agreement): κ = 0.90

κ changes greatly depending on the choice of weights, so always state which weights you used. κ with quadratic weights is known to be almost the same as the ICC (Chapter 3).

TipDividing a continuous variable makes agreement look worse

Agreement on the 4-level categories (unweighted κ 0.67) looks much lower than agreement on the original continuous values (ICC 0.95). People near the cut-points change category with a slight discrepancy. If the measurement is originally continuous, evaluate agreement on the values before categorisation (Chapters 2 and 3).

ImportantSummary of this chapter
  • Percentage agreement includes chance agreement. κ = (agreement − chance agreement) / (1 − chance agreement)
  • κ depends on the proportions in each category (prevalence). κ from different populations cannot be compared. Also report percentage agreement, proportions, and positive/negative agreement
  • For ordered categories, use weighted κ. Its value changes greatly with the type of weights, so state it
  • Dividing a continuous variable makes agreement look worse