6  Non-independent data: McNemar’s test and “many rows per person”

The χ² test assumes that each person is counted in only one place in the table, and that people are independent of each other. If you apply chisq.test() as is to data in which the same person was measured twice, or in which one person has many rows, this assumption breaks down.

6.1 The same people’s smoking, 6 years later

In Framingham, the same people were examined about every 6 years. Among people who attended both the 1st (PERIOD 1) and the 3rd (PERIOD 3) examinations, we compare whether they smoked.

6.1.1 A common mistake: comparing them as two groups

People sometimes put the smoking rates at the 1st and 3rd examinations into a table of “two groups” and run a χ² test.

In this table, the same people are counted twice each, and the 1st and 3rd examinations are treated as independent, different people.

6.1.2 The correct table: count each person once

Build the table by pairing the same person’s 1st and 3rd examinations.

  • 1647 people: non-smokers throughout
  • 1059 people: smokers throughout
  • 495 people: smoker → non-smoker (quit)
  • 62 people: non-smoker → smoker (started or resumed)

Only the people whose status changed (discordant pairs) are relevant to whether the smoking rate changed. The 2706 people whose status stayed the same carry no information about change.

6.2 McNemar’s test

If “the smoking rate has not changed”, the numbers who quit and who started should be about the same. For the discordant pairs b = 495 and c = 62, McNemar’s test compares

\[ \chi^2 = \frac{(b - c)^2}{b + c} \]

with the χ² distribution with 1 degree of freedom.

This is the same as testing whether the proportion who “quit” among the 557 discordant people is 50%, so an exact p-value can also be computed with the binomial distribution (use it when there are few discordant pairs).

NoteIn this example, both tests are highly significant

Because there are many people and the change is clear, here both the wrong χ² test and McNemar’s test give very small p-values. However, the wrong test asks a different question (it compares different populations of people), and with fewer people the conclusion can change.

6.3 An example where the conclusion changes

A fictitious example. 40 people received new lifestyle guidance and were asked “Do you eat breakfast?” before the guidance and 3 months later.

16 people ate breakfast before the guidance, and 24 after.

Treating before and after as two independent groups gives p = 0.07, not significant, but McNemar’s test gives p = 0.01 (0.02 even with the exact test). The same person’s before and after are similar (people who eat breakfast usually still do afterwards), so looking only at change has less variability and makes differences easier to find. Conversely, if many people change in opposite directions, the wrong test can give a smaller p-value. Either way, they answer different questions.

TipWhen to use McNemar’s test
  • Before and after in the same people (presence of symptoms before and after treatment, behaviour before and after an intervention)
  • Comparing positivity rates when two diagnostic methods are applied to the same people (comparing sensitivity and specificity)
  • Matched case-control studies (one case and one control form a pair)

6.4 Data with many rows per person

In the Framingham data, each person’s 3 examinations are stacked “vertically”. Tabulating as is counts the same person many times.

Using all rows increases the χ² value from 175 to 254. Because the same person’s smoking hardly changes across the 3 examinations, this is effectively counting the same information several times, inflating the number of people.

6.4.1 Simulation: when there is truly no difference

We create data in which there is truly no difference between 2 groups. The “propensity to smoke” of the 200 people (100 per group) differs between people, and when each person is measured 3 times, the same person’s results are similar.

Testing with one row per person keeps the type I error at around 5% (4.7%). Using all rows makes the result “significant” much more often than 5% (about 13%), even though there is truly no difference.

WarningWhat to do
  • Before testing, check what one row is, and align to one row per person (baseline only, last observation only, aggregate per person as “ever …”, etc.)
  • If you want to use all repeated measurements, use methods that account for within-person correlation (GEE, generalized linear mixed models). See also the linear mixed models hands-on
  • The same thing happens with multiple teeth, eyes, or lesions of one person, and with people from the same hospital or family
ImportantSummary of this chapter
  • The χ² test assumes “each person in one place, people independent of each other”
  • Before/after in the same people and matched pairs: McNemar’s test. Look only at the people who changed (discordant pairs). If they are few, use binom.test()
  • Running a χ² test treating before and after as two independent groups asks a different question, and the conclusion can change
  • Testing data with many rows per person as is inflates the number of people and easily produces false significant differences