4  r×c tables: where, and by how much, do they differ?

For tables larger than 2×2, the mechanism of the χ² test is the same. However, even if you know the p-value is small, the p-value does not tell you where in the table the difference is or how large it is. In this chapter, we practise “reading” the result of a χ² test with residuals and effect sizes.

4.1 Education and self-rated health

Among NHANES 2009–2012 participants aged 20 or over, we look at the relationship between highest level of education (5 categories) and self-rated health (5-level answer to “How is your health?”).

The χ² value is 1433 (degrees of freedom (5 − 1)(5 − 1) = 16), and the p-value is below 2.2×10⁻¹⁶ (R does not display smaller values). We now know that “education and self-rated health are related”, but this alone does not tell us which answers are more common in which education group.

NoteMissing values are dropped automatically

table() does not count people with NA in either variable. Of the 11778 people aged 20 or over, 10039 entered the table. To check who was dropped, see Chapter 4 of the descriptive statistics hands-on.

4.2 Where is the difference? Residuals

4.2.1 Look at row proportions

First, naively, look at the proportions of answers by education.

We can see a trend: the higher the education, the more “Excellent” and “Vgood (very good)” answers, and the fewer “Fair” and “Poor” answers.

4.2.2 Standardized residuals

The residual expresses, for each cell, the gap between observed and expected as “how many times the chance variation”. The result of chisq.test() contains two kinds.

  • Pearson residuals residuals: (O − E) / √E. Squaring and summing them gives the χ² value
  • Standardized residuals (adjusted residuals) stdres: the Pearson residuals further scaled by each cell’s standard error. Under independence they follow approximately a standard normal distribution

As a guide, a standardized residual beyond ±2 means “too many (too few) to be chance”.

  • 8th Grade (education up to middle school): Fair (+20.9) and Poor (+11.8) are more common
  • College Grad (college graduate): Excellent (+12.7) and Vgood (+19.0) are more common, and Fair (−16.1) is less common
  • High School and Some College are close to the overall average
WarningResiduals are the same as doing 25 tests

If you judge the 25 cells’ residuals with ±2 (about 5%), roughly one will look “significant” even with no relationship. For a stricter judgement, use the value corresponding to the significance level divided by the number of cells (Bonferroni correction).

In this table, the conclusions hardly change even when judging with ±3.09.

4.2.3 Mosaic plot

mosaicplot() makes the area of each cell proportional to the number of people and colours it by the size of the residual (solid blue: too many; dashed red: too few). What is used for the colouring is, unlike the label in the legend (Standardized Residuals), the Pearson residual (O − E)/√E. The colour intensity changes at ±2 and ±4.

4.3 How large is the difference? Effect size

4.3.1 Cramér’s V

The χ² value grows in proportion to the number of people (Chapter 3). Cramér’s V expresses the strength of the relationship in a form that does not depend on the number of people.

\[ V = \sqrt{\frac{\chi^2}{N \times (\min(r, c) - 1)}} \]

0 means completely independent; 1 means one variable completely determines the other.

V = 0.19. As a guide — the criteria vary with the degrees of freedom and so on — roughly 0.1 is a small, 0.3 a medium, and 0.5 a large association (Cohen’s criteria). The association between education and self-rated health is “small to medium”. A p-value smaller than 10⁻¹⁶ does not mean the association is very strong.

4.3.2 For a 2×2 table, express it as a difference or ratio

For 2×2 tables, expressing results as the risk difference, risk ratio, or odds ratio with their confidence intervals is easier to interpret clinically than Cramér’s V. Let us compute these for sex and physical activity (“Do you do moderate- or vigorous-intensity sports or exercise?”).

The confidence interval from prop.test() is “row 1 − row 2” (female − male), so it is negative. Reversing the direction, the proportion of men who exercise is 7.0 percentage points higher than that of women (95% CI 5.2 to 8.8), and the risk ratio is 1.16 (1.12 to 1.21). This conveys far more than “p = 3×10⁻¹⁴”.

4.4 In large samples, p-values pick up even small differences

With the same sex variable, compare the proportion with diabetes.

The difference is 1.1 percentage points, p = 0.09. What if the proportions stayed the same but there were 4 times as many people?

p = 0.0007, “highly significant”. The size of the difference (1.1 percentage points) has not changed at all. Conversely, if you study a small number of people, even large differences are not significant.

Let us randomly select only 200 people and compare sex and physical activity again.

TipWhen reporting
  • Not only the p-value, but the number and proportion in each group (the table itself)
  • For a 2×2 table, the difference or ratio with a 95% confidence interval; for a larger table, Cramér’s V or the proportions in the cells of interest
  • If you state where the difference is, use standardized residuals or pre-specified comparisons (Chapter 8)

With 200 people, the difference between men and women is 7.1 percentage points — almost the same as in the whole sample — yet p = 0.32, not significant.

4.5 The χ² test does not look at “order”

Education and self-rated health are both ordered categories. However, the χ² test does not care about the order of the rows or columns.

If you want to examine a trend such as “the higher the education, the more people say they are healthy”, a test that uses the order makes it easier to find a difference even with fewer people (Chapter 7).

ImportantSummary of this chapter
  • The χ² test for an r×c table only says “there is a relationship somewhere”
  • Where: row proportions and standardized residuals (±2, or Bonferroni-corrected for stricter judgement), mosaic plots
  • How much: Cramér’s V; for a 2×2 table, risk difference, risk ratio, odds ratio with confidence intervals
  • In large samples, even meaninglessly small differences become significant. In small samples, even large differences do not
  • The χ² test ignores order