9  Exercises and a checklist for choosing a test

Let us put what we covered in the previous chapters into practice with our own hands. The code cells in this chapter are not run automatically. Fill in the blanks (___) and press “Run Code”. Example answers are collapsed below each question.

9.1 Checklist for choosing a test

Before testing categorical data, check the following in order.

# What to check What to choose Chapter
1 Does the table contain numbers of people (frequencies)? Have you entered proportions? — 1
2 Is each person counted in only one place in the table (before/after in the same person, multiple rows, multiple lesions)? Before/after or pairs → McNemar. Multiple rows → align to one row per person, or GEE / mixed models 6
3 What is the question: compare one variable with theoretical proportions / the relationship between two variables Goodness of fit chisq.test(x, p =) / independence chisq.test(table) 3, 1
4 Did you estimate the parameters of the theoretical proportions from the data? Reduce the degrees of freedom by the number estimated 3
5 How large are the expected frequencies (none below 1, no more than 20% below 5)? Small → Fisher’s exact test or simulate.p.value 5
6 For a 2×2 table, do you apply Yates’ correction (R applies it by default)? Decide in the analysis plan and state it explicitly 5
7 Is there a confounder? Does group composition differ between strata? Stratify → Mantel-Haenszel; many variables → logistic regression 7
8 Are the categories ordered, and do you want to look at a trend? Trend test prop.trend.test() 7
9 Did you show where and by how much they differ? Standardized residuals, Cramér’s V, difference/ratio with confidence interval 4
10 Did you compare 3 or more groups pairwise? Did you do subgroup analyses? Correct with Holm or similar 8
11 Did you compute the required number in advance? power.prop.test(), noncentral χ² distribution 8
12 Did you decide the test, correction, and category merging before looking at the data? Write it in the analysis plan 5, 7, 8

9.2 Exercise 1: Is there a sex difference in smoking history?

Among NHANES participants aged 20 or over, compare “Have you smoked at least 100 cigarettes in your life?” (Smoke100) between men and women.

9.2.1 1-1. Table and expected frequencies

Make a table of sex × smoking history, and compute the expected frequencies yourself.

E1 <- outer(rowSums(tab1), colSums(tab1)) / sum(tab1)

The expected frequencies of ever-smokers are about 2681 for women and 2554 for men (the overall proportion of 44.5% multiplied by the numbers of women and men).

9.2.2 1-2. Test and effect size

Compute the χ² value and check that it matches chisq.test(), then find the difference in proportions and its 95% confidence interval.

sum((tab1 - E1)^2 / E1)
chisq.test(tab1, correct = FALSE)
prop.test(tab1, correct = FALSE)

χ² = 391 (1 degree of freedom), p < 0.001. The proportion with a smoking history is 35.6% for women and 53.8% for men; women − men = −18.1 percentage points (95% CI −19.9 to −16.4). In a report, write it as a difference with a confidence interval: “men exceed women by 18 percentage points (95% CI 16 to 20)”.

9.3 Exercise 2: Did more people take antihypertensive drugs?

Among people who attended both the 1st and 3rd Framingham examinations, examine whether the proportion taking antihypertensive medication (BPMEDS) changed.

These are the same people at the 1st and 3rd examinations, so use McNemar’s test.

mcnemar.test(tab2)

Users increased from 59 of 2783 (2.1%) to 424 (15.2%). Among those who changed, 379 “started” and 14 “stopped”; McNemar’s χ² = 337 (with correction), p < 0.001. Entering the two time points into chisq.test() as separate groups is wrong because it counts the same people twice (Chapter 6).

9.4 Exercise 3: Smoking and death over 24 years

At the 1st Framingham examination, examine the relationship between smoking and death over 24 years.

9.4.1 3-1. First, naively

NoteThink about it

The difference in the proportion who died is 2.3 percentage points (odds ratio 1.11), p = 0.11, not significant. Can we conclude that “smoking is not associated with death”? What would you check next?

9.4.2 3-2. Stratify by age group

Check the smoking rate and proportion who died by age group (30s, 40s, 50s, 60s), and run a Mantel-Haenszel test.

mantelhaen.test(tab3s, correct = FALSE)

The smoking rate is about 59% in the 30s–40s and 32% in the 60s: in this data, younger people are more likely to smoke. By age group, the proportion who died is 5–12 percentage points higher among smokers in every stratum, and the MH odds ratio adjusted for age group is 1.81 (95% CI 1.57 to 2.09), p < 0.001. The unadjusted odds ratio of 1.11 had been cancelled out by “smokers being younger” (Chapter 7).

9.5 Exercise 4: Side effects in a small trial

In a fictitious small trial, 0 of the 12 people taking drug A and 5 of the 12 people taking drug B had side effects.

NoteThink about it

How many cells have an expected frequency below 5? How do the 3 p-values differ? Which one should you have decided to use in the analysis plan?

fisher.test(tab4)$p.value

The expected frequencies of the 2 cells in the “Yes” column are 2.5, below 5 (2 of 4 cells = 50%). The p-values are 0.012 for χ² without correction, 0.044 for Yates, and 0.037 for Fisher. All are below 0.05, but the values differ considerably. With few people and small expected frequencies, it is safest to decide on Fisher’s exact test in advance (Chapter 5). Avoid choosing the smallest p-value after seeing the results.

9.6 Review questions

NoteQuestion 1

You ran a χ² test on a 3-row × 4-column table. How many degrees of freedom are there? If the χ² value was 9, is the p-value larger or smaller than 0.05?

The degrees of freedom are (3 − 1)(4 − 1) = 6. The 5% point of the χ² distribution with 6 degrees of freedom is 12.6, so with χ² = 9, p > 0.05 (p = 0.17). Judge the χ² value relative to its degrees of freedom (Chapter 2).

NoteQuestion 2

A table in a paper said only “group A 30.0%, group B 20.0%”, so you tested it with chisq.test(matrix(c(30, 70, 20, 80), 2)). What is the problem?

Proportions are entered as numbers of people, so it is computed as a table with 100 people per group. Whether the actual numbers are 20 per group or 2000 per group, you get the same p-value. Always test a table of numbers of people (Chapter 1).

NoteQuestion 3

You want to check whether the frequencies of a genotype follow Hardy-Weinberg equilibrium (given the allele frequency p, AA : Aa : aa = p² : 2p(1−p) : (1−p)²). p was estimated from the data. How many degrees of freedom are there?

There are 3 categories; one is lost to the constraint on the total, and one more because p was estimated, so the degrees of freedom are 3 − 1 − 1 = 1. chisq.test(x, p = ...) computes with 2 degrees of freedom, so correct it yourself with pchisq() (Chapter 3).

NoteQuestion 4

In health check-up data on 35,000 people (17,500 men and 17,500 women), the proportions with hypertension were 20.0% and 21.0%, with p = 0.02. How would you report this?

The difference is 1 percentage point; even if statistically significant, whether it is a clinically meaningful difference must be considered separately. Report not the p-value but the difference and its 95% confidence interval (e.g., 1.0 percentage point, 95% CI 0.2 to 1.8), and interpret its size (Chapter 4).

NoteQuestion 5

You want to compare the positivity rates of two rapid tests performed on the same 50 patients. Which test do you use?

Two tests are performed on the same patients, so use McNemar’s test (or an exact test with binom.test() if there are few discordant pairs). Running a χ² test as though the two tests were done on separate groups of 50 people is wrong (Chapter 6).

NoteQuestion 6

You compared treatment success rates across 5 sites, and the overall χ² test gave p = 0.01. You then chose sites two at a time, ran 10 χ² tests, and reported the pairs with p < 0.05 as “different”. What is the problem?

With 10 tests, the probability that one of them is significant even with no true difference is much larger than 5% (about 40% if independent). Correct the p-values with Holm or similar, or decide the comparisons in advance and correct for that number (Chapter 8). If patient severity differs between sites, also consider adjusting for confounding (Chapter 7).

9.7 Closing

chisq.test() can be written in one line, but to use its result correctly you need:

  • The table to consist of numbers of people, with each person counted in one place
  • To understand what the expected frequencies and degrees of freedom mean
  • To choose a test that matches the question (goodness of fit, independence, McNemar, MH, trend)
  • To show not only the p-value but where and by how much

When testing your own data, check this checklist before writing chisq.test().