3 Goodness-of-fit tests: comparing with theoretical proportions
The χ² tests in Chapters 1 and 2 examined the relationship (independence) between two variables. The form χ² value = Σ (O − E)²/E can be used for other questions too, simply by changing where the “expected frequencies” come from. In this chapter, we cover the goodness-of-fit test, which compares the frequencies of a single variable with proportions determined by theory.
3.1 Mendel’s peas
Mendel crossed hybrid peas with round, yellow seeds, and counted the 556 seeds of the next generation, sorting them into 4 types. If the laws of inheritance are correct, the 4 types should appear in the ratio 9 : 3 : 3 : 1.
The expected frequency is “total × theoretical proportion”. The χ² value is computed in the same way as in Chapter 1.
Passing a single vector and p = (the theoretical proportions) to chisq.test() gives a goodness-of-fit test. The χ² value is 0.47 and p = 0.93; the observations fit the theoretical proportions well.
3.1.1 The degrees of freedom are “number of categories − 1”
The 4 frequencies are constrained to total 556, so once 3 are decided, the last one is determined. The degrees of freedom are 4 − 1 = 3.
The statistician R. A. Fisher pooled Mendel’s experimental results, tested them again, and pointed out that they were far too close to the theory (the χ² value was too small; the probability of such a good fit by chance is very low). Goodness-of-fit tests are used to check not only whether deviations are too large, but also whether they are too small (a possible sign that the data were tidied up). There are various views on what really happened.
3.2 You cannot show that something “fits”
p = 0.93 means “there is no evidence of a deviation from 9:3:3:1”, and not “9:3:3:1 has been proven”. Let us keep the same proportions and multiply the counts by 100.
Even though the proportions are exactly the same, the p-value has changed from 0.93 to 0.0000000003. The χ² value grows in proportion to the number of observations, so with a large sample, even meaninglessly small deviations become significant. Conversely, with a small sample, even large deviations are missed. Concluding “no significant difference = consistent with theory” from a goodness-of-fit test is dangerous unless you take the sample size into account (Chapters 4 and 8).
3.3 Real data: the last digit of blood pressure
We count the last digit (units digit) of systolic blood pressure recorded at the first Framingham examination (4434 people). Since the blood pressure values themselves vary widely, the last digits 0–9 should each appear about equally often (10% each).
The χ² value is 160 (9 degrees of freedom), far from a uniform distribution. The standardized residuals show that 0 (+9.8) and 5 (+5.6) are too frequent. This is digit preference — rounding to convenient numbers when reading a mercury sphygmomanometer.
Comparing last-digit bias by measurer, site, or period can provide clues to differences in measurement procedures (e.g., the bias disappears after switching to automated blood pressure monitors).
3.4 When the theoretical proportions are estimated from the data
When checking whether data fit a theoretical distribution such as the Poisson distribution, the parameters of the distribution (such as the mean λ) are often estimated from the data. In that case, you need to reduce the degrees of freedom by the number of estimated parameters.
As a famous example, we use data (Bortkiewicz) counting the number of soldiers killed by horse kicks in the 19th-century Prussian army, for 10 army corps × 20 years = 200 “corps-years”.
3 and 4 deaths are rare, with expected frequencies of only about 4.0 and 0.6 respectively, so we combine 3 or more into one category. Even combined, the expected frequency is 4.8, slightly below 5 (in Chapter 5 we confirm that this is not much of a problem; it is also why chisq.test() issues a warning).
- There are 4 categories → one is lost to the constraint on the total
- λ was estimated from the data → one more is lost
- The degrees of freedom are 4 − 1 − 1 = 2
chisq.test(x, p = ...) does not know whether p was estimated from the data, so it computes with the degrees of freedom left at 3. When there are estimated parameters, use only the χ² value, correct the degrees of freedom yourself, and compute the p-value with pchisq().
3.4.1 What happens if you don’t correct the degrees of freedom
Let us generate data that truly follow a Poisson distribution 2000 times, estimate λ each time, and test.
The proportion that is “significant” even though the data truly follow a Poisson distribution (the type I error) should be 5%. With the degrees of freedom left at 3, it is much smaller, about 2% (the test becomes conservative and more likely to miss real deviations). Correcting to 2 degrees of freedom brings it to about 4.5%, close to 5%. This is because estimating the parameter “pulls” the expected frequencies towards the data, making the χ² value smaller.
- The goodness-of-fit test compares the frequencies of one variable with theoretical proportions.
chisq.test(x, p = proportions) - The degrees of freedom are number of categories − 1 − (number of parameters estimated from the data). When parameters are estimated, correct them yourself with
pchisq() - “No significant difference” does not prove “consistent with theory”. With a large sample, even small deviations become significant
- It can also be used to check data quality, e.g. digit preference