7  Before comparing groups — Table 1, confounding, and the p-value trap

In Chapter 1, we saw an example in which smokers appeared to “live longer”. In this chapter, we pose the same question in a much larger cohort and confirm the role of descriptive statistics before comparing groups (Table 1).

7.1 Question: is smoking associated with death?

Using the 4434 Framingham participants at baseline (base, already loaded), compare death over 24 years between smokers and non-smokers.

The mortality rates are 16.3 and 17.7 (per 1000 person-years), almost the same. The hazard ratio is 1.09 (95% confidence interval 0.99–1.20), and the difference is not significant.

NoteMake a prediction

Can we conclude that “in this cohort, smoking was not associated with death”? Recall Chapter 1 and think about what you should check first.

7.2 Table 1: line up the characteristics of the groups being compared

Line up the backgrounds of the two groups being compared in one table. This corresponds to Table 1 in epidemiological and clinical research papers.

Smokers were a population skewed towards lower risk of death:

  • About 4 years younger (48 vs 52 years)
  • More often male (54% vs 34%)
  • Lower blood pressure and BMI, and less diabetes

Age in particular is very strongly related to death.

7.3 Compare at the same age

In every age group, the mortality rate of smokers is 1.4–1.9 times that of non-smokers. In the overall comparison, this difference was cancelled out by smokers being younger.

Adjusting for age and sex, the hazard ratio becomes 1.40 (1.26–1.56).

In Chapter 1 (Whickham), confounding “reversed the harm”; here, it made the harm “disappear”. In both cases, you could have noticed before the analysis by looking at Table 1.

TipTable 1 is the “blueprint of the analysis”
  • Variables that differ between groups are candidate confounders (consider adjustment, stratification, or matching)
  • Variables that are almost the same raise little concern about confounding
  • If there are people who are almost absent from one of the groups (e.g., almost no smokers aged 65 or over), comparison is impossible for those people (no common support)

However, decide which variables to adjust for considering not only the differences in Table 1 but also the causal structure (e.g., whether the variable might be a consequence of the cause).

7.4 Does Table 1 need p-values?

Let us think about the p-values often attached to Table 1.

Almost everything is p < 0.001. Then is it right to “adjust for the variables with p < 0.05”?

7.4.1 p-values change with sample size, not with the size of the difference

Compare resting pulse rate between men and women among NHANES adults.

Comparing all of the roughly 10,000 people, the difference is about 3 beats/min and the p-value is around 10-44. What happens if we compare using fewer people from the same data?

The size of the difference (about 3 beats/min) is the same, yet with 50 people it is hardly ever significant, and with 1000 people it is almost always significant. In large datasets, even clinically meaningless small differences all become “significant”, while in small studies even large imbalances can be “not significant”. If you judge the presence of group imbalance by p-values, your judgement changes with the size of the data.

7.4.2 Standardized difference (SMD)

The size of group imbalance is generally described with the standardized difference (standardized mean difference, SMD), which does not depend on sample size.

\[ \text{SMD} = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{(s_1^2 + s_2^2)/2}} \]

  • The difference in means divided by the standard deviation averaged over the two groups (a similar formula exists for proportion variables)
  • As a rule of thumb, an absolute value of 0.1 or more is often regarded as a non-negligible imbalance

The SMD of the sex difference in pulse rate is about 0.27. Whether there are 50 people or 10,000, the value (expected value) of the SMD does not change.

Warningp-values in Table 1 of randomised controlled trials

In randomised trials, all differences between groups are due to chance. Attaching p-values to Table 1 and writing “no significant difference, so the groups are comparable” is meaningless, and is not recommended by the CONSORT statement either. In observational studies too, what you should look at in Table 1 is the size of the difference (SMD) and how strongly the variable is related to the outcome.

ImportantSummary of this chapter
  • Before comparing groups, line up the groups’ backgrounds in Table 1. Variables that differ are candidate confounders
  • In a crude comparison, confounding can reverse an association or make it disappear
  • Judge imbalance in Table 1 by the size of the difference, such as the SMD, not by p-values. p-values change with sample size