5 Whom does this sample represent? — Sampling and weights
The numbers from descriptive statistics are characteristics of “the people in this data”. They are not necessarily the same as the characteristics of “the population you want to know about”. In this chapter, you will experience with NHANES how large that gap can be.
5.1 A naive count: the proportion of US adults with diabetes
NHANES is “a survey to find out about the health of the US population”. Let us compute the proportion with diabetes and the racial breakdown among the 11778 people aged 20 or over.
Diabetes 14.2%, White 42.6%, Black 21.9%, Mexican 14.3% …
Around 2010, roughly what percentage of US adults do you think were White (non-Hispanic)? Is the 42.6% in this data close to that?
5.2 NHANES is collected “deliberately skewed”
NHANES does not just select people at random from across the US; it selects certain groups in larger numbers (oversampling). In 2009–2012, Hispanic people (including Mexican Americans), Black people, low-income people, and people aged 80 or over, among others (and from 2011, Asian people as well), were oversampled. This is so that estimates within each group are sufficiently precise even for groups with small numbers.
In exchange, each participant is given a weight (WTINT2YR, etc.) that indicates “how many people in the population this person represents”.
On average, one White person represents about 59,000 people, while one Black or Hispanic person represents about 20,000 people (per two-year survey cycle). So even though each is a single person, a White participant represents about 3 times as many people as a Black participant. If you count without weights, the voices of the oversampled groups become too loud.
5.3 Estimating with weights
Use the R package survey to specify the survey design (weights, strata, clusters).
| Unweighted (sample) | Weighted (estimate for US adults) | |
|---|---|---|
| Proportion White | 42.6% | 67.2% |
| Proportion aged 60+ | 32.8% | 25.3% |
| Mean age | 49.3 years | 47.2 years |
| Proportion with diabetes | 14.2% | 10.6% |
The unweighted diabetes proportion of 14.2% was the result of oversampling groups with more diabetes (Black people, Hispanic people, older people). The estimate for US adults is 10.6% (95% confidence interval 9.8–11.4%), and the unweighted value lies well outside this interval.
Looking at the proportion with diabetes by race shows why the numbers diverged.
5.3.1 Lining up unweighted and weighted
Let us make a table that lines up the proportions of categorical variables, unweighted and weighted.
With gtsummary’s tbl_svysummary(), you can make a weighted Table 1 in one line (tbl_svysummary(design_adults, include = c(Age, Race1, Diabetes))). Try it in R on your own PC (in the browser version of R, at the time of writing, it fails with an error due to the combination of package versions).
- The estimated population size is the sum of the weights (when combining 2 cycles, divide the weights by 2; otherwise the population doubles)
- Use
WTINT2YRfor interview-only items, andWTMEC2YRfor measurements at the examination centre (blood pressure, blood samples, etc.) - Restrict the target population (e.g., adults only) with
subset(design, ...). If youfilter()the data first, the standard errors will not be computed correctly
5.4 In cohorts without weights
Data that are “designed to represent the population and come with weights”, like NHANES, are actually in the minority. Many cohort studies, and data from health check-ups, claims, and electronic health records, have no weights. In that case, all you can do is describe who is included and think about which population the results apply to.
Participants in the Framingham study were:
- Residents of a single town (Framingham, Massachusetts)
- Adults who joined at ages 30–62 (the baseline in this data is ages 32–70)
- Almost all White
- People who consented to take part in the study (health-conscious people tend to gather)
For example, an association such as “a 10 mmHg higher blood pressure multiplies the risk of heart disease by X” often takes similar values in other populations, whereas a frequency such as “the incidence of heart disease is X%” cannot be applied as is to, say, Japanese adults.
In the large UK cohort UK Biobank (about 500,000 people), only about 5.5% of those invited took part, and it has been reported that participants smoke less, are less obese, and have lower mortality than the general UK population (Fry et al., Am J Epidemiol 2017). No matter how large the number of people, the bias from how people were selected does not shrink. If anything, because confidence intervals become narrower, there is a danger of confidently reporting biased estimates.
- Descriptive statistics are characteristics of “the people in the data”. They are not necessarily characteristics of the population
- In surveys with unequal selection probabilities (such as NHANES), prevalence and means are badly wrong without weights
- In cohorts without weights, describe who was selected and how they entered, and consider how far the results can be applied
- Frequencies (prevalence, incidence) are strongly affected by how people were selected. Associations (relative risks) are not immune either