Linear Mixed Models from Scratch

Learning by doing with R + lme4

Today’s goals

  1. Explain the situations where ordinary regression fails
  2. Explain fixed effects and random effects in your own words
  3. Fit models with lmer() and read the output
  4. Compare and diagnose models and report the results

Today’s outline

  1. Why linear mixed models are needed
  2. Review of simple regression
  3. Three kinds of pooling
  4. Random-intercept models
  5. Random-slope models
  6. Tests and model comparison
  7. Diagnostics and reporting

Tip

Detailed explanations and exercises are in the hands-on book

1. Why linear mixed models are needed

Today’s data (fictional)

Study time and test score recorded for students in 20 schools

d <- sim_school()
head(d, 4)
  school hours score
1    S01   4.6  47.4
2    S01   4.0  53.1
3    S01   0.8  37.6
4    S01   6.3  69.6

What we want to know: if study time increases by one hour, how many points does the score go up?

Look at everything in a scatter plot

Upward trend, but large scatter

Split by school

Every school slopes upward, but the level differs by school

What is the problem? Independence breaks

  • Ordinary regression assumes the data are independent of each other
  • But students in the same school are similar → not independent
  • It looks like “information from 315 students”, but in part it is closer to “information from 20 schools”

Grouped structures are everywhere

Schools ⊃ students, hospitals ⊃ patients, stores ⊃ sales, repeated measures on the same person …

Experience it: what if we ignore it?

Randomly assign a “tutoring program” that actually has no effect at the school level, and test 300 times

Summary so far

Key points

  • With a grouped structure, data within a group are similar (not independent)
  • Ignoring this and using ordinary regression makes it easy to wrongly conclude “significant”
  • Linear mixed models build the “offset of each group” into the model

2. Review of simple regression

The simple regression model

\[ \text{score}_i = \beta_0 + \beta_1 \cdot \text{hours}_i + \varepsilon_i, \qquad \varepsilon_i \sim N(0, \sigma^2) \]

  • \(\beta_0\): intercept (score when study time is 0)
  • \(\beta_1\): slope (increase in score per hour)
  • \(\varepsilon_i\): error (individual variation, independent of each other)
fit_lm <- lm(score ~ hours, data = d)
coef(fit_lm)
(Intercept)       hours 
  43.493521    3.408943 

Lining up the residuals by school…

Overall positive / negative depending on the school → the “school offset” is mixed into the error

3. Three kinds of pooling

How do we handle differences between schools?

Approach Differences between schools Problem
Complete pooling Ignored Independence breaks
No pooling Completely separate for each school Unstable with little data
Partial pooling Different but similar ← linear mixed model

The assumption of partial pooling:

The intercepts differ from school to school, but they come from a single normal distribution

Shrinkage

The less data a school has, the more it is pulled toward the overall mean

Fixed effects and random effects

Fixed effects

Estimate a single value common to everyone

  • Study-time slope \(\beta_1\)
  • Overall intercept \(\beta_0\)
  • Sex, drug A/B, etc.

Random effects

Differ by group and vary according to a distribution

  • School offsets \(u_j\)
  • Participants, stores, hospitals, etc.
  • Rule of thumb: 5–6 or more groups

“Mixed” = a model that mixes these two kinds of effects

4. Random-intercept models

The model equation

\[ \text{score}_{ij} = \underbrace{\beta_0 + \beta_1 \cdot \text{hours}_{ij}}_{\text{fixed effects}} + \underbrace{u_j}_{\text{school offset}} + \underbrace{\varepsilon_{ij}}_{\text{individual variation}} \]

\[ u_j \sim N(0, \tau^2), \qquad \varepsilon_{ij} \sim N(0, \sigma^2) \]

  • The only difference from simple regression: one term, \(u_j\), is added
  • What we estimate is not 20 values of \(u_j\) but just one: the variation \(\tau\)
  • \(\tau\): variation between schools; \(\sigma\): variation within schools

How to write it in R

lmer(score ~ hours + (1 | school), data = d)
Part Meaning
score ~ hours Fixed effects (same as lm())
(1 | school) The intercept (1) varies by school

Left of | = what varies; right = what it varies by

Reading the output

fit_ri <- lmer(score ~ hours + (1 | school), data = d)
summary(fit_ri)
Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: score ~ hours + (1 | school)
   Data: d

REML criterion at convergence: 2119.3

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.52549 -0.63282 -0.00379  0.65028  2.52623 

Random effects:
 Groups   Name        Variance Std.Dev.
 school   (Intercept) 62.97    7.935   
 Residual             40.54    6.367   
Number of obs: 315, groups:  school, 20

Fixed effects:
            Estimate Std. Error       df t value Pr(>|t|)    
(Intercept)  46.0878     1.9390  24.7389   23.77   <2e-16 ***
hours         3.2296     0.1299 297.5316   24.86   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Correlation of Fixed Effects:
      (Intr)
hours -0.338

Key parts of the output

VarCorr(fit_ri)    # τ (school) and σ (Residual)
 Groups   Name        Std.Dev.
 school   (Intercept) 7.9352  
 Residual             6.3668  
fixef(fit_ri)      # β0 and β1
(Intercept)       hours 
  46.087796    3.229567 
  • True values: \(\tau = 8\), \(\sigma = 6\), \(\beta_1 = 3\) → recovered quite well
  • ranef(): school offsets \(\hat u_j\); coef(): the line for each school

In a plot: parallel lines

Intraclass correlation coefficient (ICC)

The proportion of the total variation accounted for by differences between schools

\[ \text{ICC} = \frac{\tau^2}{\tau^2 + \sigma^2} \]

vc <- as.data.frame(VarCorr(fit_ri))
vc$vcov[1] / sum(vc$vcov)
[1] 0.60836
  • ICC ≈ 0 → almost the same as ordinary regression
  • Large ICC → the mixed model matters
  • Can also be read as “the correlation between the scores of two students in the same school”

5. Random-slope models

sleepstudy: a sleep deprivation experiment

18 people restricted to 3 hours of sleep per night; reaction time (ms) measured for 10 days

Not only the intercept, but also the pace of deterioration (slope) differs by person

The model equation

\[ \text{Reaction}_{ij} = (\beta_0 + u_{0j}) + (\beta_1 + u_{1j}) \cdot \text{Days}_{ij} + \varepsilon_{ij} \]

Three variation parameters are estimated:

  • \(\tau_0\): individual differences in the intercept
  • \(\tau_1\): individual differences in the slope
  • \(\rho\): correlation between intercept and slope

How to write it in R

Syntax Meaning
(1 | g) Random intercept only
(x | g) Intercept + slope, correlation also estimated
(x || g) Intercept + slope, correlation assumed to be 0
(0 + x | g) Random slope only
fit_rs <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)

Results

fixef(fit_rs)
(Intercept)        Days 
  251.40510    10.46729 
VarCorr(fit_rs)
 Groups   Name        Std.Dev. Corr  
 Subject  (Intercept) 24.7407        
          Days         5.9221  0.066 
 Residual             25.5918        
  • On average, 10.5 ms slower per day
  • SD of individual differences in slope ≈ 5.9 → about 95% of people are in -1.4 to 22.3 ms/day

Random intercept vs random slope

Further: nested and crossed

  • Nested: students ⊂ classes ⊂ schools
    • (1 | school/class)
  • Crossed: all participants × all stimulus words
    • (1 | subject) + (1 | item)

6. Tests and model comparison

REML and ML

Method When to use
REML (default) Final estimates, comparing random effects
ML Comparing models with different fixed effects
  • The difference is similar to dividing the variance by \(n-1\) (REML) or by \(n\) (ML)
  • When comparing fixed effects, anova() automatically refits with ML

Testing fixed effects

summary(fit_rs)$coefficients        # Satterthwaite method from lmerTest
             Estimate Std. Error       df   t value     Pr(>|t|)
(Intercept) 251.40510   6.824597 16.99973 36.838090 1.171558e-17
Days         10.46729   1.545790 16.99998  6.771481 3.263824e-06
fit0 <- lmer(Reaction ~ 1 + (Days | Subject), data = sleepstudy)
anova(fit0, fit_rs)                 # likelihood ratio test (automatically ML)
Data: sleepstudy
Models:
fit0: Reaction ~ 1 + (Days | Subject)
fit_rs: Reaction ~ Days + (Days | Subject)
       npar    AIC    BIC  logLik -2*log(L)  Chisq Df Pr(>Chisq)    
fit0      5 1785.5 1801.4 -887.74    1775.5                         
fit_rs    6 1763.9 1783.1 -875.97    1751.9 23.537  1  1.226e-06 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Testing random effects

anova(fit_ri2, fit_rs, refit = FALSE)   # same fixed effects → keep REML
Data: sleepstudy
Models:
fit_ri2: Reaction ~ Days + (1 | Subject)
fit_rs: Reaction ~ Days + (Days | Subject)
        npar    AIC    BIC  logLik -2*log(L)  Chisq Df Pr(>Chisq)    
fit_ri2    4 1794.5 1807.2 -893.23    1786.5                         
fit_rs     6 1755.6 1774.8 -871.81    1743.6 42.837  2   4.99e-10 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Warning

“Variance = 0” is at the edge of the possible values → p-values are conservative (too large)

Comparing with AIC

fit_nocor <- lmer(Reaction ~ Days + (Days || Subject), data = sleepstudy)
AIC(fit_ri2, fit_nocor, fit_rs)
          df      AIC
fit_ri2    4 1794.465
fit_nocor  5 1753.669
fit_rs     6 1755.628
  • Smaller is better
  • Strategy: include the theoretically plausible random effects → simplify if it does not work

7. Diagnostics and reporting

Checking the residuals

  • Left: OK if scattered evenly around 0 (watch out for funnel shapes and curves)
  • Right: normality OK if the points lie on the line

Common warnings

boundary (singular) fit

A variance was estimated as 0 / a correlation as ±1

  • Drop that random effect
  • Simplify (x | g) → (x || g) → (1 | g)

Model failed to converge

The optimization does not converge

  • Center / standardize the predictors
  • lmerControl(optimizer = "bobyqa")
  • Compare several optimizers with allFit()

What to include when reporting

  1. Model structure: response, fixed effects, random effects, groups
  2. Estimation method: REML/ML, versions of R and lme4, how p-values were computed
  3. Fixed effects: estimates, SE or 95% CI, test results
  4. Random effects: each SD, correlations

Reaction time increased on average by 10.5 ms per day of sleep restriction (95% CI [7.4, 13.5], t(17.0) = 6.77, p < .001). The SD of individual differences in the slope was 5.9 ms/day.

Summary

  • Grouped structure → not independent → linear mixed model
  • Fixed effects = common to all; random effects = vary by group
  • lmer(y ~ x + (1 | g)): random intercept
  • lmer(y ~ x + (x | g)): random slope
  • Partial pooling → shrinkage
  • Compare with anova() / AIC; diagnose with residual plots

Next steps

  • Exercises in the hands-on book (6 exercises with the Orthodont data)
  • GLMMs: 0/1 or count responses → glmer(..., family = binomial)
  • Bayesian: the brms package
  • Further reading
    • Takuya Kubo, Introduction to Statistical Modeling for Data Analysis (in Japanese)
    • Gelman & Hill “Data Analysis Using Regression and Multilevel/Hierarchical Models”