6  Tests and model comparison

6.1 Estimation methods: REML and ML

By default, lmer() estimates parameters using a method called REML (restricted maximum likelihood).

Method Characteristics When to use
REML (default) Less bias in the estimates of the variance components (τ, σ) When reporting final estimates, and when comparing random-effects structures
ML (maximum likelihood) Variance components are estimated somewhat too small When comparing models with different fixed-effects structures
NoteWhy use different methods?

Do you remember that dividing by \(n\) biases the sample variance downward, while dividing by \(n-1\) removes the bias? ML corresponds to “dividing by \(n\)”, and REML to “dividing by \(n-1\)”. However, because the REML likelihood is computed after “removing” the fixed-effects part, REML likelihoods cannot be compared between models with different fixed effects. That is why ML is used to compare fixed effects.

fit_reml <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)               # REML
fit_ml   <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy, REML = FALSE) # ML
vc_tab <- rbind(REML = as.data.frame(VarCorr(fit_reml))$sdcor,
                ML   = as.data.frame(VarCorr(fit_ml))$sdcor)
colnames(vc_tab) <- c("Intercept SD", "Slope SD", "Corr", "Residual SD")
vc_tab
     Intercept SD Slope SD       Corr Residual SD
REML     24.74066 5.922138 0.06555124    25.59180
ML       23.77976 5.716799 0.08132109    25.59191

6.2 Testing fixed effects

6.2.1 Method 1: t-test (Satterthwaite method in lmerTest)

These are the p-values shown by summary(). In mixed models, the degrees of freedom of the t distribution are not obvious, so they are obtained by approximation.

summary(fit_reml)$coefficients
             Estimate Std. Error       df   t value     Pr(>|t|)
(Intercept) 251.40510   6.824597 16.99973 36.838090 1.171558e-17
Days         10.46729   1.545790 16.99998  6.771481 3.263824e-06

6.2.2 Method 2: likelihood ratio test (LRT)

We compare the model “with Days” and the model “without Days”. anova() automatically refits models fitted with REML using ML before comparing them.

fit0 <- lmer(Reaction ~ 1    + (Days | Subject), data = sleepstudy)
fit1 <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)
anova(fit0, fit1)   # "refitting model(s) with ML" is printed
Data: sleepstudy
Models:
fit0: Reaction ~ 1 + (Days | Subject)
fit1: Reaction ~ Days + (Days | Subject)
     npar    AIC    BIC  logLik -2*log(L)  Chisq Df Pr(>Chisq)    
fit0    5 1785.5 1801.4 -887.74    1775.5                         
fit1    6 1763.9 1783.1 -875.97    1751.9 23.537  1  1.226e-06 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

How to read it:

  • AIC, BIC: smaller means a better model
  • logLik: log-likelihood (larger means a better fit)
  • Chisq, Df, Pr(>Chisq): the likelihood ratio test statistic, degrees of freedom (number of added parameters), and p-value

6.2.3 Confidence intervals

confint(fit_reml, oldNames = FALSE)   # 95% confidence intervals by profile likelihood
                                   2.5 %      97.5 %
sd_(Intercept)|Subject        14.3813860  37.7165596
sd_Days|Subject                3.8011632   8.7533849
cor_Days.(Intercept)|Subject  -0.4815008   0.6849863
sigma                         22.8982669  28.8579965
(Intercept)                  237.6806955 265.1295146
Days                           7.3586533  13.5759187
  • sd_(Intercept)|Subject: SD of the intercept (\(\tau_0\))
  • sd_Days|Subject: SD of the slope (\(\tau_1\))
  • cor_Days.(Intercept)|Subject: correlation between intercept and slope (\(\rho\))
  • sigma: residual SD (\(\sigma\))
  • (Intercept), Days: fixed effects

Without oldNames = FALSE, you get number-only names like .sig01, which make it hard to tell which is which, so add it.

Tip

With confint(fit, method = "boot", nsim = 500) you can also obtain confidence intervals by parametric bootstrap (this takes time).

6.3 Testing random effects

We use a likelihood ratio test to check “Is the random slope needed?”. Since the fixed effects are the same, we can compare the models as fitted with REML (refit = FALSE).

fit_ri <- lmer(Reaction ~ Days + (1 | Subject),    data = sleepstudy)
fit_rs <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)
anova(fit_ri, fit_rs, refit = FALSE)
Data: sleepstudy
Models:
fit_ri: Reaction ~ Days + (1 | Subject)
fit_rs: Reaction ~ Days + (Days | Subject)
       npar    AIC    BIC  logLik -2*log(L)  Chisq Df Pr(>Chisq)    
fit_ri    4 1794.5 1807.2 -893.23    1786.5                         
fit_rs    6 1755.6 1774.8 -871.81    1743.6 42.837  2   4.99e-10 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The p-value is small enough, and the AIC is also smaller for fit_rs → it is better to include the random slope.

WarningTesting variance = 0 has a “boundary” problem

“τ₁ = 0 (no individual differences in the slope)” lies at the edge of the range a variance can take (0 or more). In this case, p-values from the usual chi-squared distribution are too large (conservative). When testing the variance of a single random effect, dividing the p-value by 2 is a common correction. Either way, “if it came out significant, it is actually even more significant”, so the conclusion does not change.

6.4 Model selection with AIC

fit_nocor <- lmer(Reaction ~ Days + (Days || Subject), data = sleepstudy)  # no correlation

AIC(fit_ri, fit_nocor, fit_rs)
          df      AIC
fit_ri     4 1794.465
fit_nocor  5 1753.669
fit_rs     6 1755.628
NoteWhich model should you choose?

There are two commonly used approaches.

  1. Start from the maximal structure (Barr et al., 2013): include the maximal random-effects structure the study design allows, and simplify if it does not converge
  2. Let the data decide: keep only the structure that is needed, based on AIC or likelihood ratio tests (Matuschek et al., 2017)

As a beginner, “include the random effects that are theoretically plausible → simplify if estimation does not go well (Chapter Chapter 7)” is good enough.

ImportantChapter summary
  • The default is REML. Compare models with different fixed effects using ML (anova() does this automatically)
  • Fixed effects: p-values from summary() (Satterthwaite), likelihood ratio tests, confint()
  • Random effects: anova(..., refit = FALSE). Because of the boundary problem, the p-values are conservative
  • Smaller AIC is better