# Tests and model comparison {#sec-inference}
```{r}
#| echo: false
source("R/setup.R")
data(sleepstudy)
```
## Estimation methods: REML and ML
By default, `lmer()` estimates parameters using a method called **REML (restricted maximum likelihood)**.
| Method | Characteristics | When to use |
|---|---|---|
| REML (default) | Less bias in the estimates of the variance components (τ, σ) | When reporting final estimates, and when comparing **random-effects structures** |
| ML (maximum likelihood) | Variance components are estimated somewhat too small | When comparing models with different **fixed-effects structures** |
::: {.callout-note}
## Why use different methods?
Do you remember that dividing by $n$ biases the sample variance downward, while dividing by $n-1$ removes the bias?
ML corresponds to "dividing by $n$", and REML to "dividing by $n-1$".
However, because the REML likelihood is computed after "removing" the fixed-effects part,
**REML likelihoods cannot be compared between models with different fixed effects**. That is why ML is used to compare fixed effects.
:::
```{r}
fit_reml <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy) # REML
fit_ml <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy, REML = FALSE) # ML
vc_tab <- rbind(REML = as.data.frame(VarCorr(fit_reml))$sdcor,
ML = as.data.frame(VarCorr(fit_ml))$sdcor)
colnames(vc_tab) <- c("Intercept SD", "Slope SD", "Corr", "Residual SD")
vc_tab
```
## Testing fixed effects
### Method 1: t-test (Satterthwaite method in lmerTest)
These are the p-values shown by `summary()`. In mixed models, the degrees of freedom of the t distribution are not obvious, so they are obtained by approximation.
```{r}
summary(fit_reml)$coefficients
```
### Method 2: likelihood ratio test (LRT)
We compare the model "with `Days`" and the model "without `Days`". `anova()` automatically refits models fitted with REML using ML before comparing them.
```{r}
fit0 <- lmer(Reaction ~ 1 + (Days | Subject), data = sleepstudy)
fit1 <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)
anova(fit0, fit1) # "refitting model(s) with ML" is printed
```
How to read it:
- `AIC`, `BIC`: smaller means a better model
- `logLik`: log-likelihood (larger means a better fit)
- `Chisq`, `Df`, `Pr(>Chisq)`: the likelihood ratio test statistic, degrees of freedom (number of added parameters), and p-value
### Confidence intervals
```{r}
confint(fit_reml, oldNames = FALSE) # 95% confidence intervals by profile likelihood
```
- `sd_(Intercept)|Subject`: SD of the intercept ($\tau_0$)
- `sd_Days|Subject`: SD of the slope ($\tau_1$)
- `cor_Days.(Intercept)|Subject`: correlation between intercept and slope ($\rho$)
- `sigma`: residual SD ($\sigma$)
- `(Intercept)`, `Days`: fixed effects
Without `oldNames = FALSE`, you get number-only names like `.sig01`, which make it hard to tell which is which, so add it.
::: {.callout-tip}
With `confint(fit, method = "boot", nsim = 500)` you can also obtain confidence intervals by parametric bootstrap (this takes time).
:::
## Testing random effects
We use a likelihood ratio test to check "Is the random slope needed?". Since the fixed effects are the same, we can compare the models as fitted with REML (`refit = FALSE`).
```{r}
fit_ri <- lmer(Reaction ~ Days + (1 | Subject), data = sleepstudy)
fit_rs <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)
anova(fit_ri, fit_rs, refit = FALSE)
```
The p-value is small enough, and the AIC is also smaller for `fit_rs` → it is better to include the random slope.
::: {.callout-warning}
## Testing variance = 0 has a "boundary" problem
"τ₁ = 0 (no individual differences in the slope)" lies at the **edge** of the range a variance can take (0 or more).
In this case, p-values from the usual chi-squared distribution are **too large (conservative)**.
When testing the variance of a single random effect, dividing the p-value by 2 is a common correction.
Either way, "if it came out significant, it is actually even more significant", so the conclusion does not change.
:::
## Model selection with AIC
```{r}
fit_nocor <- lmer(Reaction ~ Days + (Days || Subject), data = sleepstudy) # no correlation
AIC(fit_ri, fit_nocor, fit_rs)
```
::: {.callout-note}
## Which model should you choose?
There are two commonly used approaches.
1. **Start from the maximal structure** (Barr et al., 2013): include the maximal random-effects structure the study design allows, and simplify if it does not converge
2. **Let the data decide**: keep only the structure that is needed, based on AIC or likelihood ratio tests (Matuschek et al., 2017)
As a beginner, "include the random effects that are theoretically plausible → simplify if estimation does not go well (Chapter @sec-diagnostics)" is good enough.
:::
::: {.callout-important}
## Chapter summary
- The default is REML. Compare models with different fixed effects using ML (`anova()` does this automatically)
- Fixed effects: p-values from `summary()` (Satterthwaite), likelihood ratio tests, `confint()`
- Random effects: `anova(..., refit = FALSE)`. Because of the boundary problem, the p-values are conservative
- Smaller AIC is better
:::