---
webr:
packages: [dplyr, ggplot2, ggbeeswarm, patchwork]
resources: [R/setup.R, py/setup.py, data/penguins.csv, data/nhanes_adults.csv, fonts/NotoSansJP-Regular.ttf, fonts/NotoSansJP-Bold.ttf]
cell-options:
autorun: true
pyodide:
packages: [pandas, matplotlib, seaborn]
cell-options:
autorun: true
---
{{< include ./_extensions/r-wasm/live/_knitr.qmd >}}
# Distributions: histograms, box plots, points, error bars {#sec-distribution}
```{webr}
#| include: false
source("R/setup.R")
library(patchwork)
library(ggbeeswarm)
peng <- penguins |> filter(!is.na(sex)) |>
mutate(Species = factor(species, c("Adelie", "Chinstrap", "Gentoo"), c("Adelie", "Chinstrap", "Gentoo")))
```
```{pyodide}
#| include: false
exec(open("py/setup.py").read())
peng = penguins.dropna(subset=["sex"]).copy()
peng["Species"] = pd.Categorical(peng["species"].map({"Adelie": "Adelie", "Chinstrap": "Chinstrap", "Gentoo": "Gentoo"}),
categories=["Adelie", "Chinstrap", "Gentoo"])
```
Figures comparing the distribution of values across groups are the most common type in clinical research. Depending on how you draw them, **what is shown and what is hidden** changes a lot.
## Histograms: the shape changes with the bin width
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
h <- function(w) ggplot(peng, aes(flipper_length_mm)) +
geom_histogram(binwidth = w, fill = "grey60", colour = "white") +
labs(x = "Flipper length (mm)", y = "Count", title = paste0("Bin width ", w, " mm"))
h(2) + h(5) + h(15)
```
## Python (seaborn / matplotlib)
```{pyodide}
fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
for ax, w in zip(axes, [2, 5, 15]):
sns.histplot(data=peng, x="flipper_length_mm", binwidth=w, color="grey", ax=ax)
ax.set(xlabel="Flipper length (mm)", ylabel="Count", title=f"Bin width {w} mm")
plt.tight_layout()
plt.show()
```
:::
With a bin width of 5 mm you can see **two peaks** (Adelie/Chinstrap, and Gentoo), but at 15 mm it looks like a single peak.
Check histograms by trying several bin widths.
## What does a box plot show?
The parts of a box plot are the following values.
- **The line inside the box**: the median
- **Top and bottom of the box**: the first quartile (25%) and the third quartile (75%). The height of the box is the interquartile range (IQR)
- **Whiskers**: from the edge of the box to **the most extreme data point within 1.5 × IQR**
- **Points**: data beyond the whiskers (candidate outliers)
Let's compute them by hand and compare with the figure.
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
x <- peng$body_mass_g[peng$Species == "Gentoo"]
q <- quantile(x, c(0.25, 0.5, 0.75)); iqr <- q[3] - q[1]
c(q, lower_whisker = min(x[x >= q[1] - 1.5 * iqr]), upper_whisker = max(x[x <= q[3] + 1.5 * iqr]))
ggplot(peng, aes(Species, body_mass_g)) + geom_boxplot() + labs(x = NULL, y = "Body mass (g)")
```
## Python (seaborn / matplotlib)
```{pyodide}
x = peng.loc[peng["Species"] == "Gentoo", "body_mass_g"]
q1, q2, q3 = x.quantile([0.25, 0.5, 0.75]); iqr = q3 - q1
print(q1, q2, q3, "whiskers:", x[x >= q1 - 1.5 * iqr].min(), x[x <= q3 + 1.5 * iqr].max())
fig, ax = plt.subplots()
sns.boxplot(data=peng, x="Species", y="body_mass_g", ax=ax)
ax.set(xlabel="", ylabel="Body mass (g)")
plt.show()
```
:::
::: {.callout-note}
## How quartiles are computed
There are many ways to compute quartiles. R's `quantile()` default (type 7) and the default of pandas / NumPy (linear interpolation) are the same method, so the box plots from the two agree. SPSS and some Excel functions may use other methods, and with small samples the values differ.
:::
## What box plots hide
Let's create **a distribution with two peaks** and **a distribution with one peak** that have similar quartiles.
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
set.seed(1)
demo <- data.frame(
group = rep(c("One peak", "Two peaks"), each = 200),
value = c(rnorm(200, 50, 15), c(rnorm(100, 35, 4), rnorm(100, 65, 4))))
p_box <- ggplot(demo, aes(group, value)) + geom_boxplot() + labs(title = "Box plot", x = NULL)
p_vio <- ggplot(demo, aes(group, value)) + geom_violin(fill = "grey85") + labs(title = "Violin", x = NULL)
p_pts <- ggplot(demo, aes(group, value)) + geom_quasirandom(size = 0.8, alpha = 0.6) + labs(title = "Points (beeswarm)", x = NULL)
p_box + p_vio + p_pts
```
## Python (seaborn / matplotlib)
```{pyodide}
rng = np.random.default_rng(1)
demo = pd.DataFrame({
"group": ["One peak"] * 200 + ["Two peaks"] * 200,
"value": np.r_[rng.normal(50, 15, 200), rng.normal(35, 4, 100), rng.normal(65, 4, 100)]})
fig, axes = plt.subplots(1, 3, figsize=(12, 3.8), sharey=True)
sns.boxplot(data=demo, x="group", y="value", ax=axes[0]); axes[0].set_title("Box plot")
sns.violinplot(data=demo, x="group", y="value", color="0.85", ax=axes[1]); axes[1].set_title("Violin")
sns.swarmplot(data=demo, x="group", y="value", size=2.5, ax=axes[2]); axes[2].set_title("Points (beeswarm)")
for a in axes: a.set_xlabel("")
plt.tight_layout()
plt.show()
```
:::
In the box plots the two groups look very similar, but the violin plots and points show that the right-hand group has **almost no values in the middle**.
With up to a few dozen observations, **showing every point** is the most honest option.
## Overlaying points on a box plot
A commonly used form is a box plot with points overlaid. If you forget to **remove the box plot's outlier points**, outliers are drawn twice.
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
ggplot(peng, aes(Species, body_mass_g, colour = Species)) +
geom_boxplot(outlier.shape = NA, width = 0.5) + # don't draw outlier points
geom_point(alpha = 0.5, size = 1.2,
position = position_jitter(width = 0.15, height = 0, seed = 1)) + # shift sideways; seed gives the same shift every time
theme(legend.position = "none") +
labs(x = NULL, y = "Body mass (g)")
```
## Python (seaborn / matplotlib)
```{pyodide}
fig, ax = plt.subplots()
sns.boxplot(data=peng, x="Species", y="body_mass_g", hue="Species", showfliers=False, width=0.5,
fill=False, legend=False, ax=ax) # don't draw outliers, don't fill boxes
sns.stripplot(data=peng, x="Species", y="body_mass_g", hue="Species", jitter=0.15, alpha=0.5, size=4,
legend=False, ax=ax)
ax.set(xlabel="", ylabel="Body mass (g)")
plt.show()
```
:::
::: {.callout-tip}
## Jitter is random
The amount by which points are shifted sideways is random, so the figure changes slightly every time you run it. For figures in papers, make them reproducible with `position_jitter(seed = )` in R, or by setting `np.random.seed()` before running in seaborn.
:::
### Splitting by two categories
If you also split by sex, you need to **align the horizontal positions** of boxes and points (`dodge`).
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
ggplot(peng, aes(Species, body_mass_g, fill = sex)) +
geom_boxplot(outlier.shape = NA, width = 0.6, position = position_dodge(width = 0.75), alpha = 0.5) +
geom_point(aes(colour = sex), size = 1, alpha = 0.6,
position = position_jitterdodge(jitter.width = 0.15, dodge.width = 0.75, seed = 1)) +
labs(x = NULL, y = "Body mass (g)", fill = "Sex", colour = "Sex")
```
## Python (seaborn / matplotlib)
```{pyodide}
fig, ax = plt.subplots()
sns.boxplot(data=peng, x="Species", y="body_mass_g", hue="sex", showfliers=False, width=0.6,
boxprops={"alpha": 0.5}, ax=ax)
sns.stripplot(data=peng, x="Species", y="body_mass_g", hue="sex", dodge=True, jitter=0.12,
size=3, alpha=0.6, legend=False, ax=ax)
ax.set(xlabel="", ylabel="Body mass (g)")
ax.legend(title="Sex")
plt.show()
```
:::
## Means and error bars: SD, SE, 95% confidence interval
Figures showing the mean as a point with **error bars** are also common. What the error bars represent changes the meaning completely.
| Error bar | What it represents | As n increases |
|---|---|---|
| **SD** (standard deviation) | Spread of the individual values | Does not change |
| **SE** (standard error = SD / √n) | Uncertainty in the estimate of the mean | Gets smaller |
| **95% confidence interval** (≈ mean ± 1.96 SE) | Range where the mean is likely to be | Gets smaller |
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
summ <- peng |> group_by(Species) |>
summarise(n = n(), Mean = mean(body_mass_g), SD = sd(body_mass_g), SE = SD / sqrt(n),
CI = qt(0.975, n - 1) * SE)
summ
eb <- function(w, lab) ggplot(summ, aes(Species, Mean)) +
geom_point(size = 2.5) +
geom_errorbar(aes(ymin = Mean - .data[[w]], ymax = Mean + .data[[w]]), width = 0.15) +
coord_cartesian(ylim = c(3000, 6000)) + labs(x = NULL, y = "Body mass (g)", title = lab)
eb("SD", "Mean ± SD") + eb("SE", "Mean ± SE") + eb("CI", "Mean and 95% CI")
```
## Python (seaborn / matplotlib)
```{pyodide}
fig, axes = plt.subplots(1, 3, figsize=(12, 3.8), sharey=True)
for ax, (eb, lab) in zip(axes, [("sd", "Mean ± SD"), ("se", "Mean ± SE"), (("ci", 95), "Mean and 95% CI")]):
sns.pointplot(data=peng, x="Species", y="body_mass_g", errorbar=eb, linestyle="none",
capsize=0.15, seed=1, ax=ax)
ax.set(xlabel="", ylabel="Body mass (g)", title=lab, ylim=(3000, 6000))
plt.tight_layout()
plt.show()
```
:::
With the same data, SE and CI error bars are much shorter than SD, and **the group differences look larger**.
**Always state what the error bars are** in the figure's description (legend, caption).
::: {.callout-warning}
## seaborn's default error bar is a "bootstrap 95% CI"
If you specify nothing, seaborn's `pointplot()`, `barplot()` and `lineplot()` draw **a 95% confidence interval computed by bootstrapping (random numbers)**. It changes slightly every run and also differs slightly from a t-distribution CI.
Write `errorbar=("ci", 95)` explicitly and specify `seed=`, or use `errorbar="se"` or `errorbar="sd"`, and state what you drew.
:::
::: {.callout-warning}
## Avoid bar chart + error bar (dynamite plots)
Showing the mean as a bar height with an error bar on top **hides the shape of the distribution, the sample size and the outliers**, so many journals recommend avoiding it. Use points, or box plots + points.
:::
::: {.callout-important}
## Chapter summary
- For histograms, try several **bin widths**
- Box plots show **median, quartiles, 1.5 × IQR whiskers and outliers**, and they **hide two-peaked distributions**
- When overlaying points, **remove the box's outliers** and fix the **random seed** for the jitter
- **Always state whether error bars are SD, SE or CI**. seaborn's default is a bootstrap CI
:::