---
webr:
packages: [dplyr, ggplot2, ggsignif]
resources: [R/setup.R, py/setup.py, data/penguins.csv, data/nhanes_adults.csv, fonts/NotoSansJP-Regular.ttf, fonts/NotoSansJP-Bold.ttf]
cell-options:
autorun: true
pyodide:
packages: [pandas, matplotlib, seaborn, scipy, statsmodels]
cell-options:
autorun: true
---
{{< include ./_extensions/r-wasm/live/_knitr.qmd >}}
# Significance asterisks and brackets {#sec-significance}
```{webr}
#| include: false
source("R/setup.R")
peng <- penguins |> filter(!is.na(sex)) |>
mutate(Species = factor(species, c("Adelie", "Chinstrap", "Gentoo"), c("Adelie", "Chinstrap", "Gentoo")),
Sex = factor(sex, c("female", "male"), c("Female", "Male")))
```
```{pyodide}
#| include: false
exec(open("py/setup.py").read())
from scipy import stats
from statsmodels.stats.multitest import multipletests
peng = penguins.dropna(subset=["sex"]).copy()
peng["Species"] = pd.Categorical(peng["species"].map({"Adelie": "Adelie", "Chinstrap": "Chinstrap", "Gentoo": "Gentoo"}),
categories=["Adelie", "Chinstrap", "Gentoo"])
peng["Sex"] = pd.Categorical(peng["sex"].map({"female": "Female", "male": "Male"}), categories=["Female", "Male"])
```
Figures comparing groups **with test results drawn in as brackets and asterisks (\*)** are very common in life-science and medical papers.
In this chapter, you will practise **running the tests yourself and drawing the results yourself**. There are convenient packages, but being able to control what you draw is what matters.
## Procedure
1. Decide **what to compare with what** (all pairs, or only against a control group)
2. Run the tests and apply a **multiple-comparison correction**
3. Convert p-values to symbols such as asterisks
4. Draw the brackets and symbols **at heights that don't overlap the data**
## 1–2. Tests and multiple-comparison correction
We compare the body mass of the 3 species two at a time (3 comparisons). So that the brackets can later be stacked without overlapping, we order **comparisons of neighbours (short brackets) first and comparisons of distant groups (long brackets) last**. Here we run t-tests that do not assume equal variances (Welch) and correct with **Holm's method**.
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
pairs <- list(c("Adelie", "Chinstrap"), c("Chinstrap", "Gentoo"), c("Adelie", "Gentoo")) # short brackets first (bottom), long ones last (top)
res <- do.call(rbind, lapply(pairs, function(pr) {
x <- peng$body_mass_g[peng$Species == pr[1]]; y <- peng$body_mass_g[peng$Species == pr[2]]
data.frame(group1 = pr[1], group2 = pr[2], diff = mean(y) - mean(x), p = t.test(x, y)$p.value)
}))
res$p_holm <- p.adjust(res$p, method = "holm")
res
```
## Python (seaborn / matplotlib)
```{pyodide}
pairs = [("Adelie", "Chinstrap"), ("Chinstrap", "Gentoo"), ("Adelie", "Gentoo")] # short brackets first (bottom), long ones last (top)
rows = []
for g1, g2 in pairs:
x = peng.loc[peng["Species"] == g1, "body_mass_g"]; y = peng.loc[peng["Species"] == g2, "body_mass_g"]
rows.append({"group1": g1, "group2": g2, "diff": y.mean() - x.mean(),
"p": stats.ttest_ind(x, y, equal_var=False).pvalue})
res = pd.DataFrame(rows)
res["p_holm"] = multipletests(res["p"], method="holm")[1]
res
```
:::
The difference between Adelie and Chinstrap is 27 g with p = 0.65 (still 0.65 after correction); the differences between Gentoo and the other 2 species are about 1360–1390 g, with very small p-values.
## 3. Turning p-values into symbols
The following mapping is common, but it **differs between journals and fields**, so always state it in the figure's description.
| p-value | Symbol |
|---|---|
| p ≥ 0.05 | ns (not significant) |
| p < 0.05 | \* |
| p < 0.01 | \*\* |
| p < 0.001 | \*\*\* |
| p < 0.0001 | \*\*\*\* |
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
p_to_star <- function(p) {
cut(p, breaks = c(-Inf, 1e-4, 1e-3, 0.01, 0.05, Inf),
labels = c("****", "***", "**", "*", "ns"), right = FALSE) |> as.character()
}
res$symbol <- p_to_star(res$p_holm)
res
```
## Python (seaborn / matplotlib)
```{pyodide}
def p_to_star(p):
for thr, s in [(1e-4, "****"), (1e-3, "***"), (0.01, "**"), (0.05, "*")]:
if p < thr:
return s
return "ns"
res["symbol"] = res["p_holm"].map(p_to_star)
res
```
:::
## 4. Drawing the brackets and symbols
### Knowing the "positions" on a categorical axis
To draw a bracket, you need each group's **position on the horizontal axis (a number)**.
- ggplot2: positions on a categorical axis are **1, 2, 3, …** (in the order of the factor levels)
- seaborn: positions on a categorical axis are **0, 1, 2, …**
Set the height a little above the maximum of the data of the two groups being compared, and **shift each bracket up one step** so they don't overlap.
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
ymax <- max(peng$body_mass_g)
step <- 0.08 * diff(range(peng$body_mass_g)) # bracket spacing (8% of the data range)
brk <- res |>
mutate(x1 = as.integer(factor(group1, levels(peng$Species))), # group positions (1, 2, 3)
x2 = as.integer(factor(group2, levels(peng$Species))),
y = ymax + step * seq_len(n())) # one step up each
brk
ggplot(peng, aes(Species, body_mass_g)) +
geom_boxplot(outlier.shape = NA, width = 0.5) +
geom_point(position = position_jitter(width = 0.12, height = 0, seed = 1), alpha = 0.4, size = 1) +
geom_segment(data = brk, aes(x = x1, xend = x2, y = y, yend = y), inherit.aes = FALSE) + # horizontal line
geom_segment(data = brk, aes(x = x1, xend = x1, y = y, yend = y - step * 0.25), inherit.aes = FALSE) + # left tick
geom_segment(data = brk, aes(x = x2, xend = x2, y = y, yend = y - step * 0.25), inherit.aes = FALSE) + # right tick
geom_text(data = brk, aes(x = (x1 + x2) / 2, y = y, label = symbol),
vjust = ifelse(brk$symbol == "ns", -0.4, 0.2), size = 4.5, inherit.aes = FALSE) +
labs(x = NULL, y = "Body mass (g)")
```
## Python (seaborn / matplotlib)
```{pyodide}
levels = list(peng["Species"].cat.categories)
ymax = peng["body_mass_g"].max()
step = 0.08 * (peng["body_mass_g"].max() - peng["body_mass_g"].min())
def bracket(ax, x1, x2, y, text, h, fontsize=13):
"""Draw a bracket at height y between x1 and x2 (both ends bent down by h) and the text"""
ax.plot([x1, x1, x2, x2], [y - h, y, y, y - h], color="black", lw=1)
ax.text((x1 + x2) / 2, y, text, ha="center", va="bottom", fontsize=fontsize if text != "ns" else 10)
fig, ax = plt.subplots()
sns.boxplot(data=peng, x="Species", y="body_mass_g", showfliers=False, width=0.5, fill=False, color="black", ax=ax)
np.random.seed(1)
sns.stripplot(data=peng, x="Species", y="body_mass_g", jitter=0.12, alpha=0.4, size=3, ax=ax)
for i, r in res.iterrows():
x1, x2 = levels.index(r["group1"]), levels.index(r["group2"]) # group positions (0, 1, 2)
bracket(ax, x1, x2, ymax + step * (i + 1), r["symbol"], h=step * 0.25)
ax.set(xlabel="", ylabel="Body mass (g)")
ax.set_ylim(top=ymax + step * (len(res) + 1.5)) # make room at the top
plt.show()
```
:::
::: {.callout-tip}
## The order of stacking brackets
If you put the long bracket (Adelie–Gentoo) at the bottom, the symbol above it collides with the short brackets. The basic rule is to stack **short brackets at the bottom and long ones on top**. If a symbol is too close to the bracket above, widen the bracket spacing (`step`).
:::
::: {.callout-tip}
## Positions based on "data values" work for any figure
Because the bracket heights are computed from the data maximum and range, the same code works even if you change the variable (flipper length, bill depth).
If you compute `ymax` and `step` from each group's maximum, you can also keep the brackets more compact.
:::
### Wrapping it in a function
Since we use this repeatedly, we make functions that build the figure from "the data, the group column, the value column, and the table of comparisons".
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
add_brackets <- function(p, res, levels, yvals, step_frac = 0.08, tip = 0.25, size = 4.5) {
ymax <- max(yvals, na.rm = TRUE); step <- step_frac * diff(range(yvals, na.rm = TRUE))
b <- res |> mutate(x1 = match(group1, levels), x2 = match(group2, levels), y = ymax + step * seq_len(n()))
p +
geom_segment(data = b, aes(x = x1, xend = x2, y = y, yend = y), inherit.aes = FALSE) +
geom_segment(data = b, aes(x = x1, xend = x1, y = y, yend = y - step * tip), inherit.aes = FALSE) +
geom_segment(data = b, aes(x = x2, xend = x2, y = y, yend = y - step * tip), inherit.aes = FALSE) +
geom_text(data = b, aes(x = (x1 + x2) / 2, y = y, label = symbol), vjust = -0.1, size = size, inherit.aes = FALSE)
}
pairwise_res <- function(d, group, value, pairs) {
r <- do.call(rbind, lapply(pairs, function(pr) {
data.frame(group1 = pr[1], group2 = pr[2],
p = t.test(d[[value]][d[[group]] == pr[1]], d[[value]][d[[group]] == pr[2]])$p.value)
}))
r$p_holm <- p.adjust(r$p, "holm"); r$symbol <- p_to_star(r$p_holm); r
}
r_fl <- pairwise_res(peng, "Species", "flipper_length_mm", pairs)
p <- ggplot(peng, aes(Species, flipper_length_mm, fill = Species)) +
geom_violin(alpha = 0.4, colour = NA) + geom_boxplot(width = 0.15, outlier.shape = NA, fill = "white") +
theme(legend.position = "none") + labs(x = NULL, y = "Flipper length (mm)")
add_brackets(p, r_fl, levels(peng$Species), peng$flipper_length_mm)
```
## Python (seaborn / matplotlib)
```{pyodide}
def pairwise_res(d, group, value, pairs):
rows = [{"group1": a, "group2": b,
"p": stats.ttest_ind(d.loc[d[group] == a, value], d.loc[d[group] == b, value], equal_var=False).pvalue}
for a, b in pairs]
r = pd.DataFrame(rows); r["p_holm"] = multipletests(r["p"], method="holm")[1]
r["symbol"] = r["p_holm"].map(p_to_star); return r
def add_brackets(ax, res, levels, yvals, step_frac=0.08, tip=0.25):
ymax = np.nanmax(yvals); step = step_frac * (np.nanmax(yvals) - np.nanmin(yvals))
for i, r in enumerate(res.itertuples()):
bracket(ax, levels.index(r.group1), levels.index(r.group2), ymax + step * (i + 1), r.symbol, h=step * tip)
ax.set_ylim(top=ymax + step * (len(res) + 1.5))
r_fl = pairwise_res(peng, "Species", "flipper_length_mm", pairs)
fig, ax = plt.subplots()
sns.violinplot(data=peng, x="Species", y="flipper_length_mm", hue="Species", alpha=0.4, inner=None, linewidth=0, legend=False, ax=ax)
sns.boxplot(data=peng, x="Species", y="flipper_length_mm", width=0.15, showfliers=False, color="white", ax=ax)
add_brackets(ax, r_fl, levels, peng["flipper_length_mm"])
ax.set(xlabel="", ylabel="Flipper length (mm)")
plt.show()
```
:::
## When each group is split in two (dodge)
In a figure comparing **males and females** within each species, the boxes are shifted sideways, so the brackets also need to be at **the shifted positions**.
- ggplot2: with `position_dodge(width = 0.75)` and 2 groups, **± 0.75 / 4 = ± 0.1875** from the centre
- seaborn: with 2 groups via `hue` (default width 0.8), **± 0.2** from the centre
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
sex_res <- peng |> group_by(Species) |>
summarise(p = t.test(body_mass_g ~ Sex)$p.value, y = max(body_mass_g)) |>
mutate(symbol = p_to_star(p), x = as.integer(Species)) # one comparison per species (3 in total)
sex_res
off <- 0.75 / 4; h <- 60
ggplot(peng, aes(Species, body_mass_g, fill = Sex)) +
geom_boxplot(outlier.shape = NA, position = position_dodge(0.75), width = 0.6) +
geom_point(position = position_jitterdodge(jitter.width = 0.15, dodge.width = 0.75, seed = 1),
alpha = 0.4, size = 0.8) +
geom_segment(data = sex_res, aes(x = x - off, xend = x + off, y = y + 2 * h, yend = y + 2 * h), inherit.aes = FALSE) +
geom_segment(data = sex_res, aes(x = x - off, xend = x - off, y = y + 2 * h, yend = y + h), inherit.aes = FALSE) +
geom_segment(data = sex_res, aes(x = x + off, xend = x + off, y = y + 2 * h, yend = y + h), inherit.aes = FALSE) +
geom_text(data = sex_res, aes(x = x, y = y + 2 * h, label = symbol), vjust = -0.1, size = 4.5, inherit.aes = FALSE) +
labs(x = NULL, y = "Body mass (g)", fill = "Sex")
```
## Python (seaborn / matplotlib)
```{pyodide}
fig, ax = plt.subplots()
sns.boxplot(data=peng, x="Species", y="body_mass_g", hue="Sex", showfliers=False, width=0.8, ax=ax)
np.random.seed(1)
sns.stripplot(data=peng, x="Species", y="body_mass_g", hue="Sex", dodge=True, jitter=0.1,
size=2.5, alpha=0.5, palette=["0.3", "0.3"], legend=False, ax=ax)
off, h = 0.2, 60
for i, sp in enumerate(levels):
d = peng[peng["Species"] == sp]
p = stats.ttest_ind(d.loc[d["Sex"] == "Female", "body_mass_g"], d.loc[d["Sex"] == "Male", "body_mass_g"],
equal_var=False).pvalue
bracket(ax, i - off, i + off, d["body_mass_g"].max() + 2 * h, p_to_star(p), h=h)
ax.set(xlabel="", ylabel="Body mass (g)")
ax.set_ylim(top=peng["body_mass_g"].max() + 6 * h)
ax.legend(title="Sex", loc="upper left")
plt.show()
```
:::
::: {.callout-note}
## Do these 3 comparisons need a correction?
The 3 tests "comparing males and females within each species" are sometimes regarded as separate questions and left uncorrected, and sometimes corrected together. State which you did in the figure's description.
:::
## Using a package: ggsignif
R's `ggsignif` draws brackets and asterisks for you with `geom_signif()`.
```{webr}
library(ggsignif)
ggplot(peng, aes(Species, body_mass_g)) +
geom_boxplot(outlier.shape = NA, width = 0.5) +
geom_signif(comparisons = pairs, test = "t.test", map_signif_level = TRUE, step_increase = 0.08) +
labs(x = NULL, y = "Body mass (g)", title = "Leaving the tests to ggsignif as well (no correction)")
```
::: {.callout-warning}
## If you leave the tests to the package, there is no correction
`geom_signif(test = ...)` **tests each comparison independently and applies no multiple-comparison correction**. The kind of t-test (Welch or not) is also left to the defaults.
The safe approach is to run the tests yourself and pass only the results via `annotations =` (symbols), `y_position =` (heights) and `xmin =`, `xmax =` (positions).
:::
```{webr}
ggplot(peng, aes(Species, body_mass_g)) +
geom_boxplot(outlier.shape = NA, width = 0.5) +
geom_signif(xmin = match(res$group1, levels(peng$Species)), xmax = match(res$group2, levels(peng$Species)),
y_position = max(peng$body_mass_g) + c(150, 400, 650),
annotations = res$symbol, tip_length = 0.02, textsize = 4.5) +
labs(x = NULL, y = "Body mass (g)", title = "Passing results you tested and corrected yourself")
```
In Python, the `statannotations` package plays the same role, but it often doesn't work with seaborn 0.13, so this hands-on recommends drawing them yourself as with `bracket()` above.
## What to think about before adding asterisks
::: {.callout-warning}
## Asterisks throw away a lot of information
- Asterisks only convey "which threshold the p-value is below". **The size, direction and confidence interval of the difference** are unknown
- "ns" does not mean "no difference" (Chi-square test hands-on, Chapter 8)
- Drawing brackets for many pairs makes the figure unreadable and makes the multiple-comparison problem bigger. **Decide the pairs to compare in advance**
- In the figure's description, state **the name of the test, the correction method, the symbol mapping and the sample size of each group**
- If possible, show **the p-value itself** (p = 0.003) or **the difference and its 95% confidence interval** instead of asterisks
:::
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
# Example: write p-values instead of asterisks
brk$label <- ifelse(brk$p_holm < 0.001, "p < 0.001", sprintf("p = %.2f", brk$p_holm))
ggplot(peng, aes(Species, body_mass_g)) +
geom_boxplot(outlier.shape = NA, width = 0.5) +
geom_point(position = position_jitter(width = 0.12, height = 0, seed = 1), alpha = 0.4, size = 1) +
geom_segment(data = brk, aes(x = x1, xend = x2, y = y, yend = y), inherit.aes = FALSE) +
geom_text(data = brk, aes(x = (x1 + x2) / 2, y = y, label = label), vjust = -0.4, size = 3.5, inherit.aes = FALSE) +
labs(x = NULL, y = "Body mass (g)", caption = "Welch's t-test, Holm correction")
```
## Python (seaborn / matplotlib)
```{pyodide}
fig, ax = plt.subplots()
sns.boxplot(data=peng, x="Species", y="body_mass_g", showfliers=False, width=0.5, fill=False, color="black", ax=ax)
for i, r in res.iterrows():
lab = "p < 0.001" if r["p_holm"] < 0.001 else f"p = {r['p_holm']:.2f}"
bracket(ax, levels.index(r["group1"]), levels.index(r["group2"]), ymax + step * (i + 1), lab, h=step * 0.25, fontsize=10)
ax.set_ylim(top=ymax + step * 4.5)
ax.set(xlabel="", ylabel="Body mass (g)")
fig.text(0.99, 0.01, "Welch's t-test, Holm correction", ha="right", fontsize=9, color="0.4")
plt.show()
```
:::
::: {.callout-important}
## Chapter summary
- **Run the tests and corrections yourself**, and draw brackets and symbols from the results table
- Category positions are **1, 2, 3** in ggplot2 and **0, 1, 2** in seaborn. When dodged, ±(width / 4) or ±0.2 from the centre
- Compute bracket heights from **the data maximum and range**, shift each one up a step, and make room at the top
- If you leave the tests to `ggsignif`, **there is no correction**. Pass symbols and positions instead
- **p-values, or differences with confidence intervals,** carry more information than asterisks. Always state the test, the correction and what the symbols mean
:::