---
webr:
packages: [dplyr, ggplot2, patchwork]
resources: [R/setup.R, py/setup.py, data/penguins.csv, data/nhanes_adults.csv, fonts/NotoSansJP-Regular.ttf, fonts/NotoSansJP-Bold.ttf]
cell-options:
autorun: true
pyodide:
packages: [pandas, matplotlib, seaborn]
cell-options:
autorun: true
---
{{< include ./_extensions/r-wasm/live/_knitr.qmd >}}
# Internals: statistical transformations, axis ranges, coordinate transforms, drawing objects {#sec-deep}
```{webr}
#| include: false
source("R/setup.R")
pg <- penguins |> filter(!is.na(body_mass_g))
```
```{pyodide}
#| include: false
exec(open("py/setup.py").read())
pg = penguins.dropna(subset=["body_mass_g"]).copy()
```
This chapter explains, from the internal mechanisms, the places in earlier chapters where you may have wondered "why is it written this way?". Once you understand this, you will know **what to fix** when things don't turn out as you want.
## ggplot2's statistical transformations: what is computed before drawing
`geom_histogram()`, `geom_boxplot()` and `geom_smooth()` don't draw the data as is; they draw **the result of summarising it with a statistical transformation (stat)**.
With `layer_data()`, you can extract the values that are actually drawn.
```{webr}
g <- ggplot(pg, aes(species, body_mass_g)) + geom_boxplot()
layer_data(g)[, c("x", "ymin", "lower", "middle", "upper", "ymax")] # the 5 values of the box plot
```
```{webr}
h <- ggplot(pg, aes(body_mass_g)) + geom_histogram(binwidth = 250)
head(layer_data(h)[, c("xmin", "xmax", "count", "density")]) # count and density of each bin
```
### `after_stat()`: using computed values
Values computed by a statistical transformation (`count`, `density`, `prop`, etc.) can be used in mappings with `after_stat()`.
```{webr}
ggplot(pg, aes(body_mass_g, fill = species)) +
geom_histogram(aes(y = after_stat(density)), binwidth = 200, alpha = 0.6, position = "identity") + # density instead of counts
geom_density(alpha = 0, linewidth = 0.8, aes(colour = species), show.legend = FALSE) +
labs(x = "Body mass (g)", y = "Density")
```
```{webr}
ggplot(pg, aes(species, body_mass_g)) +
geom_point(position = position_jitter(width = 0.15, seed = 1), alpha = 0.3) +
stat_summary(fun.data = mean_se, geom = "errorbar", width = 0.2, colour = "firebrick") + # compute and draw mean ± SE
stat_summary(fun = mean, geom = "point", size = 3, colour = "firebrick") +
labs(x = NULL, y = "Body mass (g)", caption = "Red: mean ± SE")
```
## The axis-range trap: `ylim()` vs `coord_cartesian()`
Suppose you want to narrow the vertical axis of the box plot to 3000–5000 g. ggplot2 has two ways to do this, and **they give different results**.
```{webr}
library(patchwork)
g1 <- g + labs(title = "As is")
g2 <- g + ylim(3000, 5000) + labs(title = "ylim(3000, 5000)")
g3 <- g + coord_cartesian(ylim = c(3000, 5000)) + labs(title = "coord_cartesian(ylim = ...)")
g1 + g2 + g3
```
```{webr}
rbind(as_is = layer_data(g)$middle,
ylim = suppressWarnings(layer_data(g + ylim(3000, 5000)))$middle,
coord_cartesian = layer_data(g + coord_cartesian(ylim = c(3000, 5000)))$middle)
```
`ylim()` (the same as `scale_y_continuous(limits = )`) **discards the data outside the range first** and then applies the statistical transformation. As a result, Gentoo's **median changed from 5000 to 4700**.
`coord_cartesian()` **only zooms in** on the already computed figure, so the median doesn't change.
The warning "Removed n rows containing non-finite values" is a sign that data were discarded. **To narrow the range, use `coord_cartesian()`**.
::: {.callout-note}
## In matplotlib and seaborn
matplotlib's `ax.set_ylim()` only changes the **display range** after drawing, so it is the same as `coord_cartesian()`.
The same mistake happens when you **filter the data before drawing** (e.g. drawing `df[df.y < 5000]`).
:::
## matplotlib coordinate transforms: where to put text
Every drawing in matplotlib converts the values you specify into screen positions through a **coordinate transform (transform)**. There are 4 main ones.
| Coordinates | Specify with | (0, 0) and (1, 1) | When to use |
|---|---|---|---|
| Data | `ax.transData` (default) | The axis values themselves | Points, lines, annotations tied to data |
| Panel | `ax.transAxes` | Bottom left / top right of the panel | Equations, panel labels, legend positions |
| Whole figure | `fig.transFigure` | Bottom left / top right of the figure | Headings and notes for the whole figure |
| Blended | `ax.get_xaxis_transform()` | x in data, y in panel | Keeping the label of a vertical line always at the top of the panel |
```{pyodide}
fig, ax = plt.subplots()
sns.histplot(data=pg, x="body_mass_g", binwidth=200, color="0.7", ax=ax)
m = pg["body_mass_g"].mean()
ax.axvline(m, color="firebrick")
# x is a data value (the mean), y is just below the top of the panel → stays at the top even if the vertical range changes
ax.text(m, 0.97, f" Mean {m:.0f} g", transform=ax.get_xaxis_transform(), color="firebrick", va="top")
ax.text(0.98, 0.97, "Panel coordinates (0.98, 0.97)", transform=ax.transAxes, ha="right", va="top", fontsize=9)
fig.text(0.01, 0.01, "Figure coordinates (0.01, 0.01)", fontsize=9, color="0.4")
ax.annotate("Offset from a point (points)", xy=(6000, 5), xytext=(-60, 40), textcoords="offset points",
arrowprops=dict(arrowstyle="->"), fontsize=9)
ax.set(xlabel="Body mass (g)", ylabel="Count")
plt.show()
```
ggplot2 has no mechanism for specifying coordinate transforms directly, but combining `x = -Inf`/`Inf` (the panel edges) with `hjust`/`vjust` gets you close to panel coordinates (Chapter 6).
## matplotlib drawing objects: changing things after drawing
A matplotlib figure is a tree of objects (Artists): **Figure → Axes → lines, points, rectangles, text**.
Even for figures drawn with seaborn, you can explore this tree and change things one by one afterwards.
```{pyodide}
fig, ax = plt.subplots()
sns.boxplot(data=pg, x="species", y="body_mass_g", ax=ax)
print("rectangles (boxes):", len(ax.patches), " lines:", len(ax.lines), " point collections:", len(ax.collections))
# change the box fills afterwards and add hatching
for patch, color, hatch in zip(ax.patches, ["#E69F00", "#56B4E9", "#009E73"], ["", "//", ".."]):
patch.set_facecolor(color); patch.set_alpha(0.6); patch.set_hatch(hatch)
ax.set(xlabel="", ylabel="Body mass (g)")
plt.show()
```
::: {.callout-tip}
## Stacking order (zorder) and overflow (clip_on)
- `zorder=`: larger values are drawn in front. Put grid lines behind the points and annotations at the very front
- `clip_on=False`: draw beyond the edge of the panel (for symbols and labels placed outside the panel)
:::
## Building a legend yourself
Sometimes you want an element in the legend that isn't in the data (such as "black dashed line: overall regression line").
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
ggplot(pg, aes(flipper_length_mm, body_mass_g)) +
geom_point(aes(colour = species), alpha = 0.5) +
geom_smooth(aes(linetype = "Overall regression line"), method = "lm", formula = y ~ x, se = FALSE, colour = "black") +
scale_linetype_manual(name = NULL, values = c("Overall regression line" = 2)) + # create a legend for the line type
guides(colour = guide_legend(override.aes = list(alpha = 1, size = 3))) + # make only the legend's points darker and larger
labs(x = "Flipper length (mm)", y = "Body mass (g)", colour = "Species")
```
In ggplot2, you create a legend by **putting a constant in `aes()`** (deliberately using the style introduced as a mistake in Chapter 2), and set the appearance with `scale_xxx_manual()`.
`guides(... override.aes = )` changes **only the legend's keys**, while the points in the plot stay faint.
## Python (seaborn / matplotlib)
```{pyodide}
from matplotlib.lines import Line2D
fig, ax = plt.subplots()
sns.scatterplot(data=pg, x="flipper_length_mm", y="body_mass_g", hue="species", alpha=0.5, ax=ax)
sns.regplot(data=pg, x="flipper_length_mm", y="body_mass_g", scatter=False, ci=None,
line_kws={"color": "black", "ls": "--"}, ax=ax)
handles, labels = ax.get_legend_handles_labels() # the legend elements seaborn created
handles.append(Line2D([], [], color="black", ls="--")) # add an element yourself
labels.append("Overall regression line")
for h in handles[:3]:
h.set_alpha(1) # make only the legend's points darker
ax.legend(handles, labels, loc="upper left")
ax.set(xlabel="Flipper length (mm)", ylabel="Body mass (g)")
plt.show()
```
A matplotlib legend is **a list of keys (handles) and texts (labels)**. With `Line2D` and the like, you can freely create keys that aren't in the plot.
:::
## seaborn.objects: using a "grammar" in Python
seaborn 0.12 added `seaborn.objects`, which lets you write plots with the same **grammar of graphics** approach as ggplot2.
```{pyodide}
import seaborn.objects as so
(
so.Plot(pg.dropna(subset=["bill_length_mm", "bill_depth_mm"]),
x="bill_length_mm", y="bill_depth_mm", color="species")
.add(so.Dot(alpha=0.5)) # geometry: points
.add(so.Line(), so.PolyFit(order=1)) # geometry: line; stat: degree-1 polynomial (straight line)
.label(x="Bill length (mm)", y="Bill depth (mm)", color="Species")
.layout(size=(7, 4.2))
.show()
)
```
```{pyodide}
(
so.Plot(pg, x="species", y="body_mass_g", color="sex")
.add(so.Dot(pointsize=3, alpha=0.4), so.Jitter(0.3), so.Dodge()) # jittered points
.add(so.Range(), so.Est(errorbar="sd"), so.Dodge()) # mean ± SD
.add(so.Dot(pointsize=8), so.Agg(), so.Dodge()) # mean
.label(x="", y="Body mass (g)", color="Sex")
.layout(size=(7, 4.2))
.show()
)
```
| ggplot2 | seaborn.objects |
|---|---|
| `ggplot(data, aes(x, y, colour))` | `so.Plot(data, x=, y=, color=)` |
| `+ geom_point()` | `.add(so.Dot())` |
| `+ geom_smooth(method = "lm")` | `.add(so.Line(), so.PolyFit(1))` |
| `position_jitter` / `position_dodge` | `so.Jitter()` / `so.Dodge()` |
| `stat_summary()` | `so.Agg()`, `so.Est()` |
| `facet_wrap(~ v)` | `.facet(col=)` |
| `labs()` | `.label()` |
It doesn't yet have as many features as ggplot2, but the advantage is that **you can write both languages with the same way of thinking**.
If you want to use R's grammar as is in Python, there is also **plotnine**, a near-direct port of ggplot2 (it is not available in the browser Python on this page).
## Turning it into a function for reuse
If you make figures of the same form many times, turn them into a function. In ggplot2, an easy-to-understand approach is to **take column names as strings and use them with `.data[[ ]]`**.
::: {.panel-tabset group="lang"}
## R (ggplot2)
```{webr}
box_with_points <- function(d, group, value, ylab = value) {
ggplot(d, aes(.data[[group]], .data[[value]], colour = .data[[group]])) +
geom_boxplot(outlier.shape = NA, width = 0.5) +
geom_point(position = position_jitter(width = 0.12, height = 0, seed = 1), alpha = 0.4) +
theme(legend.position = "none") +
labs(x = NULL, y = ylab)
}
box_with_points(pg, "species", "flipper_length_mm", "Flipper length (mm)") +
box_with_points(pg, "island", "body_mass_g", "Body mass (g)")
```
## Python (seaborn / matplotlib)
```{pyodide}
def box_with_points(d, group, value, ylab=None, ax=None):
ax = ax or plt.gca()
sns.boxplot(data=d, x=group, y=value, hue=group, showfliers=False, width=0.5, fill=False, legend=False, ax=ax)
np.random.seed(1)
sns.stripplot(data=d, x=group, y=value, hue=group, jitter=0.12, alpha=0.4, legend=False, ax=ax)
ax.set(xlabel="", ylabel=ylab or value)
return ax # returning the axes lets you adjust it later
fig, axes = plt.subplots(1, 2, figsize=(10, 4), layout="constrained")
box_with_points(pg, "species", "flipper_length_mm", "Flipper length (mm)", ax=axes[0])
box_with_points(pg, "island", "body_mass_g", "Body mass (g)", ax=axes[1])
plt.show()
```
If matplotlib functions **take `ax=` and return `ax`**, they can be plugged into any layout (the same design as seaborn's axes-level functions).
:::
::: {.callout-important}
## Chapter summary
- Many ggplot2 geoms draw the result of a **statistical transformation**. Inspect it with `layer_data()` and use it with `after_stat()`
- **`ylim()` discards data before computing**. Narrow the range with `coord_cartesian()` (matplotlib's `set_ylim()` only affects the display)
- In matplotlib, choose among **coordinate transforms**: data, panel (`transAxes`), whole figure, blended
- A matplotlib figure is a **tree of Artists**. seaborn's output can also be changed piece by piece afterwards. A legend is a list of handles and labels
- `seaborn.objects` is a grammar for Python. When writing functions, use `.data[[ ]]` in R; in Python, take `ax=` and return it
:::