Multiple Testing: A Hands-on Course

From why false positives increase to FWER and FDR, post hoc comparisons, hierarchical testing in clinical trials, subgroups, and interim analyses

Published

October 17, 2026

Introduction

English version — machine-assisted translation (by generative AI, Anthropic Claude) of the original Japanese hands-on. 日本語版はこちら.

In research, you almost never stop at a single test. There are several outcomes, groups are compared two at a time, analyses are repeated by sex and by age, the data are checked partway through… Even if each individual test is correct, running many tests and reporting the significant ones lets chance findings slip in.

  • If you run 10 tests, how large does the probability of a false positive become?
  • How do Bonferroni, Holm, Hochberg, and BH differ?
  • When comparing three or more groups, how do you choose between Tukey and Dunnett?
  • In a clinical trial, what must you do to claim an effect on a secondary endpoint?
  • How far can you trust the results of a subgroup analysis?
  • How can you keep the significance level at 5% even when you perform interim analyses?

In this hands-on, using R running in your browser and simulations, you will experience the multiplicity problem, compare ways of dealing with it, and learn how to plan and report.

Goals of this hands-on

  1. Explain the mechanism by which running many tests increases false positives
  2. Choose appropriately between FWER (Bonferroni, Holm, Hochberg) and FDR (BH)
  3. Select Tukey, Dunnett, or Holm for comparisons of three or more groups
  4. Explain the fixed-sequence procedure, graphical approaches, and co-primary endpoints in clinical trials
  5. Notice the multiplicity in subgroup analyses and analytic choices, and plan for it in advance
  6. Explain the idea behind interim-analysis boundaries (Pocock, O’Brien–Fleming)
  7. Write up how multiplicity was handled in protocols and papers

Who this is for

  • People involved (or about to be involved) in analysing clinical research, epidemiological, or healthcare data
  • People with a rough understanding of what p-values, significance levels, and t-tests mean (around the level of an introductory statistics certificate)
  • For R, it is enough if you can “read code, modify it a little, and run it”

R runs in your browser

The code cells in each chapter are executed by R inside your browser (a technology called webR). Edit the code and press “Run Code”, and the results change immediately. No installation is needed.

Note
  • Right after opening each chapter, loading R and the packages takes several tens of seconds to about a minute
  • Simulation cells repeat thousands of tests inside the browser, so they can take a few seconds to a dozen or so seconds
  • Viewing on a PC is recommended

Data used

Data Description Main chapters
Simulation Data generated after you decide the “true effect” yourself. Used to check false-positive probabilities and power All chapters
NHANES 2009–2012 A real sample from the US National Health and Nutrition Examination Survey (about 20,000 people). BMI and blood pressure by race/ethnicity, education, and self-rated health 4, 9
WarningA note on the data

The NHANES sample oversamples some racial and age groups. In this hands-on, to learn how multiple comparisons work, we tabulate without using the survey weights, so the numbers are not values for the US population (weights are covered in Chapter 5 of the Descriptive Statistics hands-on (in Japanese)).

Chapter overview

Chapter Contents
1 Why false positives increase (FWER, the effect of correlation, places where multiplicity hides)
2 Controlling the FWER: Bonferroni, Holm, Hochberg, Hommel, adjusted confidence intervals
3 Many tests and the FDR: the Benjamini–Hochberg procedure, choosing between it and the FWER
4 Comparing three or more groups: Tukey, Dunnett, the pitfall of the protected LSD
5 Multiplicity in clinical trials: the fixed-sequence procedure, weighted Holm (graphical approach), co-primary endpoints
6 Subgroup analyses and the “garden of forking paths”: testing for interaction, p-hacking
7 Interim analyses: looking at the data many times, the Pocock and O’Brien–Fleming boundaries, overestimation from early stopping
8 When to adjust, and how to report
9 Exercises and a multiplicity checklist

Each chapter follows the flow “experience the multiplicity problem through simulation → compare remedies → apply them to real data or clinical-trial settings”.

Related hands-ons (in Japanese): χ² test hands-on (multiple comparisons of proportions in Chapter 8), Sample size and power hands-on (sample sizes for interactions and co-primary endpoints), Confounding hands-on (interactions and subgroups).