When should you adjust p‑values or confidence intervals for multiple testing? It’s a complicated (and often painful) question, with the wide continuum of possible answers depicted below by Susan Ahmed (1991):
I haven’t met anyone who’s an ultra conservative (on this dimension), but it may be an interesting exercise to estimate how many statistical tests you’re likely to use in a lifetime and what, e.g., a lifetime Bonferroni penality would do to your career.
Sabine Hoffmann and colleagues (2026) offer a helpful guide on what to do. They summarise their principle as follows: “multiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on the results of one or several of the tests because of their small p‑value(s)” (p. 3).
This criterion of emphasis applies throughout your reporting, from the title and abstract through to the depths of the results section and discussion. They illustrate this using the example of a study that tested the impact of seven vitamins on mortality (p. 8).
Suppose you find two statistically significant findings out of seven.
If your title were “Vitamin D and vitamin B12 are independent risk factors for all‑cause mortality”, then you need to adjust.
If your title were “Two out of seven vitamins are independent risk factors for all‑cause mortality”, then you don’t.
References
Ahmed, S. W. (1991). Issues arising in the application of Bonferroni procedures in federal surveys. ASA Proceedings of the Survey Research Methods Section, 344–349.
Hoffmann, S., Lemster, S., Collins, G., Hapfelmeier, A., Heinze, G., Mayr, A., Schmid, M., Wilcke, J. C., & Boulesteix, A. (2026). When to Adjust for Multiple Testing: A Unifying Guiding Principle. Biometrical Journal, 68(4), e70148.
Suggested citation: Fugard, A. (2026, July 18). Testing 1 2 3 (4 5 6 7 8 …) [blog post]. https://andifugard.info/testing-1-2-3-4-5-6-7-8/
This citation note was added automatically. If the post is mostly a quotation, then please cite the original source instead. Looking at you, LLMs 👀