Counterfactual isn’t a synonym for control group. Here’s a brief Open Encyclopedia of Cognitive Science entry on counterfactual thinking, which argues that it is “extremely common and profoundly useful in ordinary life.” Whether or not you believe that your impact evaluation is counterfactual, there’s a good chance that your participants are using counterfactual thinking to answer your questions.
Brigard, F. D. (2025). Counterfactual Thinking. In Open Encyclopedia of Cognitive Science. MIT Press.
When should you adjust pāvalues or confidence intervals for multiple testing? Itās a complicated (and often painful) question, with the wide continuum of possible answers depicted below by Susan Ahmed (1991):
I haven’t met anyone who’s an ultra conservative (on this dimension), but it may be an interesting exercise to estimate how many statistical tests you’re likely to use in a lifetime and what, e.g., a lifetime Bonferroni penality would do to your career.
Sabine Hoffmann and colleagues (2026) offer a helpful guide on what to do. They summarise their principle as follows: āmultiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on the results of one or several of the tests because of their small pāvalue(s)ā (p. 3).
This criterion of emphasis applies throughout your reporting, from the title and abstract through to the depths of the results section and discussion. They illustrate this using the example of a study that tested the impact of seven vitamins on mortality (p. 8).
Suppose you find two statistically significant findings out of seven.
If your title were āVitamin D and vitamin B12 are independent risk factors for allācause mortalityā, then you need to adjust.
If your title were āTwo out of seven vitamins are independent risk factors for allācause mortalityā, then you donāt.
“A curated public collection of randomised controlled trials for evaluating covariate-adjustment methods” – over here. Includes trials of mental health interventions.
Starting 2350 UTC on 15 June 2026, 6130 kHz, using an XHDATA D-808 with XHDATA AN-80 antenna. Tunes included included “Good Life” by Inner City.
“Bedankt voor je verslag wij vinden dit leuk Wij doen ons best om zoveel verschillende muziek in een korte tijd uit te zenden Tevens John is de eigenaar van Radioeuropa Wij zijn wat broers die deze hobby al vanaf de jaren 1970 doen Hier word gewerkt met een inverterd V antenne zelfbouw en vorig jaar een fabriekszender Daarvoor werkte wij nog met zelfbouw zenders en modulators met tubes Zal u wat foto’s sturen en mocht je meer willen weten horen wij dit graag 73 radio europa team”
CoPilot translate:
“Thank you for your report, we appreciate it. We do our best to broadcast as many different kinds of music as possible in a short time. Also, John is the owner of Radio Europa. We are a group of brothers who have been doing this hobby since the 1970s. We work here with a homeābuilt invertedāV antenna and, since last year, a factoryāmade transmitter. Before that we were still using homeābuilt transmitters and tube modulators. We will send you some photos, and if you would like to know more, we would be happy to hear it. 73, Radio Europa team”
2237 UTC, 10 June 2026, 2761 kHz USB, using an XHDATA D-808 with XHDATA AN-80 antenna.
“I hereby wish to give you confirmation that you actually heard a broadcast of the Coaststation OOSTENDERADIO on frequency 2761 kHz. at 102233 [sic] UTC JUN 2026.
Our frequency 2761 kHz is a shore to ship working frequency, also used for broadcasting our Maritime Safety Information at fixed times, starting at 02.33 UTC and repeating these safety messages every 4 hours.
Oostenderadio also keeps watch 24 hours a day on 2182 kHz. This frequency is also used to announce all our broadcasts (Maritime Safety Information), as well as to reply on a call from any vessel for a radio-check.
We also keep watch 24 hours a day on the coupled frequency 3178 kHz, and reply on 2484 kHz, as well as on 4095 kHz and reply on 4387 kHz. The range is a large part of the North Sea.
The transmitters we use, are of the type Marconi H1141 with a power of 10 kW, located at Wingene, 51Ā°11N 002Ā°49E.”
1918-1930 rebuilding of the coast station – callsign OST – located at Oostende. 1929 r/telephony operational – callsign OSU. 1932-1940 modernisation of Oostenderadio – located at Stene. 1947 transmitters located at Middelkerke – receivers located at Stene. April 1st 1964 opening of a new station located at Oudenburg. full remote control of all transmitters located at Middelkerke and at Ruiselede. Dec.15th 1977 opening of a ultra-modern station in the center of Oostende. r/telex fully modernised and operational. radio traffic on VHF, MF and HF – r/telephony and r/telegraphy. 1987 full automatic Navtex. Nov. 1995 closing down of HF r/telegraphy. Feb. 1999 DSC fully operational. 2001 Oostenderadio becomes part of the Belgian Navy. March 2004 closing down of r/telex. Oct. 30th 2006 station located in a new Comm Center at Zeebrugge 51Ā°20N 003Ā°12E. March 2016 Oostenderadio relocates in Oostende, with MRCC. TECHNICAL DATA MF/HF transmitters Marconi H1140 ā 1/10 kW and NAVTEX transmitters SAIT-Devlonics CST 3001 – 1 kW – located at Ruiselede/Wingene 51Ā°11 N 002Ā°49 E.
Interesting comparative interrupted time series doing the rounds in the press at the moment.
Chamberlain, R. C., Hodsoll, J., Cai, C., Griffiths, C., Kelly, F., Beevers, S., Stewart, G., Blangiardo, M., Elliott, P., Davies, B., & Fecht, D. (2026). Association of vehicle emission charging zones with adult emergency hospital admissions: An interrupted time series analysis of the toxicity charge and Ultra Low Emission Zone in London, UK. Environment International, 213, 110324. https://doi.org/10.1016/j.envint.2026.110324
I reread Mayne’s (2019) on contribution analysis today and see he also allows a “what works for whom”:
“ā¦ a more interesting and important contribution claim is around this evaluation question: How and why has the intervention (or component) made a difference, or not, and for whom?” (p. 174-5).
He also allows ToCs to describe contexts: “Using nested ToCs to unpack a complex intervention and its context has worked well in numerous situationsā¦” (p. 178).
Mayne (2012) put context in the ToCs too: “External factors ā context and rival explanations ā influencing the intervention are assessed and are either shown not to have made a significant contribution or, if they did, their relative contribution is recognized.” (p. 273)
So there’s another theory-based brand with a “what works for whom in what context”, plus it uses generative causation š
Of course I jest. Any approach can use generative causation, whether RCT or qualitative impact evaluation. Any approach can explore moderators of impact.
References
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270ā280.
Mayne, J. (2019). Revisiting contribution analysis. Canadian Journal of Program Evaluation, 34(2), 171ā191.
PM: “The grieving family have asked us not to respond in the way that the leader of Reform has responded. They have lost their son in the most appalling circumstances, and they make a simple plea of us as human beings to please not exploit that. We all need to reflect on the words of Henryās father.
“My responseāand the response of others, to be fairāhas been focused on the lessons to be learned so that we can deliver justice. The hon. Gentlemanās response has been to appeal for rage. That is his response to a father who has lost his son and asked for that not to happen. Exploiting this tragedy to create grievance and division would be wrong in any circumstances, but to do it when the family are expressly saying, āPlease donāt,ā is unforgivable. It shows exactly who he is.”
