Personal website – any views expressed herein are my own unless otherwise obvious
“When the nonresponse rate is 91%, any sample is a convenience sample.” (Emphasis original.)
“Remember: just about no sample of humans is really a probability sample or even close to a probability sample, and just about no regression model applied to humans is correct or even close to correct. So we have to worry about sampling, and we have to worry about adjustment.”
(Gelman, blog [re]post, 2015)
If you need evaluation findings sooner than the time it takes (1) to recruit enough participants for adequate statistical power, plus (2) the time for the final participant to complete the programme and have their outcome measured, plus (3) analysis and reporting time, then you need to conduct a literature review instead. Though you may want to run an evaluation too, so there is relevant literature to review in future.
The paragraph above summarises many talks on policy timelines, past and future. See also Bending evaluation timelines to meet policy deadlines.
The “change” in “theory of change” is often interpreted as meaning change over time, which can be misleading. If a programme is designed to prevent an outcome from occurring (for example, a worsening of mental health), then there may be no pre-post change at all. Alternatively, any observed pre-post change may be driven by regression to the mean or by factors other than the programme under evaluation.
In a counterfactual impact evaluation, the “change” of interest is the expected or desired causal effect (or “treatment” effect). This is the difference between two potential outcomes: the outcome that would occur under exposure to the programme and the outcome that would occur under the control or comparison condition. Rather than describing outcomes in isolation, the theory should define the contrast between these two potential outcomes, for example whether the programme outcome is expected to be higher or lower than the control outcome.
Taking a logic model list, this would mean replacing the usual outcomes and impacts with causal effects on various time scales. It needs a name though. Some ideas:
Alternatively, maybe it’s time to return to “programme theory” or just “theory”.
Just use AI – to compose a literature review (it’s accurate some of the time), to write programs (why learn a language), to analyse qual interviews (it saves you from listening), to analyse quant data (it saves you from learning statistics), to synthesise data (it saves time on fieldwork), to review others’ work (which used AI), to email a colleague or client (it corrects your horribly human tone), to summarise meetings (nobody writes notes, so they will forget anything it misrepresents), to generate insights (others may have had, but few will check or read them anyway), to write polished job applications (that edit you out), to compose poetry that rhymes (no more struggling to find words) or doesn’t (just use the right prompt), to create art (that reminds you of exhibitions you once saw), to tell stories (like authors you used to read), to write letters to loved ones, to apologise with no effort, to make important decisions, to think, to dream. Just use AI, it will make you happier and more productive and free up time to do something else—ask AI for ideas. If you want it more bleak, lighter, or academic, tell me which direction and I’ll take it there.
Currently in the news (11 Oct 2026):
“Andy Burnham has said Russia ‘clearly’ interfered in the 2016 Brexit referendum […]. The comments, made during an appearance on the Rest is Politics podcast, are thought to be the first time a PM has confirmed meddling by the Kremlin […].”
Jeremy Wright (24 Oct 2018), then Secretary of State for Digital, Culture, Media and Sport, when asked whether Russia has interfered in UK elections:
“What is undoubtedly true is that there have been efforts by Russia to interfere in the process. […] We cannot, at the moment, say that there is evidence that it has been successful, although I accept that it is difficult to know exactly what people have read and what they have been influenced by.”
A sample from the DCMS Disinformation and ‘fake news’ final report (14 Feb 2019):
“Kremlin-aligned media published significant numbers of unique articles about the EU referendum. 89 Up researchers analysed the most shared of the articles, and identified 261 with a clear anti-EU bias to the reporting. The two main outlets were RT and Sputnik, with video produced by Ruptly.”
Here’s a key slide in the 89 Up report it cites:
“glmnet (Friedman et al. 2010) is a powerful R package for learning sparse models in supervised learning, but only its most basic capabilities are widely used. This book describes the full range of glmnet’s capabilities and provides code which readers can adapt for their own purposes.”
“ivs (said, “eye-vees”) is a package dedicated to working with intervals in a generic way. It introduces a new type, the interval vector, which is generally referred to as an iv. An iv is generally created from two parallel vectors representing the starts and ends of the intervals…”
Examples:
> iv_set_union(iv(2, pi), iv(3, 5))
[1] [2, 5)
> iv_set_union(iv(2, pi), iv(4, 5))
[1] [2, 3.141593) [4, 5.000000)
> iv_set_intersect(iv(0, pi), iv(3, 1000))
[1] [3, 3.141593)
> iv_set_union(iv(0, pi), iv(3, 1000))
[1] [0, 1000)
> iv_between(exp(1), c(iv(0, pi)))
[1] TRUE
> iv_between(pi, c(iv(0, pi)))
[1] FALSEINSPECT-SR (Wilkinson et al., 2026) is a new checklist for assessing the trustworthiness of RCTs in systematic reviews. It consists of 21 items across four domains to help reviewers decide whether to include a trial in a review.
Wilkinson, J., Bero, L., Heal, C., Flemyng, E., Antoniou, G. A., Aburrow, T., Alfirevic, Z., Avenell, A., Barbour, V., Berghella, V., Bishop, D. V. M., Bordewijk, E. M., Brown, N. J. L., Christopher, J., Clarke, M., Dahly, D., Dennis, J., Dicker, P., Dumville, J., … Kirkham, J. J. (2026). INSPECT-SR tool for assessing trustworthiness of randomised controlled trials. BMJ, 394, e100611.
