Personal website – any views expressed herein are my own unless otherwise obvious
Two questions that (bizarrely) seemed to preoccupy a significant number of people when I was a methods lecturer. Here I share the answers:
(A famous trialist and Genstat® user once explained to me that the difference between method and methodology is the same as the difference between transport and transportation.)
Randomised controlled trials (RCTs) are an instance of the counterfactual kind of impact evaluation and contribution analysis of the theory-based kind. For example, the UK’s Magenta Book recommends using theory-based evaluation if you can’t find a comparison group (HM Treasury, 2020, p. 47). The founding texts on contribution analysis present it as using a non-counterfactual approach to causation (e.g., Mayne, 2019, pp. 173–4). In December 2023, I gave a talk to the UK Evaluation Society (UKES) exploring what happens if we challenge this theory-based/counterfactual binary. This post summarises what I said.
Theory is required to select the variables used in RCTs and quasi-experimental designs (QEDs); that is, to decide what data to gather and include in analyses. We need to peek inside the “black box” of a programme to work out what we are evaluating and how. These variables include outcomes, moderators, mediators, competing exposures, and, in QEDs, confounders.
In QEDs, many different model structures are compatible with the data. The slogan “correlation doesn’t imply causation” can be reformulated as A causes B and B causes A are Markov equivalent if all we know is that A and B are correlated. The number of models that are Markov equivalent rises exponentially as the number of variables and statistical associations increases. Theory is needed to select between the models: the data alone cannot tell us.
Although RCTs give us an unbiased estimate of an average treatment effect, that is, the average of each individual’s unmeasurable treatment effect (difference between measured actual outcome and counterfactual outcome), they cannot tell us what that difference represents. To do that, we need a theory of the ingredients and processes in the two programmes being compared; for example, what are the similarities and differences between CBT and humanistic counselling or whatever “treatment as usual” is in practice.
There is a long history of research on counterfactual reasoning in the absence of a control group. How should we determine the truth of a counterfactual such as “If Oswald had not killed Kennedy, no one else would have” (e.g., Adams, 1970)? Clearly we ponder statements like these without running violent RCTs. Another strand of research investigates how children, adults, and animals perform counterfactual reasoning in practice (e.g., Rafetseder, et al., 2010). This research on non-experimental counterfactual reasoning includes literature in evaluation, for instance by White (2010) and more recently Reichardt (2022).
Halpern (2016) introduces a formal framework for defining causal relationships and estimating counterfactual outcomes, regardless of the sources of evidence. The diagram below illustrates the simplest version of the framework where causes and effects are binary. Equations, annotating each node, define the causal relationships and allow counterfactual outcomes to be inferred.
This provides a concrete illustration of why RCTs and QEDs are unnecessary to reason counterfactually; however, the causal model obviously must be correct for the correct conclusions to be drawn, so one might reasonably ask what sorts of evidence we can use to build these models and persuade people they are true. There is also hearty debate in the literature concerning what constitutes an “actual cause” and algorithms for determining this are fiddly to apply, even if it is assumed we have a true model of the causal relationships. I explored Halpern’s approach in a previous blog post.
The original definition of theory-based evaluation by Fitz-Gibbon and Morris (1975) included the full range of approaches, qualitative and quantitative – including RCTs and QEDs. Many others follow in the tradition. For instance, Weiss (1997, p. 512) cites path analysis (a kind of structural equation model) as being “conceptually compatible with TBE [theory-based evaluation] and has been used by evaluators”. Chen and Rossi (1980) explain how theory-based (what they term theory-driven) RCTs that include well-chosen covariates (competing exposures) yield more precise estimates of effects (reducing the probability of Type II error), even though those covariates are not needed to control Type I error. Counterfactual queries do not need a comparison group. They do need a model of how facts came about that can be modified to predict the counterfactual outcome.
Challenging the theory-based/counterfactual binary does not mean that all evaluations are the same. There can still be variation in the strength of evidence used to develop and test theories and how well theories withstand those tests. However, taking a more nuanced view of the differences and similarities between approaches leads to better evaluations.
Adams, E. W. (1970). Subjunctive and Indicative Conditionals. Foundations of Language, 6, 89–94.
Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302
Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.
Halpern, J. Y. (2016). Actual causality. The MIT press.
HM Treasury. (2020). Magenta Book.
Mayne, J. (2019). Revisiting contribution analysis. Canadian Journal of Program Evaluation, 34(2), 171–191.
Rafetseder, E., Cristi‐Vargas, R., & Perner, J. (2010). Counterfactual reasoning: Developing a sense of “nearest possible world”. Child Development, 81(1), 376-389.
Reichardt, C. S. (2022). The Counterfactual Definition of a Program Effect. American Journal of Evaluation, 43(2), 158–174.
Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501–524.
White, H. (2010). A contribution to current debates in impact evaluation. Evaluation, 16(2), 153–164.
“The preceding paragraph illustrates a procedure that statisticians of all schools find important but elusive. It has been called the interocular traumatic test; you know what the data mean when the conclusion hits you between the eyes. The interocular traumatic test is simple, commands general agreement, and is often applicable; well-conducted experiments often come out that way. But the enthusiast’s interocular trauma may be the skeptic’s random error. A little arithmetic to verify the extent of the trauma can yield great peace of mind for little cost.” (Edwards, Lindman, & Savage, 1963, p.217, who attribute the “test” to J. Berkson, personal comm., 14 July 1958)
Evaluation purists: It’s only a Logic Model if it comes from the W. K. Kellogg Foundation region of Michigan, otherwise it’s just sparkling boxes and arrows.
Alternatively:
“A program theory is an explicit theory or model of how an intervention contributes to a set of specific outcomes through a series of intermediate results. The theory needs to include an explanation of how the program’s activities contribute to the results, not simply a list of activities followed by the results, with no explanation of how these are linked, apart from a mysterious arrow.” (Funnell & Rogers, 2011, p. 31)
“A program theory is usually displayed in a diagram called a logic model.” (Funnell & Rogers, 2011, p. 32)
“… sometimes the terms logic model and theory of change have been distinguished in particular ways that are different to the ways we are using the terms here. For example Heléne Clark, from ActKnowledge, and Andrea Anderson, from the Aspen Institute Roundtable on Community Change, who employ program theory extensively, have used these terms to make the distinction between ways of representing program theory that we have labeled pipeline logic models and outcomes chain logic models (Clark and Anderson, 2004). Mhairi Mackenzie and Avril Blamey (2005) have used theories of change to refer specifically to the type of logic model advocated by the Aspen Institute […]. The key message here is to define your terms carefully and ask others to do so as well. It cannot be assumed that you mean the same thing when you use the same term or that you mean something different when you use a different term.” (Funnell & Rogers, 2011, p. 26)
“Despite occasional statements that program theory is a new approach, its roots go back more than fifty years. […] The history of program theory evaluation is not one of a steady increase in understanding. Instead, many of the key ideas have been well articulated and then ignored or forgotten in descriptions of the approach. It is not unusual to have statements that demonstrate a lack of knowledge of previous empirical and theoretical developments, such as a call for proposals from the Agency for Healthcare Research and Quality (2008) that claimed that “‘theory-based evaluation’ is a relatively new approach” (p. 14).”
Be wary of evaluations of AI classifiers that focus only on predictive accuracy as a single percentage. Instead, as a bare minimum, we need to separate quantification of how well AI is doing when it says yes and when it says no.
Predictive accuracy is strongly determined by how common the true yes and no answers are. If a system is designed to try to diagnose a rare disease, the easiest way to increase accuracy is always to predict no, since that prediction will be correct for most cases.
Instead, we need to know how frequently is AI missing diseases that have a treatment and how often it is over-diagnosing and leading to potentially harmful overtreatment.
Another application of AI that we have all been using for decades is spam detection. In this domain, we need to know how often a spam filter is exposing us to cryptocurrency grifters (for example) and how often it is quietly deleting important emails.
AI systems allow a threshold to be tweaked that pushes them towards under- or over-diagnosis. How you choose the threshold depends on what action you will take as a result of the classification, e.g., will users just be exposed to a little extra spam that they can swiftly delete or will they be subjected to unnecessary chemotherapy.
“Linear mixed models for complex survey data, by pairwise composite likelihood, as described in Lumley & Huang (2023). Supports nested and crossed random effects, and correlated random effects as in genetic models. Allows for multistage sampling and for other designs where pairwise sampling probabilities are specified or can be calculated.”
Sophie Webber and Carolyn Prouse (2018, p. 169, footnote 1) write:
“Randomistas is a slang term used by critics to describe proponents of the RCT methodology. It is almost certainly a gendered, derogatory term intended to flippantly dismiss experimental economists and their success, particularly Esther Duflo, one of the most successful experts on randomization.” [Emphasis original.]
Interesting meta-analysis by Furukawa et al. (in press) of 241 trials, aiming to work out what components of therapy lead to better outcomes for people with chronic insomnia.
Furukawa Y., Sakata M., Yamamoto R., et al. Components and Delivery Formats of Cognitive Behavioral Therapy for Chronic Insomnia in Adults: A Systematic Review and Component Network Meta-Analysis. JAMA Psychiatry. Published online January 17, 2024.
“This article presents a framework for addressing external validity bias, including a synthesis of approaches for generalizability and transportability, and the assumptions they require, as well as tests for the heterogeneity of treatment effects and differences between study and target populations.”