New systematic review by Quang Vuong et al. (in press) on analyses for generalising or transporting findings from an RCT sample to different samples and/or broader populations than the study sample.
Generalisability and transportability
Tag: policy evaluation
“Minimum alcohol pricing: what we found in Wales after five years”
ETF Evaluation Academy 2.0 resources
Resources from the Evaluation Task Force Evaluation Academy training programme.
Process tracing example (Hauter, 2023)
One of the clearest examples of process tracing I have seen is in political science. Jakob Hauter (2023) analyses open source evidence concerning the 2014 occupation of Sloviansk and Kramatorsk in Donbas, to test the extent to which State actors were involved.
Hauter, J. (2023). Forensic conflict studies: Making sense of war in the social media age. Media, War & Conflict, 16(2), 153-172.
Theory-based evaluation – a possible Venn
Theory-based evaluation – a possible Venn, with a long and often-forgotten history.
Example readings
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In A. Petrosino, P. J. Rogers, T. A. Huebner, & T. A. Hacsi (Eds.), New directions in evaluation: Program Theory in Evaluation: Challenges and Opportunities (pp. 27–34). Jossey-Bass.
Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302
Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.
Reichardt, C. S. (2022). The Counterfactual Definition of a Program Effect. American Journal of Evaluation, 43(2), 158–174.
Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.
Your control group can be sensible
There’s nothing in theories of experiments requiring that researchers use a ridiculous control group. An example of such a ridiculous (and unethical) control would include randomising people who are living in abject poverty either to a regular sufficient social security payment or leaving them in poverty. Hmmm, what group will have a better outcome…?
The first example Rubin gave in his classic “Estimating causal effects of treatments in randomized and nonrandomized studies” (Rubin, 1974, p. 689) had the following two groups:
“… we will restrict discussion to the very simple study consisting of 2N units (e.g., subjects), half having been exposed to an experimental (E) treatment (e.g., a compensatory reading program) and the other half having been exposed to a control (C) treatment (e.g., a regular reading program).”
Lind’s (1753) study of scurvy treatments compared six treatments, including cider, vinegar, and sea water, as well as the oranges and lemons. Potentially unclear what the theoretical rationale was for all of those, so not ideal I’ll admit.
It’s pretty common now to use “business as usual”, “treatment as usual”, “teaching as usual”, etc., controls and to avoid waitlist controls. But this is not a new idea.
Additionally, Rubin didn’t use Y(1) and Y(0) for the two potential outcomes, but Y(E) and Y(C). It’s a very subtle difference; however, I wonder if ditching the zero helps people’s understanding. My preference would be something even closer to the names of the actual programs, e.g., if we are evaluating whether the Acme Compensatory Reading Programme is better than regular reading, Y(Acme) and Y(Regular) to emphasise the contrast. (Also now we would use “participants” rather than “subjects”.)
References
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
Lind, J. (1753). A treatise of the scurvy. In three parts. Containing an inquiry into the nature, causes and cure, of that disease. Together with a critical and chronological view of what has been published on the subject. Edinburgh: Printed by Sands, Murray and Cochran for A. Kincaid and A. Donaldson.
Carefully controlled nonrandomised studies give us subjective randomisation
“Consider a carefully controlled nonrandomized study – a study in which there are no obviously important prior variables that systematically differ in the E trials and the C trials. In such a study, there is a real sense in which a claim of ‘subjective randomization’ can be made. For example, if the study was composed of carefully matched pairs of trials, there might be a very defensible belief that within each matched pair each unit was equally likely to receive E as C in the sense that if you were shown the units without being told which received E, only half the time would you correctly guess which received E.”
Rubin, D. B. (1974, p. 698). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
See also:
“Matching can be thought of as a technique for finding approximately ideal experimental data hidden within an observational data set.”
King, G., & Nielsen, R. (2019, p. 442). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 27(4), 435–454.
What helps/hinders school-based humanistic therapy?
Interesting thematic analysis (Cooper et al., 2015) of what helped and hindered in school-based humanistic therapy, as part of the large ETHOS RCT. The approach the researchers took started with open questions and coding inductively, then concluded with closed questions and deductive coding based on what the theory suggested might help or not. All the usual helpful things were there, like:
“[The therapist] wasn’t, like, doing something else. She was sat there all the time just like listening to me.”
“They just talked to me like I was a human being not like, ‘Oh, you know, what teenagers are like…'”
“It was like they were taking in what I was actually saying and then the questions afterwards were based upon that.”
I was struck by the number of young people who mentioned the therapist providing advice, given that the humanistic approach is supposed to be non-directive (there is some discussion of this). Sometimes this advice was welcome and young people who didn’t receive any advice indicated that they didn’t see the point of therapy without it.
“I used to have an argument with this girl…and the therapist said to just leave it because she’s just going to– she wants attention and she wants a reaction out of me. She [the therapist] was just like, “Leave it. She can say whatever she wants and then she’ll stop it herself,” and now that girl doesn’t even say anything to me because I don’t even say anything to her.”
Advice/guidance wasn’t always experienced as helpful; the examples given seem to be all “technique”:
“[The therapist] would do breathing exercises and imagine things, and I didn’t like that. I said, “I didn’t like that,” and [the therapist] was like, “Well, let’s just try.” “I don’t want to do it.” “Well, let’s just try.” [The therapist] was kind of persistent on something that I didn’t want to do.”
There’s also a collection of things some therapists did which were seen as particularly unhelpful, e.g.,
“She didn’t have a lot to say. She would just sit there and stare at me for sometimes three minutes at a time. Literally it was three minutes. She’d just sit there, and I’d be looking around the room, and every time I looked back at her, she’s just looking at me. She wouldn’t say anything.”
(Reminds me of Levitt’s 2001 thematic analysis of the twenty or so different kinds of silence!)
Wonder if anyone has tried to crowdsource these helpful/hindering ideas at scale.
References
Cooper, M., Smith, S., Sumner, A. L., Eilenberg, J., Childs-Fegredo, J., Kelly, S., Subramanian, P., Holmes, J., Barkham, M., Bower, P., Cromarty, K., Duncan, C., Hughes, S., Pearce, P., Rameswari, T., Ryan, G., Saxon, D., & Stafford, M. R. (2025). Humanistic Therapy for Young People: Client-Perceived Helpful Aspects, Hindering Aspects, and Processes of Change. Journal of Child and Family Studies.
Levitt, H. M. (2001). Sounds of Silence in Psychotherapy: The Categorization of Clients’ Pauses. Psychotherapy Research, 11(3), 295–309.
Ordinary people making sense of their social experience
“Let’s pause for a moment and ask a very basic and very important question: what is theory? We start by answering in the negative: the production of theory is not necessarily a scholastic exercise; the site of production of theory is not necessarily the ivory towers of academia; the producers of theory are not necessarily academically trained personnel in possession of qualifications that bestow on them the status of ‘officially approved theorists’.
“Positively, we would argue that the building blocks of the production of theory are the efforts of ordinary people to make sense of their social experience; the sites of production of theory are, then, everywhere such activity may take place; the producers of theory are – potentially – everyone who engages in reflection on their experiences so as to develop new and improved practices to problematic aspects of that experience.”
Cox, L. and Nilsen, A. G. (2005). Why do activists need theory? Euromovements Newsletter
Modelling theories of change – a look at Weiss (1972)
Periodically I listen to contemporary debates in evaluation and wonder how much progress there has been in the past fifty years. Here’s a picture from Carol Weiss (1972, Figure 3–3, p. 50; reprinted in 1996):
Weiss called it a model of a programme; now we would use the term programme theory or theory of change. About half of the next page (Weiss, 1972, p. 51) is filled with intriguing footnotes. The first notes the similarity between the diagram above and a statistical approach:
“As the similarity between Figure 3–3 and a path diagram suggests, path analysis is a useful way of estimating the strength of the linkages. When the necessary conditions for each path analysis are met, path coefficients will represent the magnitude of effects for each arrow.”
On path analysis, Weiss cites Blalock’s (1969) Theory construction from verbal to mathematical formulations. Structural equation modelling, path analysis plus confirmatory factor analysis, turned out to be a popular way to test theories in social science (see, e.g., MacCallum & Austin, 2000; Zyphur, Bonner, & Tay, 2023).
The same footnote also summarises the Automatic Interaction Detector (AID), an early example of a data-driven partitioning procedure:
“Another analytic device, particularly useful when one step in the chain is totally or largely conditional on the one before, is the Automatic Interaction Detector (AID). Unlike path analysis, this type of ‘branch’ analysis shows whether the program is producing strong effects for subsamples of the group. This is a matter of both theoretical and programmatic interest, because if there are differential effects, special programs can be developed to fit special groups.”
AID and its successors are data hungry and liable to overfitting data if misused; however, there is a rich literature on how to use them and stay out of trouble. Loh (2014) reviews fifty years of that literature, beginning with AID (p. 330):
“Despite their novelty, or perhaps owing to it, AID and THAID did not initially attract much interest in the statistics community. Einhorn (1972) showed by simulation that AID can severely overfit the data. Doyle (1973) pointed out that if two or more variables are highly correlated, at most one may appear in the tree structure. This problem of masking can lead to spurious conclusions about the relative importance of the variables. Bishop et al. (1975) criticized AID for ignoring the inherent sampling variability of the data. Around the same time though, the idea of recursive partitioning was gaining steam in the computer science and engineering communities as more efficient algorithms for carrying out the search for splits began to appear (Chou 1969; Henrichon & Fu, 1973; Meisel & Michalopoulos, 1977; Payne & Meisel, 1977; Sethi & Chatterjee, 1991).”
(Example newer algorithms are implemented in {rpart} in R.) Weiss’s footnote and these statistical approaches are clear examples of trying to tackle the “what works best for whom” problem, which for some reason (along with “context”) is now associated, like a Pavlovian response, with “realist” evaluation. We need to do better.
References
Blalock, H. M., Jr. (1969). Theory construction from verbal to mathematical formulations. Prentice-Hall Inc.
Loh, W. (2014). Fifty Years of Classification and Regression Trees. International Statistical Review, 82(3), 329–348.
MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.
Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.
Weiss, C. H. (1996). Excerpts from evaluation research: Methods of assessing program effectiveness. Evaluation Practice, 17(2), 173–175.
Zyphur, M. J., Bonner, C. V., & Tay, L. (2023). Structural equation modeling in organizational research: The state of our science and some proposals for its future. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 495-517.