theory-based/driven evaluation – Andi Fugard (∧⇒)

Deepening the theories of mechanisms used in policy evaluation

Theory‑based (or driven) evaluations begin with a theory of change and then design the evaluation to test the causal mechanisms it proposes (Fitz‑Gibbon & Morris, 1975). A well‑constructed theory of change sets out how a programme’s resources are deployed to support delivery activities, and how these activities activate the causal mechanisms that generate outcomes as they evolve over time. The early – and, to my mind, more coherent – formulations of theory‑based evaluation were pluralistic, encompassing all methodological approaches (see, e.g., Chen, 2015): RCTs and quasi‑experiments alongside, for example, process tracing and qualitative comparative analysis.

Mechanisms can be defined in terms of entities and what they do to bring about change (Illari & Williamson, 2011). We can identify those entities and activities with the aid of substantive theories relevant to the policy area under investigation, and the mechanisms are described using the concepts provided by those theories (Ioannidis & Psillos, 2018). A substantive theory is particularly helpful when there is evidence for the mechanisms it proposes, rather than when it rests only on armchair speculation. However, all theories are necessarily incomplete and evolve as new evidence accumulates. Some theories are more detailed than others. Some have been more thoroughly tested than others.

Substantive theories describe mechanism at different levels of explanation. Sun et al. (2005) describe the levels as follows:

Object of analysis	Type of analysis	Elements in model
Inter-agent	Social/cultural	Collections of agents
Agents	Psychological	Individual agents
Intra-agent	Componential [I think also psychological]	Modular construction of agents
Substrates	Physiological	Biological realisation of modules

For an inter‑agent analysis, the focus is on how collections of agents (often people) interact with one another. Examples include models of crowd behaviour when a fire alarm sounds, or systemic processes that shape people’s experiences, such as racism, sexism, transphobia, and their intersections.

For an agent‑level analysis, the focus shifts to individual people. This is roughly the way we talk about individuals in everyday life, including their desires, beliefs, opportunities, and actions, for instance, spending money, talking, listening, going for a run, attending a mentoring session, or doing homework.

An intra‑agent analysis draws on concepts from psychology. This may include theories of cognitive control; for instance, why people sometimes respond automatically and at other times engage in deliberate thought, and how conflicts between competing cognitive systems are resolved. It may include theories of emotion, such as how people’s goals and their (often automatic) inferences about progress toward those goals give rise to feelings of happiness, sadness, fear, or anger. Theories of memory systems also sit here, including how information is temporarily represented in visuospatial and phonological working memory, and the capacity and processing limits of these systems.

Finally, analysis at the substrate level considers the biological processes that implement psychological‑level mechanisms or reflect the consequences of behavioural change. This is most visible in dietary interventions, where targets might include blood glucose or cholesterol levels. It can also include theories from cognitive neuroscience, such as which neural systems underpin cognitive control or memory.

I revisited the theories of change in two evaluations I worked on to see which elements from these four levels of analysis were present.

Case study 1: Basic Maths Premium

Basic Maths Premium was a Department for Education pilot that provided additional funding to post‑16 providers in disadvantaged areas to improve GCSE maths resit outcomes for students with prior attainment at grade 3 or below. It tested three funding models using an RCT: two with guaranteed elements and one based entirely on payment by results (PbR):

Model	Unconditional funding	Conditional funding
A	£500 × number of eligible students
B	£250 × number of eligible students	£250 × number of successful students
C		£500 × number successful students

The theory of change was at the agent level. For unconditional funding, the theory was straightforward: funding would enable activities expected to improve outcomes, such as more teaching hours, smaller class sizes, and greater use of technology. By contrast, the theory underpinning conditional funding was less unconvincing. Financial incentives were assumed to boost staff motivation, which would in turn enhance teaching quality, student motivation, and ultimately learning outcomes. However, for Model C in particular, it was uncertain how providers were expected to finance any additional activities, given that all funding under this model depended on student pass rates after any teaching had ended. Two quotations from heads of maths, drawn from the implementation and process evaluation (IPE), captured the issue succinctly (Scott et al., 2024):

“The fact that we’ve got this potential funding in the future that will reward us for that, that’s great, but we’ve still got to find the funds now to do what we do” (Model C – conditional £500 per passing student).
“I think [payment by results is] really unfair because you would not really know how much money you were going to get. I don’t think we would have been able to spend any additional money on that basis, so for us it wouldn’t have really worked. […] I would never have been able to employ two staff on the basis that I might get a certain amount of students through a GCSE. That would have been too much of a financial risk” (Model A – unconditional £500 per student). Also a good example of counterfactual reasoning without a comparison group.

The IPE showed that actual spend was primarily driven by the unconditional funding:

Case study 2: Stop and Think

Stop and Think is a computer‑based intervention designed to help primary pupils overcome common misconceptions in maths and science. It focuses on strengthening children’s inhibitory control: the ability to pause, reflect, and override intuitive but incorrect responses, by guiding them through short, game‑like activities that present counter-intuitive problems.

The theory underlying the programme included intra-agent and substrate levels of analysis. Briefly, the idea is that when students learn new concepts, they need to overcome intuitively obvious prior beliefs. Mareschal (2016) summarises evidence of the cognitive mechanisms involved when intuitions and new learning clash with each other, e.g., the inhibition of pre-existing beliefs involves processes implemented in the dorsal lateral prefrontal cortex (DLPFC) and the anterior cingulate cortex (ACC). Picture below (Mareschal, 2016, p. 115):

One of the tests of the theory in the RCT evaluating Stop and Think (Takala, et al., 2025) involved the construction of a measure of misconceptions driven by prior beliefs. The impact of the programme on misconceptions was then tested quantitatively using mediation analysis.

Importantly, this variable-based analysis is not the mechanism. Instead, the misconceptions measure was an indirect operationalisation of a trace of the underlying mechanism illustrated in the picture above. This distinction is summarised by Paley and Lilford (2011, pp. 956-7):

“The view […] that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.”

Conclusions

Being explicit about the entities involved, the activities they carry out, the substantive theories that link these activities to change, and the levels of analysis being used has potential to improve the quality of theories of change. In the lead‑up to developing a new theory, it can be valuable to revisit earlier theories and identify their weaknesses, with the aim of learning from past mistakes and doing better.

A persistent challenge, however, is that evaluators are often brought into the process too late, when the theory is already fixed or the programme is underway. Feasibility and pilot studies, with an emphasis on implementation and process evaluation, offer a crucial opportunity to deepen theories of change, fail fast, and adjust course before public funding is committed to programmes with unconvincing rationales.

References

Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd edition). Sage Publications.

Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.

Illari, P. M., & Williamson, J. (2011). What is a mechanism? Thinking about mechanisms across the sciences. European Journal for Philosophy of Science, 2(1), 119–135.

Ioannidis, S., & Psillos, S. (2018). Mechanisms in practice: A methodological approach. Journal of Evaluation in Clinical Practice, 24(5), 1177–1183.

Mareschal, D. (2016). The neuroscience of conceptual learning in science and mathematics. Current Opinion in Behavioral Sciences, 10, 114–118.

Scott, M., Scandone, B., Griggs, J., Roberts, E., Bristow, T., Woolfe, E., Dey, M., & Fugard, A. (2024). Basic Maths Premium evaluation report. Education Endowment Foundation.

Sun, R., Coward, L. A., & Zenzen, M. J. (2005). On levels of cognitive modeling. Philosophical Psychology, 18, 613–637.

Takala, H., Kuo, T.-L., Duysak, E., Bhatti, S., Stoilova, E., Fletcher, A., McGuinness, N., McKaskill, M., & Fugard A. (2025). Stop and Think: Learning Counterintuitive Concepts Evaluation Report. Education Endowment Foundation.

Master’s in Public Policy (MPP) talk at Cambridge

This week I gave a talk to the MPP class. Slides here. I covered the usual things, e.g., how all policy evaluation is theory-based and counterfactuals don’t need a comparison group. A few new additions:

I claimed that Realistic Evaluation, i.e., the (Pawson & Tilley, 1997) book, is an intro to scientific methods and philosophy of science, rather than introducing a distinct form of evaluation. I think this is clearest to see in Pawson (2024), and it’s one of the most muddled sections of the Magenta Book.
The importance of theory for interpreting the meaning of covariates as well as merely selecting them. This is particularly important for intersectional understandings of selection into programmes, e.g., you need a theory to work out why, e.g., some combinations of gender and ethnicity, are more likely to be referred to a targetted programme. Is it driven by gender and/or racial stereotypes at time of selection, or broader societal discrimination before selection, for example?
An example from the Green Book (para 4.17) of a type of counterfactual reasoning, used when appraising different ways to realise a policy. I think the reasoning behind “deadweight” and “additionality” outcomes may use future less vivid conditionals. Still reading.

Theorising what evidence would test a theory

Policy evaluation involves speculating about what might be going on – a creative process – based on what we currently (think we) know, and then working out what evidence we would expect to see if we were correct or if plausible alternative theories were true. That logic is explicit in process tracing, but it is far more general than that. The oft‑cited Van Evera (1997) rules, such as the “smoking gun”, follow directly from Bayes’ rule. We cannot directly observe a programme’s mechanisms of change; many different theories will be compatible with both qualitative and quantitative evidence. That problem has a name, underdetermination, and it is a challenge across the sciences.

An interesting example of the gap between theory and evidence is how physicists detect gravitational waves produced by black holes merging. The Laser Interferometer Gravitational-wave Observatory (LIGO) does not observe the waves themselves; it measures how those waves distort spacetime by a miniscule amount, far smaller than a proton.

My favourite analogy comes from trying to crack encrypted texts. We can systematically analyse letter and word frequencies in ciphertexts to spot patterns, but that only takes us so far (try the exercises in Rubinstein‑Salzedo, 2018). It helps to guess what people might be trying to say to each other based on something beyond the ciphertext; for example, that they open with “How are you?” or that they are likely to be discussing a particular event. These knowledge‑grounded guesses, drawing on a wealth of experience and prior evidence, help reduce the search space of possible encryptions. Something similar is going on in evaluation when we iterate between programme theories, tests of those theories, and exploratory research to generate ideas for new theories.

References

Rubinstein‑Salzedo, S. (2018). Cryptography. Springer.

Van Evera, S. (1997). Guide to methods for students of political science. Cornell University Press.

Seven persistent myths about evaluation

Social policy evaluation inherits concepts from multiple disciplines, and somewhere along the way many of them have become muddled. The list below sketches what I think are seven of the most harmful myths.

1. “Counterfactual” is a synonym for control/comparison group

There is a long history of research on counterfactual reasoning in the absence of a control group. For example, analyses of how people should and can determine the truth of a counterfactual conditional such as “If Oswald had not killed Kennedy, no one else would have” or “If I’d left the house earlier, I’d have caught that bus” (e.g., Adams, 1970; Halpern, 2015). We ponder counterfactuals like these all the time without running RCTs.

The evaluation literature has also considered how counterfactuals can be used without a comparison group (e.g., White, 2010) – including with qualitative evidence (e.g., Reichardt, 2022). In evaluation, the counterfactuals are conditionals like, “If intervention group students hadn’t been offered the mentoring programme, x% fewer would have passed their GCSEs.”

Some evidence for counterfactual outcomes is more robust than others. One example analysed by Reichardt (2022) involves asking programme participants what they think would have happened if they had not taken part in the programme. Their ability to answer this depends on their capacity for counterfactual reasoning, which may be limited for some programmes. For instance, many people believe that homeopathy is effective, even though its remedies are so highly diluted that they contain no active ingredients and cannot have any biological effect (Ernst, 2005). Counterfactual reasoning based on such beliefs could therefore falsely conclude that a programme caused improvements that would have occurred anyway.

Quantitative designs can be used to estimate counterfactual outcomes without a comparison group. One example is interrupted time series, which uses trends in pre-intervention outcomes to estimate post-intervention counterfactual outcomes. This is illustrated below:

The standard analysis estimates whether introducing the intervention shifted the outcome and whether it changed the slope, both relative to the inferred counterfactual outcomes.

When a comparison group is used to estimate counterfactual outcomes, it is a factual group – it consists of real data from real people. If it were a counterfactual group, it couldn’t be used to estimate anything. This is why I avoid the term “counterfactual group”.

2. RCTs only estimate overall average effects

If you think causal effects may vary by, e.g., participant characteristics or the context they are in, then you can test this by using moderator analysis and obtain estimates for each group. For example, trials funded by the Education Endowment Foundation usually investigate whether the magnitude of a programme’s effects differ for students who are eligible for free school meals – a proxy for socioeconomic disadvantage (e.g., Takala, et al., 2025). Westhorp and Feeny (2024) suggest using moderator analysis to test the effect of context in realist evaluations (see also Myth 4 below).

One challenge is that when testing for moderator effects for variables with a large number of levels – for example, in intersectional analyses of gender and ethnicity – models can become unwieldy very quickly. The UK census, for example, includes 19 ethnicities and 6 genders (including cis and trans identities). That results in 114 possible combinations. A promising solution is multilevel analysis of individual heterogeneity and discriminatory accuracy – MAIHDA, for short (Merlo, 2018).

3. (Trialists believe that) RCTs don’t need a programme theory

Individual causal effects are defined as the difference between what a person’s outcome would be following intervention and what it would be following control. We can’t observe this within‑person difference because each participant only experiences one condition. With random assignment, however, the average outcome of those in the intervention group minus the average outcome of those in the control group (a between‑person difference) provides an unbiased estimate of the average of individual causal effects (averages of the unmeasureable within-person differences) – even without any covariates. That’s the great thing about RCTs.

RCTs still need theory. How do you choose a control group? Outcome variables? How do you interpret estimated causal effects, i.e., what the difference between intervention and control outcomes means? Chen and Rossi (1980) explain how theory-based (what they call theory-driven) RCTs that include theory-informed covariates yield more precise estimates of effects (reducing the probability of Type II error), even though those covariates are not needed to control Type I error. That’s why sample size calculators ask how much variance in outcomes is thought to be explained by covariates. RCTs can also test mechanisms of change, e.g., using mediation analysis or through factorial designs.

A related myth is that trialists believe reality consists of variables. Paley and Lilford (2011, pp. 956-7) dispute this:

The view attributed to positivism, that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.

Good theories of change go beyond the variables to discuss people and other entities and what they do (see Myth 6). Variables operationalise aspects of these.

Incidentally, trialists don’t need to be positivists either. Bogen and Woodward (1988) provide an accessible discussion of how to think about the relationship between theory and observed phenomena without falling into positivism.

4. Realist evaluation is a distinct type of evaluation

What people do in practice when they say they are conducting a realist evaluation or “using realist principles” often makes sense, and Pawson and Tilley’s work has helped bring theory to the fore in evaluation (I put at least one reading of theirs on my social theorising course when I was an academic). But the idea that you need a realist evaluation to find out “what works best for whom in what context” or to describe change in terms of context, mechanisms, and outcomes, is just absurd. Many evaluation approaches do this.

Paul (1967, p. 111) offers an earlier example of the what works for whom logic:

[…] the question towards which all outcome research should ultimately be directed is the following: What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances?

Pawson and Tilley (1997, p. 10) cite Palmer (1975, p. 150), who uses similar logic:

Rather than ask, “What works for offenders as a whole?” we must increasingly ask “Which methods work best for which types of offenders, and under what conditions or in what types of setting ?”

School climate research offers one example of contextual factors: the physical characteristics of schools; the formal and informal rules that operate within them; and the norms, beliefs, and values at play (Anderson, 1982). RCTs have examined the extent to which school climate moderates programme effects – i.e., how outcomes depend on the settings in which programmes are delivered (e.g., Low & Van Ryzin, 2014) – as well as the mediating role of climate, where a programme changes aspects of school climate and these changes help to explain its effects (e.g., Singla et al., 2021).

My reading of Pawson and Tilley (1997) and Pawson (2024) is that they’re introducing the scientific method and philosophy of science to evaluators who don’t have a formal training in a science. The ideas apply to any form of evaluation. For example, Pawson (2024, p. 42) argues that

All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.

Science is not simply about observing regularities or estimating average causal effects. It is about explaining why observed patterns occur, how they are produced, and under what conditions they hold. This logic applies whether you are running an RCT, conducting a qualitative impact evaluation, or analysing administrative data using a quasi-experiment – and whether or not you call what you’re doing “realist evaluation”.

The “realist” in the term has a meaning from philosophy: ontological realism. Many people are realists without carrying a realist evaluator card.

5. “Theory-based evaluation” excludes trials and quasi-experiments

The original conceptualisations of theory-based evaluation, or its many synonyms (e.g., theory-driven evaluation, program theory evaluation, program theory–driven evaluation science) includes any approach that begins with a theory of change, uses it to design an evaluation that tests the theory, and revises the theory in light of findings (see, e.g., Chen & Rossi, 1983; Cook, 2000; Fitz-Gibbon & Morris, 1975; Weiss, 1972).

Quasi‑experiments are particularly dependent on programme theory. For example, studies that construct a comparison group using matching or weighting rely on identifying all important covariates. Once those covariates are specified, we can check whether matching/weighting has made the intervention and comparison groups equivalent at baseline; however, balance checks cannot tell us whether any important covariates were left out – theory is needed for that.

Additionally, a large number of causal models will be consistent with the data, so you need a theory to choose between them. For example, given only data, the following six simple causal models are statistically equivalent. Each model has three variables: outcome (out); condition, e.g., the programme being evaluated or usual practice (prog); and mediator (med).

Model 1 is the intended causal interpretation: the effect of the programme is partially explained by the mediator. Model 2, for example, says that programme and outcome are statistically associated with each other because the supposed mediator is actually a common cause.

The Magenta Book recommends using theory-based evaluation if you can’t find a comparison group (HM Treasury, 2020, p. 47). Annex A (analytical methods for use within an evaluation) locates trials and quasi-experiments outside the theory-based list. I think the Magenta Book is partly responsible for perpetuating Myth 5 in the UK – particularly through invitations to tender that follow the book, which leave evaluators having to play a semantic game to win work. I hope the upcoming revision corrects this (update: it doesn’t), and that it becomes routine practice to commission theory-based trials, quasi-experiments, and mixed methods evaluations, alongside theory-based qualitative impact evaluations and small-n case studies.

6. Theories of change are a kind of diagram

Theories of change are theories – of change. They may be illustrated using a diagram, but there needs to be an explanation, usually using prose, describing how the resources, entities (e.g., people, organisations, laws, etc.), and activities carried out lead to outputs and outcomes. Currently entrenched practices of developing a Theory of Change™ seem to lead people to forget what they learned about theory and theorising at university.

Pawson and Tilley (1997, p. 134) provide neat examples of how to organise theories of change explanations in tables. Tables like these are part of the “realist evaluation” tradition; however, the logic is clearly much more general (see Myth 4):

Context	Mechanism	Outcome
High numbers of prepayment meters, with a high proportion of burglaries involving cash from meters	Removal of cash meters reduces incentive to burgle by decreasing actual or perceived rewards	Reduction in percentage of burglaries involving meter breakage; reduced risk of burglary at dwellings where meters are removed; reduced burglary rate overall

If a client forces you to include only a diagram in a report, plead with them to let you add the actual theory to an Appendix. We’ve been in this situation many times and used this solution once to date – let’s see if the report is published (update: it wasn’t).

7. Evaluations test “treatments”

Well, in a formally defined sense they do; however, not as people usually use the term. We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, emotional support. The term “treatment” carries with it medical baggage, which is very different to social programmes. My preference would be to use names that are as close as possible to the conditions. For instance, if we were evaluating ACME Therapy, the conditions would be something like ACME Therapy and usual practice or ACME Therapy and CBT.

The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.

Dosage is typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. People aren’t absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.

Using medical language in the context of trials in schools or criminal justice sounds especially creepy – it reminds me of A Clockwork Orange. I have been guilty of using these terms too. Sometimes it’s just too difficult to challenge medicalised tradition when there are pressing deadlines, and causal estimands like average treatment effect on the treated are pervasive in the methods literature (e.g., Cunningham, 2021, Section 4.1.2).

Revised 2 January 2026

References

Adams, E. W. (1970). Subjunctive and Indicative Conditionals. Foundations of Language, 6, 89–94.

Anderson, C. S. (1982). The Search for School Climate: A Review of the Research. Review of Educational Research, 52, 368–420.

Bogen, J., & Woodward, J. (1988). Saving the phenomena. The Philosophical Review, XCVII(3), 303–352.

Chen, H.-T., & Rossi, P. H. (1980). The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science. Social Forces, 59, 106–122.

Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302.

Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In A. Petrosino, P. J. Rogers, T. A. Huebner, & T. A. Hacsi (Eds.), New directions in evaluation: Program Theory in Evaluation: Challenges and Opportunities (pp. 27–34). Jossey-Bass.

Cunningham, S. (2021). Causal Inference: The Mixtape. Yale University Press.

Ernst, E. (2005). Is homeopathy a clinically valuable approach? Trends in Pharmacological Sciences, 26(11), 547–548.

Halpern, J. Y. (2015). A Modification of the Halpern-Pearl Definition of Causality. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015), 3022–3033.

HM Treasury. (2020). Magenta Book.

Low, S., & Van Ryzin, M. (2014). The moderating effects of school climate on bullying prevention efforts. School Psychology Quarterly, 29(3), 306–319.

Merlo, J. (2018). Multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) within an intersectional framework. Social Science & Medicine, 203, 74–80.

Paley, J., & Lilford, R. (2011). Qualitative methods: An alternative view. BMJ, 342, 956–958.

Palmer, T. (1975). Martinson Revisited. Journal of Research in Crime and Delinquency, 12(2), 133–152.

Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31(2), 109–118.

Pawson, R., & Tilley, N. (1997). Realistic Evaluation. SAGE Publications Ltd.

Pawson, R. (2024). How to Think Like a realist: A methodology for social science. Edward Elgar Publishing Limited.

Reichardt, C. S. (2022). The Counterfactual Definition of a Program Effect. American Journal of Evaluation, 43(2), 158–174.

Singla, D. R., Shinde, S., Patton, G., & Patel, V. (2021). The Mediating Effect of School Climate on Adolescent Mental Health: Findings From a Randomized Controlled Trial of a School-Wide Intervention. Journal of Adolescent Health, 69(1), 90–99.

Takala, H., Kuo, T.-L., Duysak, E., Bhatti, S., Stoilova, E., Fletcher, A., McGuinness, N., McKaskill, M., & Fugard, A. (2025). Stop and Think: Learning Counterintuitive Concepts Evaluation Report. Education Endowment Foundation.

Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.

Westhorp, G., & Feeny, S. (2024). Using surveys in realist evaluation. Evaluation Journal of Australasia, 1035719X241292083.

White, H. (2010). A contribution to current debates in impact evaluation. Evaluation, 16(2), 153–164.

Process tracing example (Hauter, 2023)

One of the clearest examples of process tracing I have seen is in political science. Jakob Hauter (2023) analyses open source evidence concerning the 2014 occupation of Sloviansk and Kramatorsk in Donbas, to test the extent to which State actors were involved.

Hauter, J. (2023). Forensic conflict studies: Making sense of war in the social media age. Media, War & Conflict, 16(2), 153-172.

Theory-based evaluation – a possible Venn

Theory-based evaluation – a possible Venn, with a long and often-forgotten history.

Two diagrams. The first, shows common usage of the terms "counterfactual impact evaluation" and "theory-based impact evaluation". They are two rectangles side-by-side, separated, with a space between.

The second more obviously Venn diagram, titled a possible alternative, depicts three kinds of evaluation that are all included within theory-based evaluation. These are quantitative impact evaluation, qualitative impact evaluation, and, in the overlapping region between then, mixed methods impact evaluation.

Example readings

Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302

Reichardt, C. S. (2022). The Counterfactual Definition of a Program Effect. American Journal of Evaluation, 43(2), 158–174.

Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.

Ordinary people making sense of their social experience

“Let’s pause for a moment and ask a very basic and very important question: what is theory? We start by answering in the negative: the production of theory is not necessarily a scholastic exercise; the site of production of theory is not necessarily the ivory towers of academia; the producers of theory are not necessarily academically trained personnel in possession of qualifications that bestow on them the status of ‘officially approved theorists’.

“Positively, we would argue that the building blocks of the production of theory are the efforts of ordinary people to make sense of their social experience; the sites of production of theory are, then, everywhere such activity may take place; the producers of theory are – potentially – everyone who engages in reflection on their experiences so as to develop new and improved practices to problematic aspects of that experience.”

Cox, L. and Nilsen, A. G. (2005). Why do activists need theory? Euromovements Newsletter

Modelling theories of change – a look at Weiss (1972)

Periodically I listen to contemporary debates in evaluation and wonder how much progress there has been in the past fifty years. Here’s a picture from Carol Weiss (1972, Figure 3–3, p. 50; reprinted in 1996):

Weiss called it a model of a programme; now we would use the term programme theory or theory of change. About half of the next page (Weiss, 1972, p. 51) is filled with intriguing footnotes. The first notes the similarity between the diagram above and a statistical approach:

“As the similarity between Figure 3–3 and a path diagram suggests, path analysis is a useful way of estimating the strength of the linkages. When the necessary conditions for each path analysis are met, path coefficients will represent the magnitude of effects for each arrow.”

On path analysis, Weiss cites Blalock’s (1969) Theory construction from verbal to mathematical formulations. Structural equation modelling, path analysis plus confirmatory factor analysis, turned out to be a popular way to test theories in social science (see, e.g., MacCallum & Austin, 2000; Zyphur, Bonner, & Tay, 2023).

The same footnote also summarises the Automatic Interaction Detector (AID), an early example of a data-driven partitioning procedure:

“Another analytic device, particularly useful when one step in the chain is totally or largely conditional on the one before, is the Automatic Interaction Detector (AID). Unlike path analysis, this type of ‘branch’ analysis shows whether the program is producing strong effects for subsamples of the group. This is a matter of both theoretical and programmatic interest, because if there are differential effects, special programs can be developed to fit special groups.”

AID and its successors are data hungry and liable to overfitting data if misused; however, there is a rich literature on how to use them and stay out of trouble. Loh (2014) reviews fifty years of that literature, beginning with AID (p. 330):

“Despite their novelty, or perhaps owing to it, AID and THAID did not initially attract much interest in the statistics community. Einhorn (1972) showed by simulation that AID can severely overfit the data. Doyle (1973) pointed out that if two or more variables are highly correlated, at most one may appear in the tree structure. This problem of masking can lead to spurious conclusions about the relative importance of the variables. Bishop et al. (1975) criticized AID for ignoring the inherent sampling variability of the data. Around the same time though, the idea of recursive partitioning was gaining steam in the computer science and engineering communities as more efficient algorithms for carrying out the search for splits began to appear (Chou 1969; Henrichon & Fu, 1973; Meisel & Michalopoulos, 1977; Payne & Meisel, 1977; Sethi & Chatterjee, 1991).”

(Example newer algorithms are implemented in {rpart} in R.) Weiss’s footnote and these statistical approaches are clear examples of trying to tackle the “what works best for whom” problem, which for some reason (along with “context”) is now associated, like a Pavlovian response, with “realist” evaluation. We need to do better.

References

Blalock, H. M., Jr. (1969). Theory construction from verbal to mathematical formulations. Prentice-Hall Inc.

Loh, W. (2014). Fifty Years of Classification and Regression Trees. International Statistical Review, 82(3), 329–348.

MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.

Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.

Weiss, C. H. (1996). Excerpts from evaluation research: Methods of assessing program effectiveness. Evaluation Practice, 17(2), 173–175.

Zyphur, M. J., Bonner, C. V., & Tay, L. (2023). Structural equation modeling in organizational research: The state of our science and some proposals for its future. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 495-517.

A review of process tracing

I was interested to see a recent review of process tracing in health research (Johnson, Beach, & Al-Janabi, 2024). How are its users getting on…?

Of the 19 evaluations satisfying review criteria, only 2 got full marks:

Alvarado, M., Penney, T. L., Unwin, N., Murphy, M. M., & Adams, J. (2021). Evidence of a health risk ‘signalling effect’ following the introduction of a sugar-sweetened beverage tax. Food Policy, 102.
Ryu, S.-K., & Chung, S.-G. (2021). Korea’s Early COVID-19 Response: Findings and Implications. International Journal of Environmental Research and Public Health, 18(16), 8316.

These were praised for how they justified using process tracing, their complete and detailed description of methods, their use of multiple types of evidence, and how they considered and attempted to rule out alternative or counterfactual explanations of outcomes. Finally, they linked their findings back to the theory. I have added them to my reading stack.

The review’s conclusions are damning:

“… a majority of studies reported do not describe in-depth how they conduct their process trace, and fewer still support their claims consistently and comprehensively with verifiable and well organised evidence. This is somewhat counterintuitive given that one of the features of process tracing is to support the more robust depth of analysis to explore hypotheses and confirm or counter with supportive evidence. However, this is not just a feature of studies reporting process tracing in health. Applications of process tracing in policy studies and political science more generally have similar trends, with many studies providing little detail of their methodology or a mechanistic depiction of the theory.”

As ever, to be continued…

References

Alvarado, M., Penney, T. L., Unwin, N., Murphy, M. M., & Adams, J. (2021). Evidence of a health risk ‘signalling effect’ following the introduction of a sugar-sweetened beverage tax. Food Policy, 102.

Johnson, R., Beach, D., & Al-Janabi, H. (2024). How is process tracing applied in health research? A systematic scoping review. Social Science & Medicine, in press, journal pre-proof.

Ryu, S.-K., & Chung, S.-G. (2021). Korea’s Early COVID-19 Response: Findings and Implications. International Journal of Environmental Research and Public Health, 18(16), 8316.

“QCA exists in a nether region between qualitative and quantitative research”

‘Highlighting QCA’s reliance on two-element Boolean algebra and propositional logic brings the weaknesses of QCA into sharp relief. The distinction between QCA and conventional statistics is not a difference between “set-theoretic” methods and what QCA researchers call “non-set-theoretic” methods (Schneider and Wagemann 2012) […]. Conventional statistics are set-theoretic methods. […] What truly sets QCA apart from conventional statistics is the former’s embrace of the limitations of two-element Boolean algebra and propositional logic. QCA exists in a nether region between qualitative and quantitative research; it exhibits neither the depth of small-n qualitative research nor the power of quantitative research.’ (Clarke 2020, p. 553)

Clarke, K. A. (2020). Logical Constraints: The Limitations of QCA in Social Science Research. Political Analysis, 28(4), 552–568.