An interesting paper on theory-ladenness in research on black holes (Doboszewski & Elder, 2025). Relevant to debates in policy evaluation on theory-based evaluation too and the pervasive role of theory whether or not you name the TBE brand.
‘Philosophers have raised a range of worries about the ways in which observations are theory-laden. In this article, we are focusing on the theory-ladenness of measurement outcomes—the ways in which the “observations” derived from an experiment or measurement are affected by theoretical assumptions that are made in the course of the experiment. The greatest concern is that theory-ladenness may lead to a vicious circularity where the theoretical assumptions made by the experimenters—either in the physical design of the experiment or in subsequent inferences from the empirical data—make it more likely or even guarantee that the observation will confirm the theory being tested. This might occur if the hypothesis being tested is presupposed by the experimental methods being used. Franklin (2015, 156) provides the following toy example of this: if we use a mercury thermometer to test a hypothesis that substances expand as their temperature increases, this is theory-laden in a way that risks vicious circularity because the proper functioning of a mercury thermometer relies on the very hypothesis being tested.’ (pp. 829-830)
‘… the impossibility of controlled experiments on astrophysical targets (stars, galaxies, etc.) places constraints on how astronomers can investigate these systems. Simulations can help make up for the lack of controlled interventions by providing a proxy system into which one can intervene.’ (p. 830)
Doboszewski, J., & Elder, J. (2025). How Theory-Laden Are Observations of Black Holes? Philosophy of Science, 92(4), 827–849.
“Change doesn’t happen in straight lines and is propagated (or not) by human actors with their own preferences and motivations. Delivery is disaggregated between departments, tiers of government, sectors, agencies, regulators and professions. The economy is not a thing but an aggregation of billions of decisions, each made on the basis of incentives, opportunities and desires. Citizens’ choices and relationships control far more of what happens in our country than any Prime Minister could ever dream. Culture eats levers for breakfast.”
Joas and Knöbl (2009, p. 6) on how Popper demolished positivism in 1934:
As Popper lays out in his now very famous book The Logic of Scientific Discovery, which first appeared in 1934, in the case of most scientific problems we cannot be certain whether a generalization, that is, a theory or hypothesis, truly applies in all cases. In all probability, we will never be able to verify once and for all the astrophysical statement that ‘All planets move around their suns along an elliptical trajectory’, because we are unlikely ever to get to know all the solar systems in the universe and therefore we will presumably never be able to confirm with absolute certainty that every single planet does in fact follow an elliptical trajectory around its sun, as opposed to some other route. Much the same applies to the statement ‘All swans are white’. Even if you have seen thousands of swans and all of them were in fact white, you can ultimately never be certain that a black, green, blue, etc. swan will not show up at some point. As a rule, universal statements cannot therefore be confirmed or verified. To put it another way: inductive arguments (that is, inference from individual instances to a totality) are neither logically valid nor truly compelling arguments; induction cannot be justified purely in terms of logic, because we are unable to rule out the possibility that one observation may eventually be made that refutes the general statement thought to be corroborated. Positivists’ attempts to trace laws back to elementary observations or to derive them from elementary observations and verify them are thus doomed to failure. This was precisely Popper’s criticism.
Joas, H., & Knöbl, W. (2009). What is theory? In Social theory: Twenty introductory lectures (pp. 1–19). Cambridge University Press.
Social policy evaluation inherits concepts from multiple disciplines, and somewhere along the way many of them have become muddled. The list below sketches what I think are seven of the most harmful myths.
1. “Counterfactual” is a synonym for control/comparison group
There is a long history of research on counterfactual reasoning in the absence of a control group. For example, analyses of how people should and can determine the truth of a counterfactual conditional such as “If Oswald had not killed Kennedy, no one else would have” or “If I’d left the house earlier, I’d have caught that bus” (e.g., Adams, 1970; Halpern, 2015). We ponder counterfactuals like these all the time without running RCTs.
The evaluation literature has also considered how counterfactuals can be used without a comparison group (e.g., White, 2010) – including with qualitative evidence (e.g., Reichardt, 2022). In evaluation, the counterfactuals are conditionals like, “If intervention group students hadn’t been offered the mentoring programme, x% fewer would have passed their GCSEs.”
Some evidence for counterfactual outcomes is more robust than others. One example analysed by Reichardt (2022) involves asking programme participants what they think would have happened if they had not taken part in the programme. Their ability to answer this depends on their capacity for counterfactual reasoning, which may be limited for some programmes. For instance, many people believe that homeopathy is effective, even though its remedies are so highly diluted that they contain no active ingredients and cannot have any biological effect (Ernst, 2005). Counterfactual reasoning based on such beliefs could therefore falsely conclude that a programme caused improvements that would have occurred anyway.
Quantitative designs can be used to estimate counterfactual outcomes without a comparison group. One example is interrupted time series, which uses trends in pre-intervention outcomes to estimate post-intervention counterfactual outcomes. This is illustrated below:
The standard analysis estimates whether introducing the intervention shifted the outcome and whether it changed the slope, both relative to the inferred counterfactual outcomes.
When a comparison group is used to estimate counterfactual outcomes, it is a factual group – it consists of real data from real people. If it were a counterfactual group, it couldn’t be used to estimate anything. This is why I avoid the term “counterfactual group”.
2. RCTs only estimate overall average effects
If you think causal effects may vary by, e.g., participant characteristics or the context they are in, then you can test this by using moderator analysis and obtain estimates for each group. For example, trials funded by the Education Endowment Foundation usually investigate whether the magnitude of a programme’s effects differ for students who are eligible for free school meals – a proxy for socioeconomic disadvantage (e.g., Takala, et al., 2025). Westhorp and Feeny (2024) suggest using moderator analysis to test the effect of context in realist evaluations (see also Myth 4 below).
One challenge is that when testing for moderator effects for variables with a large number of levels – for example, in intersectional analyses of gender and ethnicity – models can become unwieldy very quickly. The UK census, for example, includes 19 ethnicities and 6 genders (including cis and trans identities). That results in 114 possible combinations. A promising solution is multilevel analysis of individual heterogeneity and discriminatory accuracy – MAIHDA, for short (Merlo, 2018).
3. (Trialists believe that) RCTs don’t need a programme theory
Individual causal effects are defined as the difference between what a person’s outcome would be following intervention and what it would be following control. We can’t observe this within‑person difference because each participant only experiences one condition. With random assignment, however, the average outcome of those in the intervention group minus the average outcome of those in the control group (a between‑person difference) provides an unbiased estimate of the average of individual causal effects (averages of the unmeasureable within-person differences) – even without any covariates. That’s the great thing about RCTs.
RCTs still need theory. How do you choose a control group? Outcome variables? How do you interpret estimated causal effects, i.e., what the difference between intervention and control outcomes means? Chen and Rossi (1980) explain how theory-based (what they call theory-driven) RCTs that include theory-informed covariates yield more precise estimates of effects (reducing the probability of Type II error), even though those covariates are not needed to control Type I error. That’s why sample size calculators ask how much variance in outcomes is thought to be explained by covariates. RCTs can also test mechanisms of change, e.g., using mediation analysis or through factorial designs.
A related myth is that trialists believe reality consists of variables. Paley and Lilford (2011, pp. 956-7) dispute this:
The view attributed to positivism, that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.
Good theories of change go beyond the variables to discuss people and other entities and what they do (see Myth 6). Variables operationalise aspects of these.
Incidentally, trialists don’t need to be positivists either. Bogen and Woodward (1988) provide an accessible discussion of how to think about the relationship between theory and observed phenomena without falling into positivism.
4. Realist evaluation is a distinct type of evaluation
What people do in practice when they say they are conducting a realist evaluation or “using realist principles” often makes sense, and Pawson and Tilley’s work has helped bring theory to the fore in evaluation (I put at least one reading of theirs on my social theorising course when I was an academic). But the idea that you need a realist evaluation to find out “what works best for whom in what context” or to describe change in terms of context, mechanisms, and outcomes, is just absurd. Many evaluation approaches do this.
Paul (1967, p. 111) offers an earlier example of the what works for whom logic:
[…] the question towards which all outcome research should ultimately be directed is the following: What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances?
Pawson and Tilley (1997, p. 10) cite Palmer (1975, p. 150), who uses similar logic:
Rather than ask, “What works for offenders as a whole?” we must increasingly ask “Which methods work best for which types of offenders, and under what conditions or in what types of setting ?”
School climate research offers one example of contextual factors: the physical characteristics of schools; the formal and informal rules that operate within them; and the norms, beliefs, and values at play (Anderson, 1982). RCTs have examined the extent to which school climate moderates programme effects – i.e., how outcomes depend on the settings in which programmes are delivered (e.g., Low & Van Ryzin, 2014) – as well as the mediating role of climate, where a programme changes aspects of school climate and these changes help to explain its effects (e.g., Singla et al., 2021).
My reading of Pawson and Tilley (1997) and Pawson (2024) is that they’re introducing the scientific method and philosophy of science to evaluators who don’t have a formal training in a science. The ideas apply to any form of evaluation. For example, Pawson (2024, p. 42) argues that
All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.
Science is not simply about observing regularities or estimating average causal effects. It is about explaining why observed patterns occur, how they are produced, and under what conditions they hold. This logic applies whether you are running an RCT, conducting a qualitative impact evaluation, or analysing administrative data using a quasi-experiment – and whether or not you call what you’re doing “realist evaluation”.
The “realist” in the term has a meaning from philosophy: ontological realism. Many people are realists without carrying a realist evaluator card.
5. “Theory-based evaluation” excludes trials and quasi-experiments
The original conceptualisations of theory-based evaluation, or its many synonyms (e.g., theory-driven evaluation, program theory evaluation, program theory–driven evaluation science) includes any approach that begins with a theory of change, uses it to design an evaluation that tests the theory, and revises the theory in light of findings (see, e.g., Chen & Rossi, 1983; Cook, 2000; Fitz-Gibbon & Morris, 1975; Weiss, 1972).
Quasi‑experiments are particularly dependent on programme theory. For example, studies that construct a comparison group using matching or weighting rely on identifying all important covariates. Once those covariates are specified, we can check whether matching/weighting has made the intervention and comparison groups equivalent at baseline; however, balance checks cannot tell us whether any important covariates were left out – theory is needed for that.
Additionally, a large number of causal models will be consistent with the data, so you need a theory to choose between them. For example, given only data, the following six simple causal models are statistically equivalent. Each model has three variables: outcome (out); condition, e.g., the programme being evaluated or usual practice (prog); and mediator (med).
Model 1 is the intended causal interpretation: the effect of the programme is partially explained by the mediator. Model 2, for example, says that programme and outcome are statistically associated with each other because the supposed mediator is actually a common cause.
The Magenta Book recommends using theory-based evaluation if you can’t find a comparison group (HM Treasury, 2020, p. 47). Annex A (analytical methods for use within an evaluation) locates trials and quasi-experiments outside the theory-based list. I think the Magenta Book is partly responsible for perpetuating Myth 5 in the UK – particularly through invitations to tender that follow the book, which leave evaluators having to play a semantic game to win work. I hope the upcoming revision corrects this (update: it doesn’t), and that it becomes routine practice to commission theory-based trials, quasi-experiments, and mixed methods evaluations, alongside theory-based qualitative impact evaluations and small-n case studies.
6. Theories of change are a kind of diagram
Theories of change are theories – of change. They may be illustrated using a diagram, but there needs to be an explanation, usually using prose, describing how the resources, entities (e.g., people, organisations, laws, etc.), and activities carried out lead to outputs and outcomes. Currently entrenched practices of developing a Theory of Change™ seem to lead people to forget what they learned about theory and theorising at university.
Pawson and Tilley (1997, p. 134) provide neat examples of how to organise theories of change explanations in tables. Tables like these are part of the “realist evaluation” tradition; however, the logic is clearly much more general (see Myth 4):
Context
Mechanism
Outcome
High numbers of prepayment meters, with a high proportion of burglaries involving cash from meters
Removal of cash meters reduces incentive to burgle by decreasing actual or perceived rewards
Reduction in percentage of burglaries involving meter breakage; reduced risk of burglary at dwellings where meters are removed; reduced burglary rate overall
If a client forces you to include only a diagram in a report, plead with them to let you add the actual theory to an Appendix. We’ve been in this situation many times and used this solution once to date – let’s see if the report is published (update: it wasn’t).
7. Evaluations test “treatments”
Well, in a formally defined sense they do; however, not as people usually use the term. We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, emotional support. The term “treatment” carries with it medical baggage, which is very different to social programmes. My preference would be to use names that are as close as possible to the conditions. For instance, if we were evaluating ACME Therapy, the conditions would be something like ACME Therapy and usual practice or ACME Therapy and CBT.
The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.
Dosage is typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. People aren’t absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.
Using medical language in the context of trials in schools or criminal justice sounds especially creepy – it reminds me of A Clockwork Orange. I have been guilty of using these terms too. Sometimes it’s just too difficult to challenge medicalised tradition when there are pressing deadlines, and causal estimands like average treatment effect on the treated are pervasive in the methods literature (e.g., Cunningham, 2021, Section 4.1.2).
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In A. Petrosino, P. J. Rogers, T. A. Huebner, & T. A. Hacsi (Eds.), New directions in evaluation: Program Theory in Evaluation: Challenges and Opportunities (pp. 27–34). Jossey-Bass.
Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.
Suppose determinism is true. This would mean that every decision we make at any given moment follows inevitably from the laws of physics, biology, and chemistry, etc. (from here on laws of nature for short), as well as the entire chain of prior events stretching back to the beginning of the universe. Many of these laws may be neither known nor knowable. According to a widely accepted definition, free will requires the ability to have chosen otherwise. But determinism appears to rule out any possibility of having made a different choice than the one we actually did.
Philosopher Kadri Vihvelin (2004) offers a thought-provoking analysis of how free will might still be possible under determinism, a flavour of compatibilism. This post is a summary of her argument.
Before we begin, why on earth would we even entertain the idea of determinism? At first glance, the existence of randomness seems to challenge it. However, what appears random may emerge from combinations of an unimaginably vast number of deterministic causes. Pseudorandom number generators, for example, are relatively simple algorithms which produce sequences that appear random but are entirely deterministic. Algorithms are small. The universe is big. “You just won’t believe how vastly, hugely, mind-bogglingly big it is,” said Douglas Adams. The universe could be a really complicated pseudorandom generator.
Quantum mechanics also seems to challenge determinism. When measuring a quantum particle (like a photon) in superposition, the outcome follows a probability distribution (I previously tried to explain this with a toy example). This indeterminacy has led many to interpret quantum mechanics as fundamentally probabilistic. Yet some physicists defend superdeterminism, an apparently controversial idea suggesting that even quantum measurements are predetermined, including the choices physicists make when selecting measurement settings in experiments (Hossenfelder & Palmer, 2020).
That’s just a sketch; however, hopefully enough to suggest that determinism isn’t too absurd to read on.
Choosing means, more specifically, making choices on the basis of reasons. Libet and others have conducted numerous studies on the neuroscience of choosing, which may challenge the role of conscious choice (see Haggard, 2024, for discussion). However, we are more than what we are conscious of. For example, we retrieve memories without knowing how our brains searched for them, and we move through the world without being aware of how our nervous system coordinates muscle movements. I’m happy to accept a choice as mine, even if a bunch of subconscious processes did the choosing, so long as it was made on the basis of reasons – even ones I’m unaware of.
At the heart of Vihvelin’s position is the idea that our choices stem from bundles of dispositions (tendencies, causal powers, or capacities) much like the ability to speak French, drive, dance, or sing. Crucially, dispositions don’t disappear simply because we haven’t exercised them. If we chose to run at a particular moment, that doesn’t mean we lacked the disposition to walk, crawl, skip, or dance. Likewise, if we spoke French at some point, it doesn’t follow that we couldn’t have spoken English (assuming we know both languages). Or consider something simpler: a working sprinkler system. It has the dispositions required to flood a room – detecting heat or smoke (reasons) and releasing water (making a choice) – even if it never activates during its lifetime.
Free will, according to Vihvelin’s compatibilist account, is the ability to make choices on the basis of reasons and to have this ability is to have a bundle of dispositions. She offers the following menu of dispositions that are likely needed (p. 439):
“[T]he disposition to form and revise beliefs in response to evidence and argument; the disposition to form intentions (choose, try to act) in response to her desires […] and beliefs about how to achieve those desires; the disposition to engage in practical reasoning in response to her intention to make a rational (defensible, justifiable) decision about what to do and her belief that by engaging in practical reasoning she will succeed in making such a decision.”
So, given determinism (laws of nature, history of the universe) and a choice we made, we had a bundle of dispositions that would have enabled us to choose otherwise – free will.
Vihvelin spells out the argument as follows (p. 432):
Dispositions are compatible with determinism.
Abilities are dispositions or bundles of dispositions.
Therefore, the existence of abilities is compatible with determinism.
Free will is the ability to choose on the basis of reasons (I’d add, including subconscious reasons and choosing), and we have this ability by having a bundle of dispositions.
Therefore, free will (the ability to choose on the basis of reasons) is compatible with determinism.
Abilities (like other dispositions) typically continue to exist even when they are not being exercised or manifested.
Therefore, determinism is compatible with the existence of unexercised abilities, including the ability to choose on the basis of reasons.
Abilities are like dispositions with respect to the entailment from the claim that a person has the ability (disposition) to do X to the claim that the person can do X.
Therefore, determinism is compatible with the truth of the claim that persons can choose and do other than what they actually choose and do.
I think that this argument also relates to what Holland (1986, p. 947) called the fundamental problem of causal inference. When evaluating a social programme, we observe the outcomes (like mental health or education) of those who take part, but we can’t observe what someone’s outcome would have been without the programme: the counterfactual outcome. We can infer their counterfactual outcomes (on average) from a well-constructed comparison group, or we can make a reasoned argument based on a theory of what they might have been able to do without the programme, including a theory of their bundles of dispositions.
It’s a similar issue with free will. When someone chooses to do something at some point in time, we can’t observe what their outcome would have been had they chosen to do differently. But, according to Vihvelin’s analysis, if they had free will, they had a bundle of dispositions that would have enabled them to choose otherwise. We can infer how they could have chosen, perhaps by finding people like them who did choose differently, or by making a counterfactual argument on the basis of what their bundle of dispositions are.
Thank you Joanna and the Philosophy Circle at NLC (14 November 2025) for helpful discussion!
Dear fellow trialists, can we please stop using the term treatment when it’s not a medical treatment? We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, and emotional support. Using medical language in the context of schools or criminal justice sounds especially creepy. I think we should also rename estimands like average treatment effect. How about average causal effect, which is sometimes used too? Language matters, especially when we’re trying to build trust with communities who are often suspicious of randomised experiments. If we want our work to be understood and welcomed outside quantitative impact evaluation, we need to be more careful with the metaphors we borrow.
The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.
Dosage is another common term that feels weird. It’s typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. But again, it carries pharmacological baggage. People aren’t just absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.
I’ve used all these terms too as they are so pervasive in the literature – this isn’t me being holier-than-thou. But if we want trials to be accessible and respectful, we need to rethink the language we use. It’s no wonder people are put off trials when the basic terminology sounds so clinical and dehumanising. Careful use of language would make it clearer what we are actually doing and how crucial a role participants play in the programmes we’re evaluating.
Something fun (Słoczyński et al., 2025, p. 3, footnote 1):
“IPT [inverse probability tilting] coincides with Hainmueller’s (2012) entropy balancing estimator when the propensity score is estimated with the logit model”.
(It just holds for ATT.) The picture below illustrates the correspondence, using the Lalonde (1986) dataset (in the cobalt package in R). I computed (knitted R here) the weights using the WeightIt package (Greifer, 2025). Since we’re aiming for ATT, the intervention group weights are all equal to 1, so I’ve omitted them from the graph.
The weights are perfectly correlated and the effective sample sizes are the same. They are scaled differently, though (in the WeightIt implementation at least – need to wade into the original papers): entropy weights sum to the comparison group sample size and IPT weights sum to the intervention group sample size.
References
Greifer, N. (2025). WeightIt: Weighting for Covariate Balance in Observational Studies (version 1.5.0) [R Package].
Słoczyński, T., Uysal, S. D., & Wooldridge, J. M. (2025). Covariate Balancing and the Equivalence of Weighting and Doubly Robust Estimators of Average Treatment Effects. IZA Institute of Labour Economics Discussion Paper, 18147.
