Theory-based evaluation as tautology

“Realist” evaluation and the more general “theory‑based” evaluation umbrella often present themselves as distinct forms of evaluation; however, their core assumptions that programmes work (or not) through underlying causal mechanisms that cannot be directly observed are restatements of what sciences already presuppose. It’s rare indeed for an evaluation not to begin with a theory of change; the issue is not whether that’s necessary but how good theories are. We’ve all seen boxology lists in the logic model style.

The full range of evaluation methods can test mechanisms rather than only whether an outcome has been shifted by a programme and current practice involves qual and quant evidence, e.g., the latter using mediation tests. The logic of “what works best for whom in what context” predates “realist” approaches by a few decades and is used across all methods. Similarly, there’s nothing uniquely “realist” about context-mechanism-outcome triads: it’s a useful way to move from boxology to theory prose.

Advocating “realist” or “theory‑based” evaluation is essentially advocating “scientific evaluation” – a kind of tautology. It may be helpful for some readers to learn about separating ontology from epistemology (the map is not the territory, to paraphrase Korzybski), but I’d suggest that’s best left to the first chapter of an introduction to evaluation rather than something that needs to be restated in every report. There are examples of fields that stick on the “science” term: neuroscience and cognitive science are two examples. But what would unscientific policy evaluation look like? What does it get called? (That’s a genuine question – I have a blog post brewing.)

Theory-ladenness

An interesting paper on theory-ladenness in research on black holes (Doboszewski & Elder, 2025). Relevant to debates in policy evaluation on theory-based evaluation too and the pervasive role of theory whether or not you name the TBE brand.

‘Philosophers have raised a range of worries about the ways in which observations are theory-laden. In this article, we are focusing on the theory-ladenness of measurement outcomes—the ways in which the “observations” derived from an experiment or measurement are affected by theoretical assumptions that are made in the course of the experiment. The greatest concern is that theory-ladenness may lead to a vicious circularity where the theoretical assumptions made by the experimenters—either in the physical design of the experiment or in subsequent inferences from the empirical data—make it more likely or even guarantee that the observation will confirm the theory being tested. This might occur if the hypothesis being tested is presupposed by the experimental methods being used. Franklin (2015, 156) provides the following toy example of this: if we use a mercury thermometer to test a hypothesis that substances expand as their temperature increases, this is theory-laden in a way that risks vicious circularity because the proper functioning of a mercury thermometer relies on the very hypothesis being tested.’ (pp. 829-830)

‘… the impossibility of controlled experiments on astrophysical targets (stars, galaxies, etc.) places constraints on how astronomers can investigate these systems. Simulations can help make up for the lack of controlled interventions by providing a proxy system into which one can intervene.’ (p. 830)

Doboszewski, J., & Elder, J. (2025). How Theory-Laden Are Observations of Black Holes? Philosophy of Science, 92(4), 827–849.

Preprint here.

Positivism

Joas and Knöbl (2009, p. 6) on how Popper demolished positivism in 1934:

As Popper lays out in his now very famous book The Logic of Scientific Discovery, which first appeared in 1934, in the case of most scientific problems we cannot be certain whether a generalization, that is, a theory or hypothesis, truly applies in all cases. In all probability, we will never be able to verify once and for all the astrophysical statement that ‘All planets move around their suns along an elliptical trajectory’, because we are unlikely ever to get to know all the solar systems in the universe and therefore we will presumably never be able to confirm with absolute certainty that every single planet does in fact follow an elliptical trajectory around its sun, as opposed to some other route. Much the same applies to the statement ‘All swans are white’. Even if you have seen thousands of swans and all of them were in fact white, you can ultimately never be certain that a black, green, blue, etc. swan will not show up at some point. As a rule, universal statements cannot therefore be confirmed or verified. To put it another way: inductive arguments (that is, inference from individual instances to a totality) are neither logically valid nor truly compelling arguments; induction cannot be justified purely in terms of logic, because we are unable to rule out the possibility that one observation may eventually be made that refutes the general statement thought to be corroborated. Positivists’ attempts to trace laws back to elementary observations or to derive them from elementary observations and verify them are thus doomed to failure. This was precisely Popper’s criticism.

Joas, H., & Knöbl, W. (2009). What is theory? In Social theory: Twenty introductory lectures (pp. 1–19). Cambridge University Press.

Folk psychological platitudes

“[…] if I wish to be understood by others, and if I wish to enlist others for my purposes, then I must present myself in a way that conforms to folk-psychological platitudes, or laws. To explain my actions – even my most outrageous ones – to others is to find an appropriate set of folk-psychological platitudes in the light of which I am understandable and predictable. […] Given the fundamental role of folk psychology, it should not come as a surprise that it is also the intuition most strongly protected by sanctions. Inability to come up with acceptable folk-psychological accounts of ourselves and others is sanctioned by disapproval, lack of acceptance, or even referral to psychiatric services. No wonder, therefore, that social psychologists find their subjects ‘telling more than they know’: rather than admit that they have no introspective access to many of their higher cognitive processes, subjects will tell folk-psychological stories of how their mind allegedly works.”
– Martin Kusch (1999, p. 239), Psychological knowledge: A social history and philosophy. Routledge.