There’s a long history of work arguing that science is value‑laden and involves evaluative judgements. But what sorts of values and evaluative thinking are involved? I read an analysis by Zina Ward (2026) to get a sense of the latest thinking, and pondered what it might mean for our discipline of evaluation.
Ward reminds us that value‑ladenness is obvious in science. For instance, more research funding is devoted to understanding and curing diseases in humans than in koalas. A recent example is the UK’s £2 billion investment in quantum research. This investment is motivated by anticipated applications such as secure communication, faster algorithms for scientific problems, and advanced sensing. Each of these reflects underlying values: that communication should be secure; that certain scientific problems are worth prioritising; and that military capabilities, such as detecting submarines that evade current technologies, should be strengthened. Ethical values also play a role in shaping the sorts of research that is conducted.
There are at least four different ways that a choice is value-laden, according to Ward. Value-ladenness can be:
rational,
motivational,
causal, or
objectual.
Choices that are value-laden in the rational sense provide the justification for a choice. For example, we may want to promote certain kinds of research given priorities in society and some overall ideology. What actually motivates a scientist to conduct research (motivational value-ladenness) may also align with these rational values; however, scientists often conduct research for personal reasons. That has been the case for scientists focusing on Covid-related research (e.g., they lost a loved one to Covid or know someone with long Covid) or working on trans-inclusive theories of gender (e.g., they are trans or have a loved one who is trans). Outside the realm of science, Ward gives the example of someone who cites reducing their carbon footprint and reducing animal suffering as justifications for becoming vegetarian (rational), whereas in reality they did so to fit in with their vegetarian friends (motivational, but also rational if used as a justification).
A choice is causally value-laden if values influence the choices someone makes. For example, the code of ethics for a profession such as psychology constrains the sorts of research that can be conducted. This may also be an example of motivational value-ladenness – psychologists want to conduct research that aligns with these ethical frameworks, and may have been involved in developing them. However, some scientists would be motivated to conduct research that is unethical without the causal constraints – there are plenty of examples of this in history.
Finally, a choice, or an evaluation of a potential choice, can be objectually value-laden. This concerns the impact that the choice has on the world, in relation to values. Choosing to fund more human than koala health studies is an example. Another would be reforming private family law, e.g., through the recently announced national rollout of the Child Focused Model. These choices cannot be made based on facts alone, as the is-ought problem reminds us.
When evaluation is defined as “the process of determining the merit or worth” of things (Scriven, 1994, p. 152), it reflects this latter objectual value-ladenness. Given this definition, it is striking that a recent review of 13 broad evaluation approaches identified only seven that considered judgements of merit or worth “essential”, and only three provided any guidance on how to make these judgements (Schröter et al., 2026). This gap between definition and practice requires some consideration; however, I am still formulating my views on this. Where I’ve got to can be summarised in the following two paragraphs.
Firstly, objectual evaluations are pervasive across science and policy making – evaluation, as the field operates in practice, does not have a monopoly on evaluative thinking. Now it could be that there is a need for a transdisciplinary genre of evaluation (see, e.g., Scriven, 2008) – something similar to logic, psychometrics, or statistics. Conferences for such a discipline would invite anyone who reasons about objectual values, whether they be scientists, educators, vegetarians, or anyone else.
I’d conjecture instead that the current field of evaluation that we know and love is really focused on policy evaluation, and policy evaluation concerns a range of activities other than evaluative thinking. It’s about conducting social research on policies and programmes. One important aspect of policy evaluation is reasoning about objectual values – just as it is across the sciences. But there is more to policy evaluation than this, for example how to develop theories of change, apply participatory approaches in evaluation design, understand methods that can be used to test theories of change, and a huge number of practical considerations involved when conducting research on policy at scale. Finally, I’d venture the conjecture that evaluation in the transdisciplinary sense already exists. It just has a different name and lives somewhere in departments of philosophy and/or politics.
“We have argued that good evaluation is good social science. For us, this embraces the gallant aims of precision in articulation of theory, rigor in empirical testing, confederation in lines of inquiry, and cumulation in the body of findings. The ‘realist movement,’ of which we are a part, is often considered the brash upstart of the evaluation schools. In fact, it depends on these rather venerable ideas. The future, for us, thus lies in keeping faith with some of the grand old principles of social science and in not forgetting the hard-won lessons of the old studies.” (Pawson & Tilley, 2001, p. 324)
“All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.” (Pawson, 2024, p. 42)
“Despite the habitual use of the appellation ‘social science’, many of my colleagues would reject any claim to follow science, dismiss any interest in causality, deny any need for objectivity, and scorn the possibility of generalisation. They are beyond hope. I don’t seek to convert them. But in following their chosen paths these various tribes – constructivists, post-modernists, emancipators, critics, essayists, relativists, and so on – have found time to say why causality, objectivity, and generality are false idols. So, in defending the science in social science, their criticisms also need to be overturned.” (Pawson, 2024, p. xviii)
“What I’ve come up with here might well be entitled The Old Rules of Sociological Method. I have attempted to extract and justify some realist principles for conducting social research on the back of a generous portfolio of existing examples. Those illustrations reach across many research domains and a broad portfolio of practical methods. But they remain a pinprick; I could have called upon a thousand others. Accordingly, there is another way of perceiving my efforts. The book is no more and no less than an attempt to codify and formalise existing practices. I have tried to capture a tradition.” (Pawson, 2024, 251)
“Realist” evaluation and the more general “theory‑based” evaluation umbrella often present themselves as distinct forms of evaluation; however, their core assumptions that programmes work (or not) through underlying causal mechanisms that cannot be directly observed are restatements of what sciences already presuppose. It’s rare indeed for an evaluation not to begin with a theory of change; the issue is not whether that’s necessary but how good theories are. We’ve all seen boxology lists in the logic model style.
The full range of evaluation methods can test mechanisms rather than only whether an outcome has been shifted by a programme and current practice involves qual and quant evidence, e.g., the latter using mediation tests. The logic of “what works best for whom in what context” predates “realist” approaches by a few decades and is used across all methods. Similarly, there’s nothing uniquely “realist” about context-mechanism-outcome triads: it’s a useful way to move from boxology to theory prose.
Advocating “realist” or “theory‑based” evaluation is essentially advocating “scientific evaluation” – a kind of tautology. It may be helpful for some readers to learn about separating ontology from epistemology (the map is not the territory, to paraphrase Korzybski), but I’d suggest that’s best left to the first chapter of an introduction to evaluation rather than something that needs to be restated in every report. There are examples of fields that stick on the “science” term: neuroscience and cognitive science are two examples. But what would unscientific policy evaluation look like? What does it get called? (That’s a genuine question – I have a blog post brewing.)
Policy evaluation involves speculating about what might be going on – a creative process – based on what we currently (think we) know, and then working out what evidence we would expect to see if we were correct or if plausible alternative theories were true. That logic is explicit in process tracing, but it is far more general than that. The oft‑cited Van Evera (1997) rules, such as the “smoking gun”, follow directly from Bayes’ rule. We cannot directly observe a programme’s mechanisms of change; many different theories will be compatible with both qualitative and quantitative evidence. That problem has a name, underdetermination, and it is a challenge across the sciences.
An interesting example of the gap between theory and evidence is how physicists detect gravitational waves produced by black holes merging. The Laser Interferometer Gravitational-wave Observatory (LIGO) does not observe the waves themselves; it measures how those waves distort spacetime by a miniscule amount, far smaller than a proton.
My favourite analogy comes from trying to crack encrypted texts. We can systematically analyse letter and word frequencies in ciphertexts to spot patterns, but that only takes us so far (try the exercises in Rubinstein‑Salzedo, 2018). It helps to guess what people might be trying to say to each other based on something beyond the ciphertext; for example, that they open with “How are you?” or that they are likely to be discussing a particular event. These knowledge‑grounded guesses, drawing on a wealth of experience and prior evidence, help reduce the search space of possible encryptions. Something similar is going on in evaluation when we iterate between programme theories, tests of those theories, and exploratory research to generate ideas for new theories.
References
Rubinstein‑Salzedo, S. (2018). Cryptography. Springer.
Van Evera, S. (1997). Guide to methods for students of political science. Cornell University Press.
Joas and Knöbl (2009, p. 6) on how Popper demolished positivism in 1934:
As Popper lays out in his now very famous book The Logic of Scientific Discovery, which first appeared in 1934, in the case of most scientific problems we cannot be certain whether a generalization, that is, a theory or hypothesis, truly applies in all cases. In all probability, we will never be able to verify once and for all the astrophysical statement that ‘All planets move around their suns along an elliptical trajectory’, because we are unlikely ever to get to know all the solar systems in the universe and therefore we will presumably never be able to confirm with absolute certainty that every single planet does in fact follow an elliptical trajectory around its sun, as opposed to some other route. Much the same applies to the statement ‘All swans are white’. Even if you have seen thousands of swans and all of them were in fact white, you can ultimately never be certain that a black, green, blue, etc. swan will not show up at some point. As a rule, universal statements cannot therefore be confirmed or verified. To put it another way: inductive arguments (that is, inference from individual instances to a totality) are neither logically valid nor truly compelling arguments; induction cannot be justified purely in terms of logic, because we are unable to rule out the possibility that one observation may eventually be made that refutes the general statement thought to be corroborated. Positivists’ attempts to trace laws back to elementary observations or to derive them from elementary observations and verify them are thus doomed to failure. This was precisely Popper’s criticism.
Joas, H., & Knöbl, W. (2009). What is theory? In Social theory: Twenty introductory lectures (pp. 1–19). Cambridge University Press.
“[…] the most reckless and treacherous of all theorists is he who professes to let facts and figures speak for themselves, who keeps in the background the part he has played, perhaps unconsciously, in selecting and grouping them […]”
– Alfred Marshall (1885, February 24). The present position of economics. Inaugural lecture, University of Cambridge. Reprinted in Hodgson, G. M. (2005). Journal of Institutional Economics, 1(1), 121–137.
Social policy evaluation inherits concepts from multiple disciplines, and somewhere along the way many of them have become muddled. The list below sketches what I think are seven of the most harmful myths.
1. “Counterfactual” is a synonym for control/comparison group
There is a long history of research on counterfactual reasoning in the absence of a control group. For example, analyses of how people should and can determine the truth of a counterfactual conditional such as “If Oswald had not killed Kennedy, no one else would have” or “If I’d left the house earlier, I’d have caught that bus” (e.g., Adams, 1970; Halpern, 2015). We ponder counterfactuals like these all the time without running RCTs.
The evaluation literature has also considered how counterfactuals can be used without a comparison group (e.g., White, 2010) – including with qualitative evidence (e.g., Reichardt, 2022). In evaluation, the counterfactuals are conditionals like, “If intervention group students hadn’t been offered the mentoring programme, x% fewer would have passed their GCSEs.”
Some evidence for counterfactual outcomes is more robust than others. One example analysed by Reichardt (2022) involves asking programme participants what they think would have happened if they had not taken part in the programme. Their ability to answer this depends on their capacity for counterfactual reasoning, which may be limited for some programmes. For instance, many people believe that homeopathy is effective, even though its remedies are so highly diluted that they contain no active ingredients and cannot have any biological effect (Ernst, 2005). Counterfactual reasoning based on such beliefs could therefore falsely conclude that a programme caused improvements that would have occurred anyway.
Quantitative designs can be used to estimate counterfactual outcomes without a comparison group. One example is interrupted time series, which uses trends in pre-intervention outcomes to estimate post-intervention counterfactual outcomes. This is illustrated below:
The standard analysis estimates whether introducing the intervention shifted the outcome and whether it changed the slope, both relative to the inferred counterfactual outcomes.
When a comparison group is used to estimate counterfactual outcomes, it is a factual group – it consists of real data from real people. If it were a counterfactual group, it couldn’t be used to estimate anything. This is why I avoid the term “counterfactual group”.
2. RCTs only estimate overall average effects
If you think causal effects may vary by, e.g., participant characteristics or the context they are in, then you can test this by using moderator analysis and obtain estimates for each group. For example, trials funded by the Education Endowment Foundation usually investigate whether the magnitude of a programme’s effects differ for students who are eligible for free school meals – a proxy for socioeconomic disadvantage (e.g., Takala, et al., 2025). Westhorp and Feeny (2024) suggest using moderator analysis to test the effect of context in realist evaluations (see also Myth 4 below).
One challenge is that when testing for moderator effects for variables with a large number of levels – for example, in intersectional analyses of gender and ethnicity – models can become unwieldy very quickly. The UK census, for example, includes 19 ethnicities and 6 genders (including cis and trans identities). That results in 114 possible combinations. A promising solution is multilevel analysis of individual heterogeneity and discriminatory accuracy – MAIHDA, for short (Merlo, 2018).
3. (Trialists believe that) RCTs don’t need a programme theory
Individual causal effects are defined as the difference between what a person’s outcome would be following intervention and what it would be following control. We can’t observe this within‑person difference because each participant only experiences one condition. With random assignment, however, the average outcome of those in the intervention group minus the average outcome of those in the control group (a between‑person difference) provides an unbiased estimate of the average of individual causal effects (averages of the unmeasureable within-person differences) – even without any covariates. That’s the great thing about RCTs.
RCTs still need theory. How do you choose a control group? Outcome variables? How do you interpret estimated causal effects, i.e., what the difference between intervention and control outcomes means? Chen and Rossi (1980) explain how theory-based (what they call theory-driven) RCTs that include theory-informed covariates yield more precise estimates of effects (reducing the probability of Type II error), even though those covariates are not needed to control Type I error. That’s why sample size calculators ask how much variance in outcomes is thought to be explained by covariates. RCTs can also test mechanisms of change, e.g., using mediation analysis or through factorial designs.
A related myth is that trialists believe reality consists of variables. Paley and Lilford (2011, pp. 956-7) dispute this:
The view attributed to positivism, that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.
Good theories of change go beyond the variables to discuss people and other entities and what they do (see Myth 6). Variables operationalise aspects of these.
Incidentally, trialists don’t need to be positivists either. Bogen and Woodward (1988) provide an accessible discussion of how to think about the relationship between theory and observed phenomena without falling into positivism.
4. Realist evaluation is a distinct type of evaluation
What people do in practice when they say they are conducting a realist evaluation or “using realist principles” often makes sense, and Pawson and Tilley’s work has helped bring theory to the fore in evaluation (I put at least one reading of theirs on my social theorising course when I was an academic). But the idea that you need a realist evaluation to find out “what works best for whom in what context” or to describe change in terms of context, mechanisms, and outcomes, is just absurd. Many evaluation approaches do this.
Paul (1967, p. 111) offers an earlier example of the what works for whom logic:
[…] the question towards which all outcome research should ultimately be directed is the following: What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances?
Pawson and Tilley (1997, p. 10) cite Palmer (1975, p. 150), who uses similar logic:
Rather than ask, “What works for offenders as a whole?” we must increasingly ask “Which methods work best for which types of offenders, and under what conditions or in what types of setting ?”
School climate research offers one example of contextual factors: the physical characteristics of schools; the formal and informal rules that operate within them; and the norms, beliefs, and values at play (Anderson, 1982). RCTs have examined the extent to which school climate moderates programme effects – i.e., how outcomes depend on the settings in which programmes are delivered (e.g., Low & Van Ryzin, 2014) – as well as the mediating role of climate, where a programme changes aspects of school climate and these changes help to explain its effects (e.g., Singla et al., 2021).
My reading of Pawson and Tilley (1997) and Pawson (2024) is that they’re introducing the scientific method and philosophy of science to evaluators who don’t have a formal training in a science. The ideas apply to any form of evaluation. For example, Pawson (2024, p. 42) argues that
All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.
Science is not simply about observing regularities or estimating average causal effects. It is about explaining why observed patterns occur, how they are produced, and under what conditions they hold. This logic applies whether you are running an RCT, conducting a qualitative impact evaluation, or analysing administrative data using a quasi-experiment – and whether or not you call what you’re doing “realist evaluation”.
The “realist” in the term has a meaning from philosophy: ontological realism. Many people are realists without carrying a realist evaluator card.
5. “Theory-based evaluation” excludes trials and quasi-experiments
The original conceptualisations of theory-based evaluation, or its many synonyms (e.g., theory-driven evaluation, program theory evaluation, program theory–driven evaluation science) includes any approach that begins with a theory of change, uses it to design an evaluation that tests the theory, and revises the theory in light of findings (see, e.g., Chen & Rossi, 1983; Cook, 2000; Fitz-Gibbon & Morris, 1975; Weiss, 1972).
Quasi‑experiments are particularly dependent on programme theory. For example, studies that construct a comparison group using matching or weighting rely on identifying all important covariates. Once those covariates are specified, we can check whether matching/weighting has made the intervention and comparison groups equivalent at baseline; however, balance checks cannot tell us whether any important covariates were left out – theory is needed for that.
Additionally, a large number of causal models will be consistent with the data, so you need a theory to choose between them. For example, given only data, the following six simple causal models are statistically equivalent. Each model has three variables: outcome (out); condition, e.g., the programme being evaluated or usual practice (prog); and mediator (med).
Model 1 is the intended causal interpretation: the effect of the programme is partially explained by the mediator. Model 2, for example, says that programme and outcome are statistically associated with each other because the supposed mediator is actually a common cause.
The Magenta Book recommends using theory-based evaluation if you can’t find a comparison group (HM Treasury, 2020, p. 47). Annex A (analytical methods for use within an evaluation) locates trials and quasi-experiments outside the theory-based list. I think the Magenta Book is partly responsible for perpetuating Myth 5 in the UK – particularly through invitations to tender that follow the book, which leave evaluators having to play a semantic game to win work. I hope the upcoming revision corrects this (update: it doesn’t), and that it becomes routine practice to commission theory-based trials, quasi-experiments, and mixed methods evaluations, alongside theory-based qualitative impact evaluations and small-n case studies.
6. Theories of change are a kind of diagram
Theories of change are theories – of change. They may be illustrated using a diagram, but there needs to be an explanation, usually using prose, describing how the resources, entities (e.g., people, organisations, laws, etc.), and activities carried out lead to outputs and outcomes. Currently entrenched practices of developing a Theory of Change™ seem to lead people to forget what they learned about theory and theorising at university.
Pawson and Tilley (1997, p. 134) provide neat examples of how to organise theories of change explanations in tables. Tables like these are part of the “realist evaluation” tradition; however, the logic is clearly much more general (see Myth 4):
Context
Mechanism
Outcome
High numbers of prepayment meters, with a high proportion of burglaries involving cash from meters
Removal of cash meters reduces incentive to burgle by decreasing actual or perceived rewards
Reduction in percentage of burglaries involving meter breakage; reduced risk of burglary at dwellings where meters are removed; reduced burglary rate overall
If a client forces you to include only a diagram in a report, plead with them to let you add the actual theory to an Appendix. We’ve been in this situation many times and used this solution once to date – let’s see if the report is published (update: it wasn’t).
7. Evaluations test “treatments”
Well, in a formally defined sense they do; however, not as people usually use the term. We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, emotional support. The term “treatment” carries with it medical baggage, which is very different to social programmes. My preference would be to use names that are as close as possible to the conditions. For instance, if we were evaluating ACME Therapy, the conditions would be something like ACME Therapy and usual practice or ACME Therapy and CBT.
The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.
Dosage is typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. People aren’t absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.
Using medical language in the context of trials in schools or criminal justice sounds especially creepy – it reminds me of A Clockwork Orange. I have been guilty of using these terms too. Sometimes it’s just too difficult to challenge medicalised tradition when there are pressing deadlines, and causal estimands like average treatment effect on the treated are pervasive in the methods literature (e.g., Cunningham, 2021, Section 4.1.2).
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In A. Petrosino, P. J. Rogers, T. A. Huebner, & T. A. Hacsi (Eds.), New directions in evaluation: Program Theory in Evaluation: Challenges and Opportunities (pp. 27–34). Jossey-Bass.
Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.
