Tag: policy evaluation

The value-ladenness of science, and what it means for evaluation

There’s a long history of work arguing that science is value‑laden and involves evaluative judgements. But what sorts of values and evaluative thinking are involved? I read an analysis by Zina Ward (2026) to get a sense of the latest thinking, and pondered what it might mean for our discipline of evaluation.

Ward reminds us that value‑ladenness is obvious in science. For instance, more research funding is devoted to understanding and curing diseases in humans than in koalas. A recent example is the UK’s £2 billion investment in quantum research. This investment is motivated by anticipated applications such as secure communication, faster algorithms for scientific problems, and advanced sensing. Each of these reflects underlying values: that communication should be secure; that certain scientific problems are worth prioritising; and that military capabilities, such as detecting submarines that evade current technologies, should be strengthened. Ethical values also play a role in shaping the sorts of research that is conducted.

There are at least four different ways that a choice is value-laden, according to Ward. Value-ladenness can be:

  • rational,
  • motivational,
  • causal, or
  • objectual.

Choices that are value-laden in the rational sense provide the justification for a choice. For example, we may want to promote certain kinds of research given priorities in society and some overall ideology. What actually motivates a scientist to conduct research (motivational value-ladenness) may also align with these rational values; however, scientists often conduct research for personal reasons. That has been the case for scientists focusing on Covid-related research (e.g., they lost a loved one to Covid or know someone with long Covid) or working on trans-inclusive theories of gender (e.g., they are trans or have a loved one who is trans). Outside the realm of science, Ward gives the example of someone who cites reducing their carbon footprint and reducing animal suffering as justifications for becoming vegetarian (rational), whereas in reality they did so to fit in with their vegetarian friends (motivational, but also rational if used as a justification).

A choice is causally value-laden if values influence the choices someone makes. For example, the code of ethics for a profession such as psychology constrains the sorts of research that can be conducted. This may also be an example of motivational value-ladenness – psychologists want to conduct research that aligns with these ethical frameworks, and may have been involved in developing them. However, some scientists would be motivated to conduct research that is unethical without the causal constraints – there are plenty of examples of this in history.

Finally, a choice, or an evaluation of a potential choice, can be objectually value-laden. This concerns the impact that the choice has on the world, in relation to values. Choosing to fund more human than koala health studies is an example. Another would be reforming private family law, e.g., through the recently announced national rollout of the Child Focused Model. These choices cannot be made based on facts alone, as the is-ought problem reminds us.

When evaluation is defined as “the process of determining the merit or worth” of things (Scriven, 1994, p. 152), it reflects this latter objectual value-ladenness. Given this definition, it is striking that a recent review of 13 broad evaluation approaches identified only seven that considered judgements of merit or worth “essential”, and only three provided any guidance on how to make these judgements (Schröter et al., 2026). This gap between definition and practice requires some consideration; however, I am still formulating my views on this. Where I’ve got to can be summarised in the following two paragraphs.

Firstly, objectual evaluations are pervasive across science and policy making – evaluation, as the field operates in practice, does not have a monopoly on evaluative thinking. Now it could be that there is a need for a transdisciplinary genre of evaluation (see, e.g., Scriven, 2008) – something similar to logic, psychometrics, or statistics. Conferences for such a discipline would invite anyone who reasons about objectual values, whether they be scientists, educators, vegetarians, or anyone else.

I’d conjecture instead that the current field of evaluation that we know and love is really focused on policy evaluation, and policy evaluation concerns a range of activities other than evaluative thinking. It’s about conducting social research on policies and programmes. One important aspect of policy evaluation is reasoning about objectual values – just as it is across the sciences. But there is more to policy evaluation than this, for example how to develop theories of change, apply participatory approaches in evaluation design, understand methods that can be used to test theories of change, and a huge number of practical considerations involved when conducting research on policy at scale. Finally, I’d venture the conjecture that evaluation in the transdisciplinary sense already exists. It just has a different name and lives somewhere in departments of philosophy and/or politics.

References

Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2026). The garden of evaluation approaches: Supporting explicit, theory-informed evaluation practice. Evaluation.

Scriven, M. (1994). Evaluation as a discipline. Studies in Educational Evaluation, 20(1), 147–166.

Scriven, M. (2008). The concept of a transdiscipline: And of evaluation as a transdiscipline. Journal of MultiDisciplinary Evaluation, 5(10), 65–66.

Ward, Z. B. (2026). What does it mean to say that science is value-laden? In K. C. Elliott & T. Richards, The Routledge Handbook of Values and Science (pp. 74–83). Routledge.

Deepening the theories of mechanisms used in policy evaluation

Theory‑based (or driven) evaluations begin with a theory of change and then design the evaluation to test the causal mechanisms it proposes (Fitz‑Gibbon & Morris, 1975). A well‑constructed theory of change sets out how a programme’s resources are deployed to support delivery activities, and how these activities activate the causal mechanisms that generate outcomes as they evolve over time. The early – and, to my mind, more coherent – formulations of theory‑based evaluation were pluralistic, encompassing all methodological approaches (see, e.g., Chen, 2015): RCTs and quasi‑experiments alongside, for example, process tracing and qualitative comparative analysis.

Mechanisms can be defined in terms of entities and what they do to bring about change (Illari & Williamson, 2011). We can identify those entities and activities with the aid of substantive theories relevant to the policy area under investigation, and the mechanisms are described using the concepts provided by those theories (Ioannidis & Psillos, 2018). A substantive theory is particularly helpful when there is evidence for the mechanisms it proposes, rather than when it rests only on armchair speculation. However, all theories are necessarily incomplete and evolve as new evidence accumulates. Some theories are more detailed than others. Some have been more thoroughly tested than others.

Substantive theories describe mechanism at different levels of explanation. Sun et al. (2005) describe the levels as follows:

Object of analysisType of analysisElements in model
Inter-agentSocial/culturalCollections of agents
AgentsPsychologicalIndividual agents
Intra-agentComponential [I think also psychological]Modular construction of agents
SubstratesPhysiologicalBiological realisation of modules

For an inter‑agent analysis, the focus is on how collections of agents (often people) interact with one another. Examples include models of crowd behaviour when a fire alarm sounds, or systemic processes that shape people’s experiences, such as racism, sexism, transphobia, and their intersections.

For an agent‑level analysis, the focus shifts to individual people. This is roughly the way we talk about individuals in everyday life, including their desires, beliefs, opportunities, and actions, for instance, spending money, talking, listening, going for a run, attending a mentoring session, or doing homework.

An intra‑agent analysis draws on concepts from psychology. This may include theories of cognitive control; for instance, why people sometimes respond automatically and at other times engage in deliberate thought, and how conflicts between competing cognitive systems are resolved. It may include theories of emotion, such as how people’s goals and their (often automatic) inferences about progress toward those goals give rise to feelings of happiness, sadness, fear, or anger. Theories of memory systems also sit here, including how information is temporarily represented in visuospatial and phonological working memory, and the capacity and processing limits of these systems.

Finally, analysis at the substrate level considers the biological processes that implement psychological‑level mechanisms or reflect the consequences of behavioural change. This is most visible in dietary interventions, where targets might include blood glucose or cholesterol levels. It can also include theories from cognitive neuroscience, such as which neural systems underpin cognitive control or memory.

I revisited the theories of change in two evaluations I worked on to see which elements from these four levels of analysis were present.

Case study 1: Basic Maths Premium

Basic Maths Premium was a Department for Education pilot that provided additional funding to post‑16 providers in disadvantaged areas to improve GCSE maths resit outcomes for students with prior attainment at grade 3 or below. It tested three funding models using an RCT: two with guaranteed elements and one based entirely on payment by results (PbR):

ModelUnconditional fundingConditional
funding
A£500 × number of eligible students
B£250 × number of eligible students£250 × number of successful students
C£500 × number successful students

The theory of change was at the agent level. For unconditional funding, the theory was straightforward: funding would enable activities expected to improve outcomes, such as more teaching hours, smaller class sizes, and greater use of technology. By contrast, the theory underpinning conditional funding was less unconvincing. Financial incentives were assumed to boost staff motivation, which would in turn enhance teaching quality, student motivation, and ultimately learning outcomes. However, for Model C in particular, it was uncertain how providers were expected to finance any additional activities, given that all funding under this model depended on student pass rates after any teaching had ended. Two quotations from heads of maths, drawn from the implementation and process evaluation (IPE), captured the issue succinctly (Scott et al., 2024):

  • “The fact that we’ve got this potential funding in the future that will reward us for that, that’s great, but we’ve still got to find the funds now to do what we do” (Model C – conditional £500 per passing student).
  • “I think [payment by results is] really unfair because you would not really know how much money you were going to get. I don’t think we would have been able to spend any additional money on that basis, so for us it wouldn’t have really worked. […] I would never have been able to employ two staff on the basis that I might get a certain amount of students through a GCSE. That would have been too much of a financial risk” (Model A – unconditional £500 per student). Also a good example of counterfactual reasoning without a comparison group.

The IPE showed that actual spend was primarily driven by the unconditional funding:

Case study 2: Stop and Think

Stop and Think is a computer‑based intervention designed to help primary pupils overcome common misconceptions in maths and science. It focuses on strengthening children’s inhibitory control: the ability to pause, reflect, and override intuitive but incorrect responses, by guiding them through short, game‑like activities that present counter-intuitive problems.

The theory underlying the programme included intra-agent and substrate levels of analysis. Briefly, the idea is that when students learn new concepts, they need to overcome intuitively obvious prior beliefs. Mareschal (2016) summarises evidence of the cognitive mechanisms involved when intuitions and new learning clash with each other, e.g., the inhibition of pre-existing beliefs involves processes implemented in the dorsal lateral prefrontal cortex (DLPFC) and the anterior cingulate cortex (ACC). Picture below (Mareschal, 2016, p. 115):

One of the tests of the theory in the RCT evaluating Stop and Think (Takala, et al., 2025) involved the construction of a measure of misconceptions driven by prior beliefs. The impact of the programme on misconceptions was then tested quantitatively using mediation analysis.

Importantly, this variable-based analysis is not the mechanism. Instead, the misconceptions measure was an indirect operationalisation of a trace of the underlying mechanism illustrated in the picture above. This distinction is summarised by Paley and Lilford (2011, pp. 956-7):

“The view […] that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.”

Conclusions

Being explicit about the entities involved, the activities they carry out, the substantive theories that link these activities to change, and the levels of analysis being used has potential to improve the quality of theories of change. In the lead‑up to developing a new theory, it can be valuable to revisit earlier theories and identify their weaknesses, with the aim of learning from past mistakes and doing better.

A persistent challenge, however, is that evaluators are often brought into the process too late, when the theory is already fixed or the programme is underway. Feasibility and pilot studies, with an emphasis on implementation and process evaluation, offer a crucial opportunity to deepen theories of change, fail fast, and adjust course before public funding is committed to programmes with unconvincing rationales.

References

Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd edition). Sage Publications.

Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.

Illari, P. M., & Williamson, J. (2011). What is a mechanism? Thinking about mechanisms across the sciences. European Journal for Philosophy of Science, 2(1), 119–135.

Ioannidis, S., & Psillos, S. (2018). Mechanisms in practice: A methodological approach. Journal of Evaluation in Clinical Practice, 24(5), 1177–1183.

Mareschal, D. (2016). The neuroscience of conceptual learning in science and mathematics. Current Opinion in Behavioral Sciences, 10, 114–118.

Scott, M., Scandone, B., Griggs, J., Roberts, E., Bristow, T., Woolfe, E., Dey, M., & Fugard, A. (2024). Basic Maths Premium evaluation report. Education Endowment Foundation.

Sun, R., Coward, L. A., & Zenzen, M. J. (2005). On levels of cognitive modeling. Philosophical Psychology, 18, 613–637.

Takala, H., Kuo, T.-L., Duysak, E., Bhatti, S., Stoilova, E., Fletcher, A., McGuinness, N., McKaskill, M., & Fugard A. (2025). Stop and Think: Learning Counterintuitive Concepts Evaluation Report. Education Endowment Foundation.

Evaluation as a social science

“We have argued that good evaluation is good social science. For us, this embraces the gallant aims of precision in articulation of theory, rigor in empirical testing, confederation in lines of inquiry, and cumulation in the body of findings. The ‘realist movement,’ of which we are a part, is often considered the brash upstart of the evaluation schools. In fact, it depends on these rather venerable ideas. The future, for us, thus lies in keeping faith with some of the grand old principles of social science and in not forgetting the hard-won lessons of the old studies.” (Pawson & Tilley, 2001, p. 324)

“All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.” (Pawson, 2024, p. 42)

“Despite the habitual use of the appellation ‘social science’, many of my colleagues would reject any claim to follow science, dismiss any interest in causality, deny any need for objectivity, and scorn the possibility of generalisation. They are beyond hope. I don’t seek to convert them. But in following their chosen paths these various tribes – constructivists, post-modernists, emancipators, critics, essayists, relativists, and so on – have found time to say why causality, objectivity, and generality are false idols. So, in defending the science in social science, their criticisms also need to be overturned.” (Pawson, 2024, p. xviii)

“What I’ve come up with here might well be entitled The Old Rules of Sociological Method. I have attempted to extract and justify some realist principles for conducting social research on the back of a generous portfolio of existing examples. Those illustrations reach across many research domains and a broad portfolio of practical methods. But they remain a pinprick; I could have called upon a thousand others. Accordingly, there is another way of perceiving my efforts. The book is no more and no less than an attempt to codify and formalise existing practices. I have tried to capture a tradition.” (Pawson, 2024, 251)

References

Pawson, R., & Tilley, N. (2001). Realistic Evaluation Bloodlines. American Journal of Evaluation, 22(3), 317–324.

Pawson, R. (2024). How to Think Like a realist: A methodology for social science. Edward Elgar Publishing Limited.

The bargraphs of evaluation approaches

Scroll down for a redrawing of Schröter, Becho, and Montrosse-Moorhead’s (2026) characterisation of broad evaluation approaches. R code here.

Their assessments of each approach on 8 dimensions are linked here:

References

Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2026). The garden of evaluation approaches: Supporting explicit, theory-informed evaluation practice. Evaluation.

Master’s in Public Policy (MPP) talk at Cambridge

This week I gave a talk to the MPP class. Slides here. I covered the usual things, e.g., how all policy evaluation is theory-based and counterfactuals don’t need a comparison group. A few new additions:

  • I claimed that Realistic Evaluation, i.e., the (Pawson & Tilley, 1997) book, is an intro to scientific methods and philosophy of science, rather than introducing a distinct form of evaluation. I think this is clearest to see in Pawson (2024), and it’s one of the most muddled sections of the Magenta Book.
  • The importance of theory for interpreting the meaning of covariates as well as merely selecting them. This is particularly important for intersectional understandings of selection into programmes, e.g., you need a theory to work out why, e.g., some combinations of gender and ethnicity, are more likely to be referred to a targetted programme. Is it driven by gender and/or racial stereotypes at time of selection, or broader societal discrimination before selection, for example?
  • An example from the Green Book (para 4.17) of a type of counterfactual reasoning, used when appraising different ways to realise a policy. I think the reasoning behind “deadweight” and “additionality” outcomes may use future less vivid conditionals. Still reading.

The Westminster Model and why it’s useful even though it’s wrong

This paper by Paul Cairney (2025) is fun.

The Westminster Model (WM) of policymaking:

“… political parties present manifestos to compete for public votes in general elections, the winner forms a government, and government ministers oversee the delivery of their manifesto by civil servants and public bodies.” (p. 297)

What’s wrong with the British Political Tradition (BPT), the UK flavour of the model:

“… politicians do not live up to the BPT; party manifestos combine some specific and many vague promises; parties don’t always gain a majority; not all MPs are easy to keep in line with government policy; ministers could not understand all of the choices made in their name; EU entry and exit was a bruising experience; and, the UK’s ability to reform devolved or subnational government is constrained in practice. In other words, the first aim of a ‘governance thesis’ is to use the WM as an ideal-type to compare with more accurate stories of the real world.” (p. 298)

Cairney then argues that “these differences between aspiration and reality provide the launching point for a more realistic story” – and goes on to have a go, e.g., noting the impact of the finite time and capacity of ministers, expanding on the range of individuals and informal and formal groups that influence policymaking

The approach reminds me of how classical logic and probability are used in the psychology of reasoning: they offer frameworks for understanding how people ought to reason, and then researchers adjust that framework to match how non‑mathematicians actually do reason, constrained by finite working memory, drawing on broader interpretations of terms like “if” than are used in logic, etc.

References

Cairney, P. (2025). Governance and the Westminster model: What exactly is the dominant story of UK policymaking? British Politics, 20(3), 295–315.

Theory-based evaluation as tautology

“Realist” evaluation and the more general “theory‑based” evaluation umbrella often present themselves as distinct forms of evaluation; however, their core assumptions that programmes work (or not) through underlying causal mechanisms that cannot be directly observed are restatements of what sciences already presuppose. It’s rare indeed for an evaluation not to begin with a theory of change; the issue is not whether that’s necessary but how good theories are. We’ve all seen boxology lists in the logic model style.

The full range of evaluation methods can test mechanisms rather than only whether an outcome has been shifted by a programme and current practice involves qual and quant evidence, e.g., the latter using mediation tests. The logic of “what works best for whom in what context” predates “realist” approaches by a few decades and is used across all methods. Similarly, there’s nothing uniquely “realist” about context-mechanism-outcome triads: it’s a useful way to move from boxology to theory prose.

Advocating “realist” or “theory‑based” evaluation is essentially advocating “scientific evaluation” – a kind of tautology. It may be helpful for some readers to learn about separating ontology from epistemology (the map is not the territory, to paraphrase Korzybski), but I’d suggest that’s best left to the first chapter of an introduction to evaluation rather than something that needs to be restated in every report. There are examples of fields that stick on the “science” term: neuroscience and cognitive science are two examples. But what would unscientific policy evaluation look like? What does it get called? (That’s a genuine question – I have a blog post brewing.)

Theorising what evidence would test a theory

Policy evaluation involves speculating about what might be going on – a creative process – based on what we currently (think we) know, and then working out what evidence we would expect to see if we were correct or if plausible alternative theories were true. That logic is explicit in process tracing, but it is far more general than that. The oft‑cited Van Evera (1997) rules, such as the “smoking gun”, follow directly from Bayes’ rule. We cannot directly observe a programme’s mechanisms of change; many different theories will be compatible with both qualitative and quantitative evidence. That problem has a name, underdetermination, and it is a challenge across the sciences.

An interesting example of the gap between theory and evidence is how physicists detect gravitational waves produced by black holes merging. The Laser Interferometer Gravitational-wave Observatory (LIGO) does not observe the waves themselves; it measures how those waves distort spacetime by a miniscule amount, far smaller than a proton.

My favourite analogy comes from trying to crack encrypted texts. We can systematically analyse letter and word frequencies in ciphertexts to spot patterns, but that only takes us so far (try the exercises in Rubinstein‑Salzedo, 2018). It helps to guess what people might be trying to say to each other based on something beyond the ciphertext; for example, that they open with “How are you?” or that they are likely to be discussing a particular event. These knowledge‑grounded guesses, drawing on a wealth of experience and prior evidence, help reduce the search space of possible encryptions. Something similar is going on in evaluation when we iterate between programme theories, tests of those theories, and exploratory research to generate ideas for new theories.

References

Rubinstein‑Salzedo, S. (2018). Cryptography. Springer.

Van Evera, S. (1997). Guide to methods for students of political science. Cornell University Press.

Interventions to reduce social media use

Goldfield et al.’s (2026) RCT looks interesting – the intervention was reducing social media use to max an hour a day. They found a 0.11 reduction in loneliness versus control on the UCLA Loneliness Scale. SD at endline was 0.65, so standardised mean difference (SMD) = 0.17.

A systematic review (Burnell, 2025) found an SMD of 0.12 (95% CI -0.02 to 0.26) for loneliness, 0.16 (CI = 0.04 to 0.28) when they screened out one study judged to have issues, leaving 13. (Haven’t waded into the details of each intervention or what those issues are.)

References

Burnell, K., Meter, D. J., Andrade, F. C., Slocum, A. N., & George, M. J. (2025). The effects of social media restriction: Meta-analytic evidence from randomized controlled trials. SSM – Mental Health, 7, 100459.

Goldfield, G. S., Lopes, M. V. V., Mahboob, W., Perry, S., & Davis, C. (2026). Reducing social media use decreases loneliness regardless of gender or level of social comparisons in youth with anxiety and depression: A randomized controlled trial. Journal of Affective Disorders, in press.

Explaining evaluation to the public

If you have ever introduced yourself to someone as an “evaluator”, you will quickly discover that very few people outside the field have the first clue what that involves.

The term is incredibly broad. Am I evaluating your dress sense or how you dance? Am I running psychometric assessments to evaluate staff performance? We are all evaluators, regardless of how often we share the judgements.

One of the functions of the UK Evaluation Society is to explain to the public what we do. So, how can we do this?

I think there are two main communication challenges for evaluation. The first challenge is finding a concise term that signals what evaluation involves to people who do not work in the field. It seems likely the easiest way to do this is simply to add a word or two to “evaluation”, so the problem is: which words? The second challenge is to find ways to explain what evaluation involves in practice, by using clear, straightforward language that can also ideally be applied more consistently in evaluation reports.

Words for evaluation – the case for social policy evaluation

Given how baffled people tend to look when I call myself an evaluator, I’ve previously said I’m a social researcher and work in social policy, and then just explained what that means through examples. Since October, I’ve tried “policy evaluator” or “social policy evaluator” and they seem to fare surprisingly well. The most obvious alternative to these might have been “programme evaluator”, but my problem with this is that I think programmes are more obviously associated with software or what’s on TV.

One possible objection to the term policy evaluation is that policies are rather broad, e.g., the Green Book (p. 6) says a policy is “a statement of intent that is implemented through a procedure or a protocol and a deliberate system of principles to guide decisions and achieve rational outcomes.” Evaluations often focus more specifically on programmes and service delivery. However, programmes put policies into practice, and evaluation findings should inform those policies. If a programme is shown to work, that has clear implications for policy. Plus, policies aren’t just found in government: schools, charities, and other organisations have them too. I think it’s important to show we engage in this broader context.

Explaining what we do

Policy evaluation is a broad field, covering many sectors, services and types of activity. It can look at whether public services, the work of charities, or even individual practitioners are making a meaningful difference through what they’re doing. It might tackle local programmes or national reforms. Evaluation also uses the full range of methods, mixing qualitative and quantitative evidence, in large surveys and focused case studies. Examples that illustrate this variety of topics and methods are essential, as Sarah Mason highlighted in one of the few studies exploring how evaluators explain their work to the public.

Some of the terminology we use within evaluation contexts is unnecessarily opaque and misleading, which is unhelpful for understanding the role. A good example – a word I’ve used more often than I should – is “treatment”. We’re not administering drugs – we’re evaluating programmes like mentoring, approaches to teaching, or emotional support. The term “treatment” carries medical baggage, that doesn’t fit with social programmes. My preference would be to label study conditions after what’s actually being tested. For instance, if we were evaluating ACME Therapy, the conditions would be something like ACME Therapy and usual practice. Using medical language in the context of trials in schools or criminal justice can sound especially creepy – it reminds me of A Clockwork Orange.

Evaluating how we explain ourselves

As policy evaluators we should evaluate how well we communicate with the public. For example, participants could be provided with information about what evaluation is and how it is used, enabling them to take an informed, participatory role in co‑designing how evaluation itself is explained. We could then use surveys, following Sarah Mason’s example, to test the effectiveness of different ways of explaining evaluation at a larger scale.

Evaluation is a broad transdiscipline, working across a wide range of contexts, so it is unlikely that we will find a single way of explaining ourselves that works everywhere. However, a collection of different terms and explanations that work well in at least some contexts would be valuable in reconnecting our work with the people we hope to serve. This would also help to attract new policy evaluators to what is one of the most important and exciting social sciences.

Originally posted on the UKES blog (February 4, 2026).