Tag: policy evaluation

Evaluation as a social science

“We have argued that good evaluation is good social science. For us, this embraces the gallant aims of precision in articulation of theory, rigor in empirical testing, confederation in lines of inquiry, and cumulation in the body of findings. The ‘realist movement,’ of which we are a part, is often considered the brash upstart of the evaluation schools. In fact, it depends on these rather venerable ideas. The future, for us, thus lies in keeping faith with some of the grand old principles of social science and in not forgetting the hard-won lessons of the old studies.” (Pawson & Tilley, 2001, p. 324)

“All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.” (Pawson, 2024, p. 42)

“Despite the habitual use of the appellation ‘social science’, many of my colleagues would reject any claim to follow science, dismiss any interest in causality, deny any need for objectivity, and scorn the possibility of generalisation. They are beyond hope. I don’t seek to convert them. But in following their chosen paths these various tribes – constructivists, post-modernists, emancipators, critics, essayists, relativists, and so on – have found time to say why causality, objectivity, and generality are false idols. So, in defending the science in social science, their criticisms also need to be overturned.” (Pawson, 2024, p. xviii)

“What I’ve come up with here might well be entitled The Old Rules of Sociological Method. I have attempted to extract and justify some realist principles for conducting social research on the back of a generous portfolio of existing examples. Those illustrations reach across many research domains and a broad portfolio of practical methods. But they remain a pinprick; I could have called upon a thousand others. Accordingly, there is another way of perceiving my efforts. The book is no more and no less than an attempt to codify and formalise existing practices. I have tried to capture a tradition.” (Pawson, 2024, 251)

References

Pawson, R., & Tilley, N. (2001). Realistic Evaluation Bloodlines. American Journal of Evaluation, 22(3), 317–324.

Pawson, R. (2024). How to Think Like a realist: A methodology for social science. Edward Elgar Publishing Limited.

The bargraphs of evaluation approaches

Scroll down for a redrawing of Schröter, Becho, and Montrosse-Moorhead’s (2026) characterisation of broad evaluation approaches. R code here.

Their assessments of each approach on 8 dimensions are linked here:

References

Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2026). The garden of evaluation approaches: Supporting explicit, theory-informed evaluation practice. Evaluation.

Master’s in Public Policy (MPP) talk at Cambridge

This week I gave a talk to the MPP class. Slides here. I covered the usual things, e.g., how all policy evaluation is theory-based and counterfactuals don’t need a comparison group. A few new additions:

  • I claimed that Realistic Evaluation, i.e., the (Pawson & Tilley, 1997) book, is an intro to scientific methods and philosophy of science, rather than introducing a distinct form of evaluation. I think this is clearest to see in Pawson (2024), and it’s one of the most muddled sections of the Magenta Book.
  • The importance of theory for interpreting the meaning of covariates as well as merely selecting them. This is particularly important for intersectional understandings of selection into programmes, e.g., you need a theory to work out why, e.g., some combinations of gender and ethnicity, are more likely to be referred to a targetted programme. Is it driven by gender and/or racial stereotypes at time of selection, or broader societal discrimination before selection, for example?
  • An example from the Green Book (para 4.17) of a type of counterfactual reasoning, used when appraising different ways to realise a policy. I think the reasoning behind “deadweight” and “additionality” outcomes may use future less vivid conditionals. Still reading.

The Westminster Model and why it’s useful even though it’s wrong

This paper by Paul Cairney (2025) is fun.

The Westminster Model (WM) of policymaking:

“… political parties present manifestos to compete for public votes in general elections, the winner forms a government, and government ministers oversee the delivery of their manifesto by civil servants and public bodies.” (p. 297)

What’s wrong with the British Political Tradition (BPT), the UK flavour of the model:

“… politicians do not live up to the BPT; party manifestos combine some specific and many vague promises; parties don’t always gain a majority; not all MPs are easy to keep in line with government policy; ministers could not understand all of the choices made in their name; EU entry and exit was a bruising experience; and, the UK’s ability to reform devolved or subnational government is constrained in practice. In other words, the first aim of a ‘governance thesis’ is to use the WM as an ideal-type to compare with more accurate stories of the real world.” (p. 298)

Cairney then argues that “these differences between aspiration and reality provide the launching point for a more realistic story” – and goes on to have a go, e.g., noting the impact of the finite time and capacity of ministers, expanding on the range of individuals and informal and formal groups that influence policymaking

The approach reminds me of how classical logic and probability are used in the psychology of reasoning: they offer frameworks for understanding how people ought to reason, and then researchers adjust that framework to match how non‑mathematicians actually do reason, constrained by finite working memory, drawing on broader interpretations of terms like “if” than are used in logic, etc.

References

Cairney, P. (2025). Governance and the Westminster model: What exactly is the dominant story of UK policymaking? British Politics, 20(3), 295–315.

Theory-based evaluation as tautology

“Realist” evaluation and the more general “theory‑based” evaluation umbrella often present themselves as distinct forms of evaluation; however, their core assumptions that programmes work (or not) through underlying causal mechanisms that cannot be directly observed are restatements of what sciences already presuppose. It’s rare indeed for an evaluation not to begin with a theory of change; the issue is not whether that’s necessary but how good theories are. We’ve all seen boxology lists in the logic model style.

The full range of evaluation methods can test mechanisms rather than only whether an outcome has been shifted by a programme and current practice involves qual and quant evidence, e.g., the latter using mediation tests. The logic of “what works best for whom in what context” predates “realist” approaches by a few decades and is used across all methods. Similarly, there’s nothing uniquely “realist” about context-mechanism-outcome triads: it’s a useful way to move from boxology to theory prose.

Advocating “realist” or “theory‑based” evaluation is essentially advocating “scientific evaluation” – a kind of tautology. It may be helpful for some readers to learn about separating ontology from epistemology (the map is not the territory, to paraphrase Korzybski), but I’d suggest that’s best left to the first chapter of an introduction to evaluation rather than something that needs to be restated in every report. There are examples of fields that stick on the “science” term: neuroscience and cognitive science are two examples. But what would unscientific policy evaluation look like? What does it get called? (That’s a genuine question – I have a blog post brewing.)

Theorising what evidence would test a theory

Policy evaluation involves speculating about what might be going on – a creative process – based on what we currently (think we) know, and then working out what evidence we would expect to see if we were correct or if plausible alternative theories were true. That logic is explicit in process tracing, but it is far more general than that. The oft‑cited Van Evera (1997) rules, such as the “smoking gun”, follow directly from Bayes’ rule. We cannot directly observe a programme’s mechanisms of change; many different theories will be compatible with both qualitative and quantitative evidence. That problem has a name, underdetermination, and it is a challenge across the sciences.

An interesting example of the gap between theory and evidence is how physicists detect gravitational waves produced by black holes merging. The Laser Interferometer Gravitational-wave Observatory (LIGO) does not observe the waves themselves; it measures how those waves distort spacetime by a miniscule amount, far smaller than a proton.

My favourite analogy comes from trying to crack encrypted texts. We can systematically analyse letter and word frequencies in ciphertexts to spot patterns, but that only takes us so far (try the exercises in Rubinstein‑Salzedo, 2018). It helps to guess what people might be trying to say to each other based on something beyond the ciphertext; for example, that they open with “How are you?” or that they are likely to be discussing a particular event. These knowledge‑grounded guesses, drawing on a wealth of experience and prior evidence, help reduce the search space of possible encryptions. Something similar is going on in evaluation when we iterate between programme theories, tests of those theories, and exploratory research to generate ideas for new theories.

References

Rubinstein‑Salzedo, S. (2018). Cryptography. Springer.

Van Evera, S. (1997). Guide to methods for students of political science. Cornell University Press.

Interventions to reduce social media use

Goldfield et al.’s (2026) RCT looks interesting – the intervention was reducing social media use to max an hour a day. They found a 0.11 reduction in loneliness versus control on the UCLA Loneliness Scale. SD at endline was 0.65, so standardised mean difference (SMD) = 0.17.

A systematic review (Burnell, 2025) found an SMD of 0.12 (95% CI -0.02 to 0.26) for loneliness, 0.16 (CI = 0.04 to 0.28) when they screened out one study judged to have issues, leaving 13. (Haven’t waded into the details of each intervention or what those issues are.)

References

Burnell, K., Meter, D. J., Andrade, F. C., Slocum, A. N., & George, M. J. (2025). The effects of social media restriction: Meta-analytic evidence from randomized controlled trials. SSM – Mental Health, 7, 100459.

Goldfield, G. S., Lopes, M. V. V., Mahboob, W., Perry, S., & Davis, C. (2026). Reducing social media use decreases loneliness regardless of gender or level of social comparisons in youth with anxiety and depression: A randomized controlled trial. Journal of Affective Disorders, in press.

Explaining evaluation to the public

If you have ever introduced yourself to someone as an “evaluator”, you will quickly discover that very few people outside the field have the first clue what that involves.

The term is incredibly broad. Am I evaluating your dress sense or how you dance? Am I running psychometric assessments to evaluate staff performance? We are all evaluators, regardless of how often we share the judgements.

One of the functions of the UK Evaluation Society is to explain to the public what we do. So, how can we do this?

I think there are two main communication challenges for evaluation. The first challenge is finding a concise term that signals what evaluation involves to people who do not work in the field. It seems likely the easiest way to do this is simply to add a word or two to “evaluation”, so the problem is: which words? The second challenge is to find ways to explain what evaluation involves in practice, by using clear, straightforward language that can also ideally be applied more consistently in evaluation reports.

Words for evaluation – the case for social policy evaluation

Given how baffled people tend to look when I call myself an evaluator, I’ve previously said I’m a social researcher and work in social policy, and then just explained what that means through examples. Since October, I’ve tried “policy evaluator” or “social policy evaluator” and they seem to fare surprisingly well. The most obvious alternative to these might have been “programme evaluator”, but my problem with this is that I think programmes are more obviously associated with software or what’s on TV.

One possible objection to the term policy evaluation is that policies are rather broad, e.g., the Green Book (p. 6) says a policy is “a statement of intent that is implemented through a procedure or a protocol and a deliberate system of principles to guide decisions and achieve rational outcomes.” Evaluations often focus more specifically on programmes and service delivery. However, programmes put policies into practice, and evaluation findings should inform those policies. If a programme is shown to work, that has clear implications for policy. Plus, policies aren’t just found in government: schools, charities, and other organisations have them too. I think it’s important to show we engage in this broader context.

Explaining what we do

Policy evaluation is a broad field, covering many sectors, services and types of activity. It can look at whether public services, the work of charities, or even individual practitioners are making a meaningful difference through what they’re doing. It might tackle local programmes or national reforms. Evaluation also uses the full range of methods, mixing qualitative and quantitative evidence, in large surveys and focused case studies. Examples that illustrate this variety of topics and methods are essential, as Sarah Mason highlighted in one of the few studies exploring how evaluators explain their work to the public.

Some of the terminology we use within evaluation contexts is unnecessarily opaque and misleading, which is unhelpful for understanding the role. A good example – a word I’ve used more often than I should – is “treatment”. We’re not administering drugs – we’re evaluating programmes like mentoring, approaches to teaching, or emotional support. The term “treatment” carries medical baggage, that doesn’t fit with social programmes. My preference would be to label study conditions after what’s actually being tested. For instance, if we were evaluating ACME Therapy, the conditions would be something like ACME Therapy and usual practice. Using medical language in the context of trials in schools or criminal justice can sound especially creepy – it reminds me of A Clockwork Orange.

Evaluating how we explain ourselves

As policy evaluators we should evaluate how well we communicate with the public. For example, participants could be provided with information about what evaluation is and how it is used, enabling them to take an informed, participatory role in co‑designing how evaluation itself is explained. We could then use surveys, following Sarah Mason’s example, to test the effectiveness of different ways of explaining evaluation at a larger scale.

Evaluation is a broad transdiscipline, working across a wide range of contexts, so it is unlikely that we will find a single way of explaining ourselves that works everywhere. However, a collection of different terms and explanations that work well in at least some contexts would be valuable in reconnecting our work with the people we hope to serve. This would also help to attract new policy evaluators to what is one of the most important and exciting social sciences.

Originally posted on the UKES blog (February 4, 2026).

Theory-ladenness

An interesting paper on theory-ladenness in research on black holes (Doboszewski & Elder, 2025). Relevant to debates in policy evaluation on theory-based evaluation too and the pervasive role of theory whether or not you name the TBE brand.

‘Philosophers have raised a range of worries about the ways in which observations are theory-laden. In this article, we are focusing on the theory-ladenness of measurement outcomes—the ways in which the “observations” derived from an experiment or measurement are affected by theoretical assumptions that are made in the course of the experiment. The greatest concern is that theory-ladenness may lead to a vicious circularity where the theoretical assumptions made by the experimenters—either in the physical design of the experiment or in subsequent inferences from the empirical data—make it more likely or even guarantee that the observation will confirm the theory being tested. This might occur if the hypothesis being tested is presupposed by the experimental methods being used. Franklin (2015, 156) provides the following toy example of this: if we use a mercury thermometer to test a hypothesis that substances expand as their temperature increases, this is theory-laden in a way that risks vicious circularity because the proper functioning of a mercury thermometer relies on the very hypothesis being tested.’ (pp. 829-830)

‘… the impossibility of controlled experiments on astrophysical targets (stars, galaxies, etc.) places constraints on how astronomers can investigate these systems. Simulations can help make up for the lack of controlled interventions by providing a proxy system into which one can intervene.’ (p. 830)

Doboszewski, J., & Elder, J. (2025). How Theory-Laden Are Observations of Black Holes? Philosophy of Science, 92(4), 827–849.

Preprint here.

Levers

“Change doesn’t happen in straight lines and is propagated (or not) by human actors with their own preferences and motivations. Delivery is disaggregated between departments, tiers of government, sectors, agencies, regulators and professions. The economy is not a thing but an aggregation of billions of decisions, each made on the basis of incentives, opportunities and desires. Citizens’ choices and relationships control far more of what happens in our country than any Prime Minister could ever dream. Culture eats levers for breakfast.”

– Polly Mackenzie (2025, 19 Dec). There aren’t any levers.