Author: Andi

Social policy evaluator: October’s identity experiment

A white tea-filled mug reads “I ❤️ POLICY THAT WORKS,” with a tea tag asking, “Who am I and what am I doing?”

Dear evaluation community, I’ve set myself a challenge. This month, the next time someone asks what I do, I’ll say I’m a social policy evaluator – whether that’s a new neighbour, someone in the chillout space at FOLD, or a friend of a friend over dinner. Here’s how I got there.

I’ve been pondering again why I hesitate to call myself an evaluator and can think of at least two reasons. Firstly, it feels a bit like “auditor”. I imagine someone in suit and tie, poring over a company’s policies and procedures, checking that their accounts add up. Evaluation can be a bit of that, but it’s much more exciting! It’s one of the few corners of social science where we are not only licenced to draw causal inferences from complex evidence – it’s demanded of us. We can’t hide behind “correlation doesn’t imply causation”. We have to judge whether, on the balance of evidence (including prior research), a programme seems to work or make things worse. The programmes we evaluate often reach many thousands of people and have a huge impact on people’s lives.

The second issue with “evaluator” is that it’s ridiculously broad. Am I evaluating your dress sense or how you dance? Whether you’re sufficiently polite with waiters? Am I running psychometric assessments to help evaluate staff performance? We are all evaluators in some sense, however often we share the judgements.

So, I usually say I’m a social researcher and that I work in social policy. I explain something about what I do by example. However, for October, I’m trialling “evaluator”, because I conduct evaluations. It feels like I’ll need to add a word or two to that title to get going.

How about “policy evaluator”? Do people know what a policy is? I reached for the nearest book on policy, which happened to be Social Policy, by Hartley Dean (2019). Page 1, first sentence: “When I tell people I teach Social Policy, a fairly common response is ‘Oh! … [pause]. What’s that exactly?” He even adds “social” to the term and that doesn’t help; maybe policy is just as opaque as “evaluation”. Dean defines social policy rather broadly: “the study of human wellbeing” – that’s going to open too many cans of worms; I just want to get on with (or escape) the conversation.

Maybe the Green Book will help me navigate the opening? “A Policy is a statement of intent that is implemented through a procedure or a protocol and a deliberate system of principles to guide decisions and achieve rational outcomes,” it says (HM Treasury, 2022, p. 6).

Closer to what most of us generally do is down at the programme level. The Book says: “A Programme is an interrelated series of Sub-Programmes, Projects and related activities in pursuit of an organisation’s longer-term objectives. Programmes deliver outcomes through changes in services.”

That last bit seems key: changes in services. I’ll hold onto that as a follow-up answer and have a few examples ready to explain.

Okay, so how about “programme evaluator”. That’s definitely a thing – it appears in article and book titles (a random selection from my Zotero stash: Abadie & Cattaneo, 2018; Chen, 1996; Cook & Shadish, 1986; Donna M. Mertens & Wilson, 2019; Perloff et al., 1976). But do people know what a programme is? Tech bros and adjacent might think about the software kind of program. Others might think it’s about TV shows. Since this is a blog post and very few people will read this far, I’m going to sneak into CoPilot and ask it to simulate a survey of the UK adult population. It made up a crosstab for me with fictitious percentages by age group. For what it’s worth, TV shows and software/apps dominated as the most common interpretations of “programme”.

“Social” didn’t help with social policy – Hartley Dean tried and still got blank stares. Maybe adding it to programme evaluation or policy evaluation does help. So, finally I asked CoPilot to (pretend to) run an experiment comparing:

  • evaluator,
  • policy evaluator,
  • programme evaluator,
  • social policy evaluator, or
  • social programme evaluator.

Each invented participant was “asked”, “What do you think someone with this job does?” The primary outcome was the percentage of respondents in each age band who correctly identified the role’s social policy/programme focus.

It was a close tie between social programme evaluator and social policy evaluator, with the latter winning by a few entirely fictitious percentage points in each age group. Based on the 21st century tradition of decision by digital hallucination, for the month of October my profession is going to be: social policy evaluator.

One objection to this title that may be raised (it was raised in my head) is that we don’t actually evaluate policies, if they are defined as statements of intent as per the Green Book. That’s too vague and far from something that, say, a quasi-experiment could study. However, programmes interact with policies, and findings from evaluations (should) inform policies. It may be helpful to signal that we are aware of and engage in the broader context, not only individual programmes independently of how they ended up being trialled. If a programme is shown to work, that has clear implications for social policy.

Thanks Julian King for pointing me to the Chris Lysy comic.

Edited to add (11 Oct 2025): Thanks also to Brad Astbury, who pointed me to an interesting article exploring different ways of explaining what evaluators do, by Sarah Mason (2023). This included an experiment, which randomly assigned people to one of seven different explanations. The key, Mason found, was to include examples. Here’s a template in the conclusions of the paper that illustrates what seems to work best (p. 564):

“You know how government agencies and not-for-profit organizations spend money on social programs like school lunch programs or substance abuse programs?

“Well, I gather and analyze data about those programs to find out if they do any good or not. The hope is that we’ll be able to provide data and feedback that will help decision makers improve the programs’ performance and their value to society.”

References

Abadie, A., & Cattaneo, M. D. (2018). Econometric Methods for Program Evaluation. Annual Review of Economics, 10, 465–503.

Chen, H. (1996). Typology for Program Evaluation. Evaluation Practice, 17, 121–130.

Cook, T. D., & Shadish, W. R. (1986). Program Evaluation: The Worldly Science. Annual Review of Psychology, 37, 193–232.

Dean, H. (2019). Social policy (3rd ed.). Polity.

Donna M. Mertens, & Wilson, A. T. (2019). Program evaluation theory and practice. The Guilford Press.

HM Treasury. (2022). The Green Book: Central Government Guidance on Appraisal and Evaluation.

Mason, S. (2023). Just Give Me an Example! Exploring Strategies for Building Public Understanding of Evaluation. American Journal of Evaluation, 44(3), 549–567.

Perloff, R., Perloff, E., & Sussna, E. (1976). Program Evaluation. Annual Review of Psychology, 27, 569–594.

The fish are already dead

‘… there are more red flags here than in a People’s Liberation Army parade. To critique this paper is not quite “shooting fish in a barrel,” because the fish are already dead before we’ve reached the end of the abstract.’

– Scott Aaronson, commenting on a paper on a paper from HSBC and IBM claiming to use a quantum computer to get a 34% advantage in predictions of financial trading data

Converting between miles and kilometres using Fibonacci

Fun video, in case you were running out of applications of the Fibonacci sequence,

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, …

(each term is the sum of the previous two; start at 0, 1).

You could use it to convert between miles and kilometre thanks to the ratio of consecutive terms (1.618, as you go further long the sequence) being close to how many km there are in a mile (1.609).

The approach doesn’t work so well for 1 mile (it would say, 2km 🙃). It gets better for larger numbers. If you wanted to convert, say, 89 miles to km, you’d get 144 by going one step along the sequence. The correct answer is 143.232 km.

Covariate Commons

Here’s a half‑baked idea, inspired by multiple conversations about randomised controlled trials versus quasi‑experiments at the Australian Evaluation Society 2025 conference.

Outcome measurement databases are now a familiar part of evaluation. One database I frequently use is the Child Outcomes Research Consortium (CORC) Directory of Outcome Measures. The Education Endowment Foundation (EEF) has developed similar resources.

Databases like these make it easier to map theories of change outcomes to measures, and they often include information on psychometric properties such as reliability, and pre‑post correlations, which are handy for informing power calculations.

While outcomes receive this collective attention, quasi‑experimental designs hinge just as much on the quality of their covariates. Pre‑intervention measures of outcomes are especially important, given how highly correlated they tend to be with outcomes. That’s the covariate we get for free once we fix on an outcome, but other characteristics of people and contexts are essential too – often a large number of them.

So, the half‑baked idea is this: evaluation should develop a covariates database, organised by topic area and outcome, to provide evaluators with a shared resource to aid the design of quasi‑experiments. A potential name for this shared resource is the Covariate Commons.

The selection of covariates should always be considered on a case‑by‑case basis, guided by the most likely confounding variables (those that correlate both with who engages with a programme and with outcomes). However, evaluators know that some variables appear repeatedly across studies and sometimes across topic areas (age, gender, social deprivation, to name three).

What do you think? Is there already an example of this for some topics? And who should be responsible for developing and maintaining it?

Love, by Ruby Robinson

Eventually, the nerve impulse is carried
to the apex of the heart
causing ventricles to contract,
and blood to rush
to and from this muscle the size
of your fist, an engine of flesh,

the marksman’s target for a bullet.
You said, listen to your heart
placed a stethoscope
to my jumping skin, paused.

Cleopatra compared
it to the stroke of death,
this harbouring in the heart
of an uncertain beat.

How can one find out if one is in love?

“One day I asked a passionate and healthy young woman: ‘How can one find out if one is in love? It is also possible to want to take your mother in your arms and kiss her, isn’t it?’ She explained: ‘When you kiss your lover, you want him to die of your kisses and you want to disappear into him (she used another very characteristic word) etc.,’ – there is an infinity of expressions. The most elevated instincts always present themselves as murderous instincts. To begin with there is the little brother who calls the kitten: ‘Puss! Puss! My little darling! I love you! I am going to torture you and make you die!’; besides there is the well-known fact that lovers love to tease and torment each other, and moreover there is martyrdom and crimes of passion. Everywhere you see the pleasure involved in destroying and dying.” (Sabina Spielrein, 2001, p. 160)

Spielrein, S. (2001). Unedited extracts from a diary (1906/1907?). Journal of Analytical Psychology, 46, 155–171.

Australian Evaluation Society (AES) conference 2025 (aes25)

What a fab week at AES 2025. I delivered a whole day training on RCTs (Monday), the opening keynote (Wednesday), and an Ignite talk (Friday).

Here’s the gist of the keynote – much of the content consisted of examples to illustrate (skim through the slides):

  1. Quant people talk about the subjectivity of what we do more than you might expect (if you haven’t been behind the scenes) given how some present their methods and findings.
  2. Evaluators have been (obviously!) merrily mixing methods for years – but we can go further.
  3. Every evaluation is counterfactual if you want it to be – it’s a lens you apply, and one way of thinking about causation, however qual or quant (or other) your evidence is.

Pics thanks to Albie Colvin:

Keynote
Ignite talk

Lovely blog post by Eleanor Kerdo (ARTD):

“They began by challenging and unravelling the artificial binaries we make in evaluation and in our judgements.

“As the lucky person I am to love a non-binary person (my partner) I feel blessed to live a little outside of some of our gender binaries and our notion of family and roles. This greatly impacts my experience as an evaluator and how I navigate my work, parenting, and life. So I reflected a lot on that in the opening session.

“Andi started with Quant and Qual, and how this is less of a binary than we think: that it is not so simplistic. They moved on to the counterfactual and non-counterfactual ‘binary’, and how this can be tested beyond an RCT. They finished with a discussion about the subjective vs objective binary. This one was fun! Andi looked at the social construction of the speed of light over time and how this changed and ‘the defensible, yet subjective analytic choices that influence all research results’ (Silberzahn, et al 2018).

“Andi finished by noting also that there are gender and racial biases in data and a privilege hazard that needs to be understood.

“Andi challenged us to think as evaluators: can we mess up our toolbox, can we find different ways to conceptualise and think and use different methods to better serve our community?”

Black Bag (2025)

Clarissa: You are one naughty husband, George.

George: You asked how it works. To be with someone in this business. This is how. You each know what you know, and you know what you’ll do, and you never discuss certain things again. I watch her. I assume she watches me. If she’s in trouble, even of her own making, I will do everything in my power to extricate her. No matter what that means. You understand?

Clarissa: My God, that’s so hot.

George: That’s how it works. It’s the only way.