Nice explanations at Sealed Envelope:
Tag: policy evaluation
Extremes along an evaluation spectrum
Evaluation is often divided into two kinds: formative evaluation, which is conducted to support the improvement of a programme, and summative evaluation, which is conducted to make (often high-stakes) decisions about a programme, such as whether to expand its delivery so it reaches more people or discontinue it.
Huey Chen (1996) challenged this dichotomy. The issue is that formative evaluation can lead to summative conclusions, e.g., the decision not to continue with a fully-powered impact evaluation. Conversely, summative evaluation can lead to formative improvements that can be tested in the next summative trial.
A related distinction, better understood as a spectrum, concerns the stance evaluators adopt before setting off on an evaluation. The following two David Shrigley pieces may offer a way to distil the extremes at either end of this spectrum:
A illustrates puff-piece evaluation: it is compromised by conflict of interest and often conducted by those who devised or deliver the programme under evaluation. If an organisation exists to deliver a programme, then evaluators within that organisation are going to find it challenging to evaluate themselves out of a job. I have seen this kind in the wild, for example evaluations of apps conducted by the company providing them. Don’t mention AI.
B illustrates adversarial evaluation: it’s assumed from the outset that the programme has no impact or causes harm. These are usually independent evaluations. The examples that come to mind are evaluations of interventions that persist despite a long history of evidence that they cannot work and the aim of the evaluation is to finally make a compelling case that funding for them should cease.
Reality is usually more nuanced and evaluations land somewhere in between. Programmes sometimes work, but maybe not as well as we hoped or not for everyone we thought would benefit. Still, I think identifying your location on this spectrum before you begin an evaluation can help clarify what safeguards are needed to support its objectivity. What do you believe the findings will be and why?
The various replication crises expose how we mere mortals are individually biased. We achieve objectivity collectively, through social systems such as peer review and pre-registered protocols. If users of an evaluation, whether potential programme beneficiaries or policy teams, suspect unmanaged bias, then it will be difficult to persuade them to trust what you find.
References
Chen, H. (1996). Typology for Program Evaluation. Evaluation Practice, 17(2), 121–130.
Government Evaluation Conference 2025 – two quick thoughts
It was a privilege to attend the Government Evaluation Conference this week. One new mantra for me is going to be, ensure evaluations are robust and proportionate. An evaluation should be rigorous enough to learn about programmes and policies, and to rule out alternative explanations for change, but it shouldn’t place excessive burdens on service users or programme delivery teams.
A practical way to ensure proportionality is to be disciplined with evaluation questions. Each additional question increases the burden for everyone involved and raises the likelihood that answers will be superficial or inaccurate. The critical questions to ask are: What do you really (really) need to know, and how will the answers be used?
There were also interesting discussions about (what are currently being called) “test and learn” approaches, which reminded me of continuous quality improvement (CQI) – something that is (or at least was) widely practised in healthcare. The idea is to identify aspects of service delivery that can be implemented quickly, measured easily, and that are known (from previous evaluations) to contribute to important impacts downstream. However, while such rapid implementation and feedback loops can be helpful, there will always be a need for longer-term follow-up with large samples to confirm that change has scaled and that desired impacts are actually being achieved.
Key study design considerations in state policy evaluation
A new preprint by Elizabeth Stone and colleagues explores methods for evaluating state-level policies in the US, offering insights relevant to UK policy evaluation. It covers selection of a range of approaches based on, for example, how a policy varies across the country, whether there is staggered roll out, and the availability of baseline data.
References
Stone, E. M., Schuler, M. S., Stuart, E. A., Rubinstein, M., Griswold, M., Stein, B. D., & Griffin, B. A. (2025). Choosing an analytic approach: Key study design considerations in state policy evaluation.
RCTs can and do use sensible control groups
If there are effective treatments for a condition, then a placebo-controlled randomised trial would generally be unethical (see below for exceptions discussed in the literature). But RCTs do not need to use placebo control.
There is a common misconception that an RCT should have a “pure” control (whatever that means) and if an active control of some description is used, e.g., another treatment, then it becomes A/B testing. This is false. A/B testing is just a type of trial and the term is more commonly used in marketing, e.g., in studies of variations of a website design.
RCTs can and do have active controls. Education trials and psychological therapy trials often use usual practice controls. Once effective Covid vaccines had been found, trials investigated the comparative efficacy and safety of different vaccines.
Example discussions
“Often, trials are conducted to compare a new intervention against current practice. The new intervention might be a small change, or a set of small changes to current practice; or it could be a whole new approach which is proving to be successful in a different country or context, or that has sound theoretical backing.” (Haynes, Goldacre, & Torgerson, 2012, p. 20)
“An RCT is not necessarily a test between doing something and doing nothing. Many interventions might be expected to do better than nothing at all. Instead, trials can be used to establish which of a number of policy intervention options is best.” (Haynes, Goldacre, & Torgerson, 2012, p. 21)
“Placebo controls are clearly inappropriate for conditions in which delay or omission of available treatments would increase mortality or irreversible morbidity in the population to be studied. For conditions in which forgoing therapy imposes no important risk, however, the participation of patients in placebo-controlled trials seems appropriate and ethical, as long as patients are fully informed.” (Temple & Ellenberg, 2000, p. 460)
“In cases where an available treatment is known to prevent serious harm, such as death or irreversible morbidity in the study population, it is generally inappropriate to use a placebo control. There are occasional exceptions, however, such as cases in which standard therapy has toxicity so severe that many patients have refused to receive it.” (European Medicines Agency, 2001, p. 16)
“Many placebo-controlled trials are conducted as add-on trials, where all patients receive a specified standard therapy or therapy left to the choice of the treating physician or institution.” (European Medicines Agency, 2001, p. 16)
References
European Medicines Agency (2001). ICH Topic E 10: Choice of Control Group in Clinical Trials.
Haynes, L., Goldacre, B., & Torgerson, D. (2012). Test, learn, adapt: developing public policy with randomised controlled trials. Cabinet Office and Behavioural Insights Team.
Temple, R., & Ellenberg, S. S. (2000). Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 1: ethical and scientific issues. Annals of Internal Medicine, 133(6), 455-463.
Green Book review
Debated 2 April 2025 Westminster Hall. Hansard link.
Sarah Russell (Lab): “We believe that the review must be done with ambition and a willingness to challenge underlying customs and practices. In particular, it needs to learn from the successes and failures—largely failures, I would say—of previous Green Book reviews, in particular the 2020 review.”
“There is a consultancy industry around the Green Book, and that raises the cost of developing a successful business case. It also makes it very difficult for smaller local authorities to successfully put together a business case because the complexity of doing so is absolutely mind boggling. Frankly, the whole thing is just a bunch of piffle—we need to make this very simple and outcomes driven.”
Jeevun Sandher (Lab): “For too long, investment has been biased towards London and the south-east. Someone living in London gets £800 more infrastructure spending per year than the national average. That is because the cost-benefit analysis set out in the Green Book to evaluate projects has a hardwired London bias. First, the Green Book prices the benefits of projects in a way that benefits places with higher wages—namely, London. Secondly, it does not estimate the wider impact of investment on growth.”
Andrew Cooper (Lab): “The Green Book is part of a colourful spectrum of current Government guidance and standards documents, which alongside the Aqua Book, the Magenta Book, the Orange Book and 19 other publications and pieces of supplementary guidance forms the basis of how the Treasury appraises policies, programmes and projects. On top of that, several Departments have issued their own interpretation of the guidance as it applies to them, thankfully avoiding extending the colourful metaphors any further.
“That complexity—well over 1,000 pages of guidance—is at the root of the criticism of the small industry and almost mysticism around navigating the assessment process and of the poor outcomes to which it leads.”
“Local authorities tell me it is still common practice for response letters to open with sentences such as, “We note that the BCR for the proposed scheme is 1.8.” That belies the intent of the 2020 review. […] The review must end the arbitrary BCR thresholds across Government.”
“That’s a very nice mediation analysis you have there. It would be a shame if something happened to it.”
Great blog post, by the always insightful Julia Rohrer
200 years of diff-in-diffs
“The starting point of this paper was a 2 × 2 DiD design that researchers have been using for almost 200 years. The end point was a design with five treatment groups, 11 years of data, six covariates, three types of parallel trends assumptions, and four estimation techniques.”
Baker, A., Callaway, B., Cunningham, S., Goodman-Bacon, A., & Sant’Anna, P. H. C. (2025). Difference-in-Differences Designs: A Practitioner’s Guide (No. arXiv:2503.13323). arXiv. https://doi.org/10.48550/arXiv.2503.13323
Self-reported versus statistically inferred attribution
“A starting point for the QuIP [Qualitative Impact Protocol] is the premise that those who were intended to benefit from an intervention know a great deal about what has caused and affected changes in their (and their households’) lives in the recent past, and what has influenced their active decisions to start or stop doing certain activities. Relying on the narrative testimonies of intended beneficiaries removes the need for an independent counterfactual based on interviews with a control or comparison group. This is because comparisons between what happened and what would have happened otherwise are embedded in causal claims within the narrative – our thinking and language is laden with ways of doing this (e.g. through use of the conditional tense and the many ways we can answer ‘why’ questions). Collected with care, narrative accounts are full of explicit and latent counterfactuals; the task is to identify and interpret them. […] We refer to reliance on respondents to provide evidence of the causal chain puzzle themselves as self-reported attribution, and we distinguish it from statistically inferred attribution that generally relies on exposure variation, including comparing treatment and control groups.” (Copestake, Morsink, & Remnant, 2019, pp. 7–8.)
Copestake, J., Morsink, M., & Remnant, F. (2019). Attributing Development Impact. Practical Action Publishing Ltd. https://doi.org/10.3362/9781780447469
Testimony
I would conjecture* that qualitative testimony about the impact of social programmes is often highly sensitive (if a programme works, then people will tell you it works) but has very poor specificity (if it doesn’t work, people are unlikely to tell you so, e.g., not recognising alternative explanations of change). If this conjecture is true, then it follows that positive feedback about a programme doesn’t tell us much; however, negative feedback would be highly informative. This follows from Bayes’ rule.
Play around with the probabilities here.
Sensitivity is the probability that someone will tell you the programme works, if it does actually work.
Specificity is the probability that someone will tell you the programme doesn’t work, if it doesn’t actually work.
Prevalence is the probability that programmes of the type you are asking about actually work, e.g., 0.5 would mean for a given type of programme you think the probability it does work is the same as the probability it doesn’t.
Positive predictive value (PPV) is the probability that the programme actually does work if someone tells you it does.
Negative predictive value (NPV) is the probability that the programme doesn’t actually work if someone tells you it doesn’t.
* This conjecture is at the bottom of the evidence hierarchy, whatever the opposite of a gold standard is. Balsa wood standard?