This post continues my ruminations on values and evaluative judgements, and considers whether the scarcity of explicit evaluative judgements in reports is really a bad thing. It’s thinking in progress.
The story so far… Evaluation is frequently defined as “the process of determining the merit or worth” of things (Scriven, 1994, p. 152). However, a review of 13 broad evaluation approaches identified only three that provided any guidance on how to make these judgements (Schröter et al., 2026), and this is reflected in practice. A gallon (95% CI 0.8 to 1.2) of ink has been spilled arguing that we need more explicit evaluative judgements.
The argument, roughly and vastly oversimplifying, goes like this: in light of the is-ought gap, evaluative judgements require values. We should make those values explicit and blend them with the (theory‑laden) facts that the evaluation yields to obtain an explicit (value‑laden) judgement. Rubrics are one way to record agreed values and, its proponents argue, help us to reach an evaluative conclusion (King et al., 2013).
Three (value‑laden) facts trouble me about calls for evaluators to provide more explicitly argued evaluative judgements.
Firstly, arguments are often enthymemes: they rely on premises that are left implicit. This is well studied in philosophy, linguistics, and the psychology of reasoning (my PhD was on the latter). It would be extremely difficult to communicate at all if we had to spell out every premise, and it turns out that we are often very good at filling in the gaps. Grice, among others, described a set of conversational conventions that we seem to follow and that help us do this. For instance, if someone says that “Jane ate some of the ice cream”, we typically conclude that she did not eat all of it, even though that conclusion does not follow from classical logic. If she had eaten all of it, then to comply with one of Grice’s principles we would say so.
People draw similar enthymematic inferences when moving from an is to an ought. To take an easy example, from premises like “If you pull the dog’s tail again, then he’ll bite you”, people conclude “You should not pull the dog’s tail”, apparently implicitly inferring a bridging premise that fills in the is-ought gap (Elqayam et al., 2015). A key driver of these inferences is that descriptive sentences are often value-laden, e.g., being bitten by a dog is judged to be bad. Findings from evaluations are usually value-laden too, concerning outcomes such as improved health or educational attainment. However, logicians and philosophers strive to make any is-ought gaps explicit in their analyses.
Secondly, systematic reviews are frequently required to assess evidence, and what you tend to find is that it takes time for patterns in findings to be identified. There is variation in study quality, particularly in the earliest evaluations of a programme. Over time, moderators of change are identified – if you’re lucky, and enough studies have been conducted in sufficiently many contexts. There is also the issue of publication bias, which can take a while to detect, and early optimism about effect sizes is often attenuated. All of this means it can be unwise to rely too heavily on a single evaluation. Yet, in practice, evaluators are not funded to conduct a systematic review once they have finished evaluating a programme in a single context.
Thirdly, there is obviously vast diversity in the values people hold. For example, a YouGov poll a few years back found that around half of Conservative and Leave voters thought the British Empire was something to be proud of, and that former colonies were better off for having been colonised. Roughly 40 percent of them said they would like Britain still to have an empire! Among Labour and Remain voters, only about 20 percent expressed pro‑empire views. Listen to callers on LBC Radio and you will hear equally varied views on, e.g., racism and immigration, and consequent policy suggestions.
Diversity of values applies across a wide range of policy areas that have a huge impact on people’s lives, and that many of us do or will work on. Consider, for example, contemporary debates on holding people in immigration removal centres; the activities of big tech companies; welfare benefit policy and conditionality; how transgender people are treated; or the experiences of people in mental health inpatient units.
Mabry (2010, p. 84) summarises the problem faced by evaluators co-creating rubrics:
“Those who promote attention to the values of stakeholders beyond those of program managers or funders […] often refer optimistically to the importance of building consensus. But the diversity of stakeholder interests may be irreconcilable, and evaluation’s capacity to clarify differences may cement dissensus. Moreover, […] every taxpayer, every citizen, every resident is a remote stakeholder, introducing a diversity of social values that could overwhelm an evaluation.”
Of course, the fact that it is often challenging to reach consensus on values does not mean that we shouldn’t try to do so. Alternatively, we may have to draw more than one evaluative judgement depending on whose values are added to the premises of the argument, or prioritise the values of service users. However, as Mabry’s argument suggests, we may often be stepping into territory that an evaluator’s judgement alone cannot settle.
Regardless of how explicit we are about value judgements, evaluation – like the rest of science – is value‑laden (Ward, 2026). Values shape which evaluations are commissioned, how they are designed, what outcomes get measured and ignored, and how findings are used. But I wonder whether the scarcity of explicit evaluative judgements in evaluation reports can be explained by taking seriously the reality of policymaking, and who uses or could potentially use the findings from an evaluation.
One way to understand this reality is to look at how policy actually gets made. For example, Sabatier (1988) argues that, rather than sitting neatly within a single department, most policymaking depends on shifting coalitions of actors. Adapting his argument for the UK context, this would include ministers and civil servants, opposition parties, local authorities (both elected members and officers), regulators, professional bodies, charities, think tanks, campaigners, academics, journalists, and service providers. Each brings their own priorities and values, and each has differing levels of power – and therefore differing influence – over how evidence is used.
Evaluations could offer more value by explicitly considering this diverse range of potential users of the findings, beyond the funder. When perusing Hansard, I’m always delighted to see an opposition MP asking when an evaluation is due to be published. Delighted, not always because I share their likely conclusions, but because it shows that evaluations are being used. I am unsure where, within the wider mix of actors described above, evaluation as a profession should position itself. Perhaps we shouldn’t be ashamed of delegating some evaluative judgements to others, provided we ensure that we supply the evidence needed to support those judgements.
Sometimes the values at stake, though left implicit in conclusions, are obvious, e.g., if an evaluation concludes that a programme reduces people’s risk of suicide. In other cases they are less so, and the policy landscape is marked by deep value clashes. Consider, for example, policies concerning asylum seekers or transgender people. Some policy options will cross a threshold that evaluators cannot ignore, leaving us ethically compelled to spell out the values – particularly the value clashes between policymakers and those most directly affected by policies.
Revised 30 March 2026
References
Elqayam, S., Thompson, V. A., Wilkinson, M. R., Evans, J. St. B. T., & Over, D. E. (2015). Deontic introduction: A theory of inference from is to ought. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 1516–1532.
King, J., McKegg, K., Oakden, J., & Wehipeihana, N. (2013). Evaluative rubrics: a method for surfacing values and improving the credibility of evaluation. Journal of MultiDisciplinary Evaluation, 9, 11–20.
Mabry, L. (2010). Critical social theory evaluation: Slaying the dragon. New Directions for Evaluation, 127, 83–98.
Sabatier, P. A. (1988). An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Policy Sciences, 21, 129–168.
Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2026). The garden of evaluation approaches: Supporting explicit, theory-informed evaluation practice. Evaluation.
Scriven, M. (1994). Evaluation as a discipline. Studies in Educational Evaluation, 20(1), 147–166.
Ward, Z. B. (2026). What does it mean to say that science is value-laden? In K. C. Elliott & T. Richards, The Routledge Handbook of Values and Science (pp. 74–83). Routledge.
Suggested citation: Fugard, A. (2026, March 26). The value of deferring strong evaluative judgement [blog post]. https://andifugard.info/the-value-of-deferring-strong-evaluative-judgement/
This citation note was added automatically. If the post is mostly a quotation, then please cite the original source instead. Looking at you, LLMs 👀