INSPECT-SR (Wilkinson et al., 2026) is a new checklist for assessing the trustworthiness of RCTs in systematic reviews. It consists of 21 items across four domains to help reviewers decide whether to include a trial in a review.
Wilkinson, J., Bero, L., Heal, C., Flemyng, E., Antoniou, G. A., Aburrow, T., Alfirevic, Z., Avenell, A., Barbour, V., Berghella, V., Bishop, D. V. M., Bordewijk, E. M., Brown, N. J. L., Christopher, J., Clarke, M., Dahly, D., Dennis, J., Dicker, P., Dumville, J., … Kirkham, J. J. (2026). INSPECT-SR tool for assessing trustworthiness of randomised controlled trials. BMJ, 394, e100611.
Goldfield et al.’s (2026) RCT looks interesting – the intervention was reducing social media use to max an hour a day. They found a 0.11 reduction in loneliness versus control on the UCLA Loneliness Scale. SD at endline was 0.65, so standardised mean difference (SMD) = 0.17.
A systematic review (Burnell, 2025) found an SMD of 0.12 (95% CI -0.02 to 0.26) for loneliness, 0.16 (CI = 0.04 to 0.28) when they screened out one study judged to have issues, leaving 13. (Haven’t waded into the details of each intervention or what those issues are.)
Social policy evaluation inherits concepts from multiple disciplines, and somewhere along the way many of them have become muddled. The list below sketches what I think are seven of the most harmful myths.
1. “Counterfactual” is a synonym for control/comparison group
There is a long history of research on counterfactual reasoning in the absence of a control group. For example, analyses of how people should and can determine the truth of a counterfactual conditional such as “If Oswald had not killed Kennedy, no one else would have” or “If I’d left the house earlier, I’d have caught that bus” (e.g., Adams, 1970; Halpern, 2015). We ponder counterfactuals like these all the time without running RCTs.
The evaluation literature has also considered how counterfactuals can be used without a comparison group (e.g., White, 2010) – including with qualitative evidence (e.g., Reichardt, 2022). In evaluation, the counterfactuals are conditionals like, “If intervention group students hadn’t been offered the mentoring programme, x% fewer would have passed their GCSEs.”
Some evidence for counterfactual outcomes is more robust than others. One example analysed by Reichardt (2022) involves asking programme participants what they think would have happened if they had not taken part in the programme. Their ability to answer this depends on their capacity for counterfactual reasoning, which may be limited for some programmes. For instance, many people believe that homeopathy is effective, even though its remedies are so highly diluted that they contain no active ingredients and cannot have any biological effect (Ernst, 2005). Counterfactual reasoning based on such beliefs could therefore falsely conclude that a programme caused improvements that would have occurred anyway.
Quantitative designs can be used to estimate counterfactual outcomes without a comparison group. One example is interrupted time series, which uses trends in pre-intervention outcomes to estimate post-intervention counterfactual outcomes. This is illustrated below:
The standard analysis estimates whether introducing the intervention shifted the outcome and whether it changed the slope, both relative to the inferred counterfactual outcomes.
When a comparison group is used to estimate counterfactual outcomes, it is a factual group – it consists of real data from real people. If it were a counterfactual group, it couldn’t be used to estimate anything. This is why I avoid the term “counterfactual group”.
2. RCTs only estimate overall average effects
If you think causal effects may vary by, e.g., participant characteristics or the context they are in, then you can test this by using moderator analysis and obtain estimates for each group. For example, trials funded by the Education Endowment Foundation usually investigate whether the magnitude of a programme’s effects differ for students who are eligible for free school meals – a proxy for socioeconomic disadvantage (e.g., Takala, et al., 2025). Westhorp and Feeny (2024) suggest using moderator analysis to test the effect of context in realist evaluations (see also Myth 4 below).
One challenge is that when testing for moderator effects for variables with a large number of levels – for example, in intersectional analyses of gender and ethnicity – models can become unwieldy very quickly. The UK census, for example, includes 19 ethnicities and 6 genders (including cis and trans identities). That results in 114 possible combinations. A promising solution is multilevel analysis of individual heterogeneity and discriminatory accuracy – MAIHDA, for short (Merlo, 2018).
3. (Trialists believe that) RCTs don’t need a programme theory
Individual causal effects are defined as the difference between what a person’s outcome would be following intervention and what it would be following control. We can’t observe this within‑person difference because each participant only experiences one condition. With random assignment, however, the average outcome of those in the intervention group minus the average outcome of those in the control group (a between‑person difference) provides an unbiased estimate of the average of individual causal effects (averages of the unmeasureable within-person differences) – even without any covariates. That’s the great thing about RCTs.
RCTs still need theory. How do you choose a control group? Outcome variables? How do you interpret estimated causal effects, i.e., what the difference between intervention and control outcomes means? Chen and Rossi (1980) explain how theory-based (what they call theory-driven) RCTs that include theory-informed covariates yield more precise estimates of effects (reducing the probability of Type II error), even though those covariates are not needed to control Type I error. That’s why sample size calculators ask how much variance in outcomes is thought to be explained by covariates. RCTs can also test mechanisms of change, e.g., using mediation analysis or through factorial designs.
A related myth is that trialists believe reality consists of variables. Paley and Lilford (2011, pp. 956-7) dispute this:
The view attributed to positivism, that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.
Good theories of change go beyond the variables to discuss people and other entities and what they do (see Myth 6). Variables operationalise aspects of these.
Incidentally, trialists don’t need to be positivists either. Bogen and Woodward (1988) provide an accessible discussion of how to think about the relationship between theory and observed phenomena without falling into positivism.
4. Realist evaluation is a distinct type of evaluation
What people do in practice when they say they are conducting a realist evaluation or “using realist principles” often makes sense, and Pawson and Tilley’s work has helped bring theory to the fore in evaluation (I put at least one reading of theirs on my social theorising course when I was an academic). But the idea that you need a realist evaluation to find out “what works best for whom in what context” or to describe change in terms of context, mechanisms, and outcomes, is just absurd. Many evaluation approaches do this.
Paul (1967, p. 111) offers an earlier example of the what works for whom logic:
[…] the question towards which all outcome research should ultimately be directed is the following: What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances?
Pawson and Tilley (1997, p. 10) cite Palmer (1975, p. 150), who uses similar logic:
Rather than ask, “What works for offenders as a whole?” we must increasingly ask “Which methods work best for which types of offenders, and under what conditions or in what types of setting ?”
School climate research offers one example of contextual factors: the physical characteristics of schools; the formal and informal rules that operate within them; and the norms, beliefs, and values at play (Anderson, 1982). RCTs have examined the extent to which school climate moderates programme effects – i.e., how outcomes depend on the settings in which programmes are delivered (e.g., Low & Van Ryzin, 2014) – as well as the mediating role of climate, where a programme changes aspects of school climate and these changes help to explain its effects (e.g., Singla et al., 2021).
My reading of Pawson and Tilley (1997) and Pawson (2024) is that they’re introducing the scientific method and philosophy of science to evaluators who don’t have a formal training in a science. The ideas apply to any form of evaluation. For example, Pawson (2024, p. 42) argues that
All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.
Science is not simply about observing regularities or estimating average causal effects. It is about explaining why observed patterns occur, how they are produced, and under what conditions they hold. This logic applies whether you are running an RCT, conducting a qualitative impact evaluation, or analysing administrative data using a quasi-experiment – and whether or not you call what you’re doing “realist evaluation”.
The “realist” in the term has a meaning from philosophy: ontological realism. Many people are realists without carrying a realist evaluator card.
5. “Theory-based evaluation” excludes trials and quasi-experiments
The original conceptualisations of theory-based evaluation, or its many synonyms (e.g., theory-driven evaluation, program theory evaluation, program theory–driven evaluation science) includes any approach that begins with a theory of change, uses it to design an evaluation that tests the theory, and revises the theory in light of findings (see, e.g., Chen & Rossi, 1983; Cook, 2000; Fitz-Gibbon & Morris, 1975; Weiss, 1972).
Quasi‑experiments are particularly dependent on programme theory. For example, studies that construct a comparison group using matching or weighting rely on identifying all important covariates. Once those covariates are specified, we can check whether matching/weighting has made the intervention and comparison groups equivalent at baseline; however, balance checks cannot tell us whether any important covariates were left out – theory is needed for that.
Additionally, a large number of causal models will be consistent with the data, so you need a theory to choose between them. For example, given only data, the following six simple causal models are statistically equivalent. Each model has three variables: outcome (out); condition, e.g., the programme being evaluated or usual practice (prog); and mediator (med).
Model 1 is the intended causal interpretation: the effect of the programme is partially explained by the mediator. Model 2, for example, says that programme and outcome are statistically associated with each other because the supposed mediator is actually a common cause.
The Magenta Book recommends using theory-based evaluation if you can’t find a comparison group (HM Treasury, 2020, p. 47). Annex A (analytical methods for use within an evaluation) locates trials and quasi-experiments outside the theory-based list. I think the Magenta Book is partly responsible for perpetuating Myth 5 in the UK – particularly through invitations to tender that follow the book, which leave evaluators having to play a semantic game to win work. I hope the upcoming revision corrects this (update: it doesn’t), and that it becomes routine practice to commission theory-based trials, quasi-experiments, and mixed methods evaluations, alongside theory-based qualitative impact evaluations and small-n case studies.
6. Theories of change are a kind of diagram
Theories of change are theories – of change. They may be illustrated using a diagram, but there needs to be an explanation, usually using prose, describing how the resources, entities (e.g., people, organisations, laws, etc.), and activities carried out lead to outputs and outcomes. Currently entrenched practices of developing a Theory of Change™ seem to lead people to forget what they learned about theory and theorising at university.
Pawson and Tilley (1997, p. 134) provide neat examples of how to organise theories of change explanations in tables. Tables like these are part of the “realist evaluation” tradition; however, the logic is clearly much more general (see Myth 4):
Context
Mechanism
Outcome
High numbers of prepayment meters, with a high proportion of burglaries involving cash from meters
Removal of cash meters reduces incentive to burgle by decreasing actual or perceived rewards
Reduction in percentage of burglaries involving meter breakage; reduced risk of burglary at dwellings where meters are removed; reduced burglary rate overall
If a client forces you to include only a diagram in a report, plead with them to let you add the actual theory to an Appendix. We’ve been in this situation many times and used this solution once to date – let’s see if the report is published (update: it wasn’t).
7. Evaluations test “treatments”
Well, in a formally defined sense they do; however, not as people usually use the term. We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, emotional support. The term “treatment” carries with it medical baggage, which is very different to social programmes. My preference would be to use names that are as close as possible to the conditions. For instance, if we were evaluating ACME Therapy, the conditions would be something like ACME Therapy and usual practice or ACME Therapy and CBT.
The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.
Dosage is typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. People aren’t absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.
Using medical language in the context of trials in schools or criminal justice sounds especially creepy – it reminds me of A Clockwork Orange. I have been guilty of using these terms too. Sometimes it’s just too difficult to challenge medicalised tradition when there are pressing deadlines, and causal estimands like average treatment effect on the treated are pervasive in the methods literature (e.g., Cunningham, 2021, Section 4.1.2).
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In A. Petrosino, P. J. Rogers, T. A. Huebner, & T. A. Hacsi (Eds.), New directions in evaluation: Program Theory in Evaluation: Challenges and Opportunities (pp. 27–34). Jossey-Bass.
Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.
If you take 30 draws from a Bernoulli distribution, e.g., to randomise people to one of two conditions, there’s a \(\frac{2}{2^{30}} = \frac{1}{2^{29}} = \frac{1}{536,870,912}\) chance that all 30 will be assigned to the same condition (all 1s or all 0s).
That’s vanishingly unlikely, but using the following seed with R’s rbinom produces 30 ones:
Unlikely but possible – even if your random numbers come from atmospheric noise or quantum measurements rather than a pseudorandom generator. If you want to rule out this outcome entirely (make the probability zero), then you don’t actually want independent Bernoulli draws. You’d need a different randomisation scheme, such as blocked randomisation.
I often see reports describing checks on random sequences after they’ve been generated, when the researchers could have saved themselves the effort by building in the kind of randomness they wanted from the start.
P.S. The longest initial runs I know about are for seed 796110494, which yields 31 zeros, and 1002755089, which yields 32 ones.
Dear fellow trialists, can we please stop using the term treatment when it’s not a medical treatment? We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, and emotional support. Using medical language in the context of schools or criminal justice sounds especially creepy. I think we should also rename estimands like average treatment effect. How about average causal effect, which is sometimes used too? Language matters, especially when we’re trying to build trust with communities who are often suspicious of randomised experiments. If we want our work to be understood and welcomed outside quantitative impact evaluation, we need to be more careful with the metaphors we borrow.
The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.
Dosage is another common term that feels weird. It’s typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. But again, it carries pharmacological baggage. People aren’t just absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.
I’ve used all these terms too as they are so pervasive in the literature – this isn’t me being holier-than-thou. But if we want trials to be accessible and respectful, we need to rethink the language we use. It’s no wonder people are put off trials when the basic terminology sounds so clinical and dehumanising. Careful use of language would make it clearer what we are actually doing and how crucial a role participants play in the programmes we’re evaluating.
“… random allocation in a clinical trial means that all patients have the same chance of receiving any particular treatment (and in most cases each treatment is equally likely). Patients should be entered into a trial before their allocation to a particular treatment is known. A common misconception exists that allocation based on, for example, odd or even dates of birth or hospital numbers is random. These systematic allocation methods, however, clearly violate the requirement that all patients have the same chance of receiving each treatment. Alternate allocation does not in principle suffer from such problems, but there is a risk of abuse because the investigator’s knowledge of the next treatment may lead to some patients being excluded from the trial – making this method inadvisable.”
If you’re running an RCT, why bother balancing covariates between treatment and control, e.g., using blocking or minimisation?
Doing so doesn’t make it any more likely that your confidence interval will include the true treatment effect – if it did, then what about the thousands of unmeasured covariates, which randomisation is supposed to address…?
Instead, balancing covariates makes inference more efficient when you statistically adjust for those measured covariates – as you will want to do if they’re correlated with the outcome variable.
Here’s Senn (1994, p. 1721) – a paper I seem to cite every few weeks:
“… what is the value of balance as regards validity of an inference? Answer: none. This does not mean that balance is not useful. Its use is that, having decided to condition, we minimize the standard errors of our treatment effects if our covariate is orthogonal to treatment. This is the true value of balance which is an issue of efficiency, not validity.”
So it’s regression analysis 101: if you have an imbalanced covariate, then it is correlated with your treatment variable. The greater the correlation, the greater the standard error of the treatment effect estimate. Though this will be mitigated if that covariate is also highly correlated with the outcome, which will reduce the standard error.
Pinference: Probability Inference for Propositional Logic. “Implementation of T. Hailperin’s procedure to calculate lower and upper bounds of the probability for a propositional-logic expression, given equality and inequality constraints on the probabilities for other expressions. Truth-valuation is included as a special case. Applications range from decision-making and probabilistic reasoning, to pedagogical for probability and logic courses.”
RobinCar2: ROBust INference for Covariate Adjustment in Randomized Clinical Trials. “Performs robust estimation and inference when using covariate adjustment and/or covariate-adaptive randomization in randomized controlled trials.” Though see this post.
I have some lived-experience to report and a brief reflection on an RCT. Content warning: it involves a brief description of an injury, but I lived happily ever after!
I recently dislocated my shoulder after slipping on a traverse climbing wall – ten minutes before the end of an introductory bouldering session. Thankfully, I wasn’t far off the ground. As far as these things go, it was a “good” dislocation: I feel fine and am starting physio imminently.
It wasn’t my first. Over a decade ago, I had a dislocation and was given nitrous oxide (gas and air) at A&E while they reattached my arm. This time, I was handed Penthrox – also known as the green whistle – a vape-sized handheld device that delivers methoxyflurane.
Subjectively, Penthrox felt much better than gas and air. During the procedure, I’d describe my mental state as “relaxed, pain-free, and a bit messed up but in a good way.” Once my shoulder was back in place, I would say I was “elated and very chatty” (emphasis on the “very”: apologies to everyone else waiting for an x-ray with me).
In contrast, gas and air didn’t do as much for the pain, though it did have the “messed up” component, which was helpfully distracting – at least as I remember it. There were plenty of confounding variables: injury severity (previously I needed an ambulance, this time not) and memory effects among them. So I wondered what the evidence says about the average differences between the two.
I came across a systematic review (Porter et al., 2017). Both gas and air and Penthrox outperformed placebo; however, no statistically significant difference was found between them in direct comparisons. But hang on – someone ran a study comparing Penthrox with placebo? I’m not sure I’d have met the inclusion criteria, but imagining being invited to take part in a placebo-controlled trial while hanging onto my arm, I suspect my response would have involved expressive language and a firm decline.
The justification for the placebo-control was: “Use of an active comparator, although preferred, would have posed considerable challenges to keep the study blind because of the unique mode of delivery and smell of methoxyflurane” (Coffey et al., 2014, p. 614).
An interesting case example to discuss.
References
Coffey, F., Wright, J., Hartshorn, S., Hunt, P., Locker, T., Mirza, K., & Dissmann, P. (2014). STOP!: A randomised, double-blind, placebo-controlled study of the efficacy and safety of methoxyflurane for the treatment of acute pain. Emergency Medicine Journal, 31(8), 613–618.
Porter, K. M., Siddiqui, M. K., Sharma, I., Dickerson, S., & Eberhardt, A. (2017). Management of trauma pain in the emergency setting: Low-dose methoxyflurane or nitrous oxide? A systematic review and indirect treatment comparison. Journal of Pain Research, 11, 11–21.
