Blog 🎤

Totally random

If you take 30 draws from a Bernoulli distribution, e.g., to randomise people to one of two conditions, there’s a \(\frac{2}{2^{30}} = \frac{1}{2^{29}} = \frac{1}{536,870,912}\) chance that all 30 will be assigned to the same condition (all 1s or all 0s).

That’s vanishingly unlikely, but using the following seed with R’s rbinom produces 30 ones:

> set.seed(236903047)
> rbinom(30, size = 1, prob = 0.5)
[1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

The seed below produces 30 zeros:

> set.seed(627034302)
> rbinom(30, size = 1, prob = 0.5)
[1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Unlikely but possible – even if your random numbers come from atmospheric noise or quantum measurements rather than a pseudorandom generator. If you want to rule out this outcome entirely (make the probability zero), then you don’t actually want independent Bernoulli draws. You’d need a different randomisation scheme, such as blocked randomisation.

I often see reports describing checks on random sequences after they’ve been generated, when the researchers could have saved themselves the effort by building in the kind of randomness they wanted from the start.

P.S. The longest initial runs I know about are for seed 796110494, which yields 31 zeros, and 1002755089, which yields 32 ones.

Against treatment

A scattered group of colourful capsules containing some pharma substance or other

Dear fellow trialists, can we please stop using the term treatment when it’s not a medical treatment? We’re not administering drugs – we’re evaluating, e.g., mentoring, approaches to teaching, and emotional support. Using medical language in the context of schools or criminal justice sounds especially creepy. I think we should also rename estimands like average treatment effect. How about average causal effect, which is sometimes used too? Language matters, especially when we’re trying to build trust with communities who are often suspicious of randomised experiments. If we want our work to be understood and welcomed outside quantitative impact evaluation, we need to be more careful with the metaphors we borrow.

The same goes for terms like contamination and dosage. Contamination is usually invoked when making the case for cluster randomisation, referring to the risk that individuals in the control group receive elements of the programme being evaluated too, e.g., students telling each other what they learned working with a mentor. Calling it “contamination” implies something toxic or infectious. A better term may be spillover, diffusion, or cross-group knowledge sharing.

Dosage is another common term that feels weird. It’s typically used to describe how much of a programme someone engages with, e.g., the number of sessions attended or hours of support. But again, it carries pharmacological baggage. People aren’t just absorbing a dose, they’re participating, doing activities that are suggested, sharing how they feel with a practitioner. Terms like engagement level, participation intensity, or simply sessions attended might better reflect what’s actually going on.

I’ve used all these terms too as they are so pervasive in the literature – this isn’t me being holier-than-thou. But if we want trials to be accessible and respectful, we need to rethink the language we use. It’s no wonder people are put off trials when the basic terminology sounds so clinical and dehumanising. Careful use of language would make it clearer what we are actually doing and how crucial a role participants play in the programmes we’re evaluating.

optweight: Optimization-Based Stable Balancing Weights

Something I’d like to play with when I get the chance:

“optweight contains functions to estimate stable balancing weights that balance covariates up to given thresholds. It solves a convex optimization problem to minimize a function of the weights that captures their variability (or divergence from a set of base weights). This is the method described in Zubizarreta (2015), Källberg and Waernbaum (2023), and Wang and Zubizarreta (2020). optweight extends the method to multi-category, continuous, and multivariate treatments and provides a simple user interface and compatibility with the cobalt package for balance assessment. See vignette("optweight") for a more thorough description of the package’s capabilities.”

Entropy balancing and inverse probability tilting

Something fun (Słoczyński et al., 2025, p. 3, footnote 1):

“IPT [inverse probability tilting] coincides with Hainmueller’s (2012) entropy balancing estimator when the propensity score is estimated with the logit model”.

(It just holds for ATT.) The picture below illustrates the correspondence, using the Lalonde (1986) dataset (in the cobalt package in R). I computed (knitted R here) the weights using the WeightIt package (Greifer, 2025). Since we’re aiming for ATT, the intervention group weights are all equal to 1, so I’ve omitted them from the graph.

The weights are perfectly correlated and the effective sample sizes are the same. They are scaled differently, though (in the WeightIt implementation at least – need to wade into the original papers): entropy weights sum to the comparison group sample size and IPT weights sum to the intervention group sample size.

References

Greifer, N. (2025). WeightIt: Weighting for Covariate Balance in Observational Studies (version 1.5.0) [R Package].

Słoczyński, T., Uysal, S. D., & Wooldridge, J. M. (2025). Covariate Balancing and the Equivalence of Weighting and Doubly Robust Estimators of Average Treatment Effects. IZA Institute of Labour Economics Discussion Paper, 18147.

What do social policy evaluators do?

Social policy evaluators investigate whether public services and infrastructure – such as housing, education, health care, policing, and transport – are making a meaningful difference to the people who rely on them, not just meeting official targets. Evaluations span many levels, from local support services (such as mentoring programmes for young people) to major infrastructure projects (such as transport links) and national reforms (such as changes to sentencing guidelines).

Evaluations often combine different approaches. These might include analysing trends and differences in outcomes (such as school attendance rates or reoffending figures) and gathering personal accounts of how services affect people’s lives (such as whether someone feels safer after contact with the police). Evaluators also examine how services are delivered in practice: what staff and service users actually do when engaging with services, the challenges they face, and the conditions that shape their experiences and outcomes.

Evaluation is not only about finding out whether service use is “associated with” outcomes – it’s about testing whether a service actually caused a change. This often involves using advanced methods from statistics, economics, and computer science. There is also growing recognition that people affected by services should play a key role in shaping how those services are designed and evaluated – advice from experts by lived experience complements advice provided by professionals.

Beyond identifying what works and what doesn’t, social policy evaluation aims to deepen our understanding of how and why services function. It does this by developing and testing theories of change. The vision is for decisions about public services to be grounded in cumulative evidence of effectiveness, and shaped by the people those services are meant to support.

Why randomise rather than alternate?

Altman (1991, p. 1481):

“… random allocation in a clinical trial means that all patients have the same chance of receiving any particular treatment (and in most cases each treatment is equally likely). Patients should be entered into a trial before their allocation to a particular treatment is known. A common misconception exists that allocation based on, for example, odd or even dates of birth or hospital numbers is random. These systematic allocation methods, however, clearly violate the requirement that all patients have the same chance of receiving each treatment. Alternate allocation does not in principle suffer from such problems, but there is a risk of abuse because the investigator’s knowledge of the next treatment may lead to some patients being excluded from the trial – making this method inadvisable.”

References

Altman, D. G. (1991). Randomisation. BMJ, 302(6791), 1481–1482. https://doi.org/10.1136/bmj.302.6791.1481

Balance again

If you’re running an RCT, why bother balancing covariates between treatment and control, e.g., using blocking or minimisation?

Doing so doesn’t make it any more likely that your confidence interval will include the true treatment effect – if it did, then what about the thousands of unmeasured covariates, which randomisation is supposed to address…?

Instead, balancing covariates makes inference more efficient when you statistically adjust for those measured covariates – as you will want to do if they’re correlated with the outcome variable.

Here’s Senn (1994, p. 1721) – a paper I seem to cite every few weeks:

“… what is the value of balance as regards validity of an inference? Answer: none. This does not mean that balance is not useful. Its use is that, having decided to condition, we minimize the standard errors of our treatment effects if our covariate is orthogonal to treatment. This is the true value of balance which is an issue of efficiency, not validity.”

So it’s regression analysis 101: if you have an imbalanced covariate, then it is correlated with your treatment variable. The greater the correlation, the greater the standard error of the treatment effect estimate. Though this will be mitigated if that covariate is also highly correlated with the outcome, which will reduce the standard error.

There is more on balance over here.

References

Senn, S. (1994). Testing for baseline balance in clinical trials. Statistics in Medicine, 13, 1715–1726.

Is it a psychological treatment?

This by Tolin et al. (2025) looks interesting and controversial: Tolin, D. F., Grasso, D., Boness, C. L., Beck, J. G., Keane, T. M., Leichsenring, F., Olatunji, B. O., Otto, M. W., & Weinand, J. (2025). A proposed definition of psychological treatment and its relation to empirically supported treatments. Clinical Psychology: Science and Practice, 32(3), 213–225. https://doi.org/10.1037/cps0000220

Picture:

Two fun R packages

Pinference: Probability Inference for Propositional Logic. “Implementation of T. Hailperin’s procedure to calculate lower and upper bounds of the probability for a propositional-logic expression, given equality and inequality constraints on the probabilities for other expressions. Truth-valuation is included as a special case. Applications range from decision-making and probabilistic reasoning, to pedagogical for probability and logic courses.”

RobinCar2: ROBust INference for Covariate Adjustment in Randomized Clinical Trials. “Performs robust estimation and inference when using covariate adjustment and/or covariate-adaptive randomization in randomized controlled trials.” Though see this post.

Statistics as a valid form of Indigenous knowledge

Chris Andersen (2025, p. 31):

‘[…] talking about statistics as a valid form of Indigenous knowledge and as an important tool for Indigenous decolonization efforts raises strong feelings for many. We continue to take the position that Indigenous knowledge must include any form of knowledge that benefits Indigenous Peoples. We are not suggesting that no differences exist between Indigenous and “Western” forms of knowledge (to give one example of many, ceremonial knowledge can and does form the backbone of Indigenous governing systems, though it would rarely be confused with that of Western governing systems). Our point instead is simply that Indigenous Peoples are powerfully embedded in the last five centuries of colonial projects. As such, any tools that help us push back against such projects, and that assist us in “constituting ourselves anew,” are worth discussing. And we will argue further that statistics—literally the “language of state”—must be included as one of those tools. It is too important a vector in the fight for Indigenous control of our collective lives and territories not to.’

Andersen, C. (2025). The Statistical Field, Writ Indigenous. In C. Andersen, M. Walter, T. Kukutai, & C. Gabel, Indigenous Statistics (2nd ed., pp. 30–47). Routledge. https://doi.org/10.4324/9781003173342-3