Tag: methodology

Balance again

If you’re running an RCT, why bother balancing covariates between treatment and control, e.g., using blocking or minimisation?

Doing so doesn’t make it any more likely that your confidence interval will include the true treatment effect – if it did, then what about the thousands of unmeasured covariates, which randomisation is supposed to address…?

Instead, balancing covariates makes inference more efficient when you statistically adjust for those measured covariates – as you will want to do if they’re correlated with the outcome variable.

Here’s Senn (1994, p. 1721) – a paper I seem to cite every few weeks:

“… what is the value of balance as regards validity of an inference? Answer: none. This does not mean that balance is not useful. Its use is that, having decided to condition, we minimize the standard errors of our treatment effects if our covariate is orthogonal to treatment. This is the true value of balance which is an issue of efficiency, not validity.”

So it’s regression analysis 101: if you have an imbalanced covariate, then it is correlated with your treatment variable. The greater the correlation, the greater the standard error of the treatment effect estimate. Though this will be mitigated if that covariate is also highly correlated with the outcome, which will reduce the standard error.

There is more on balance over here.

References

Senn, S. (1994). Testing for baseline balance in clinical trials. Statistics in Medicine, 13, 1715–1726.

Two fun R packages

Pinference: Probability Inference for Propositional Logic. “Implementation of T. Hailperin’s procedure to calculate lower and upper bounds of the probability for a propositional-logic expression, given equality and inequality constraints on the probabilities for other expressions. Truth-valuation is included as a special case. Applications range from decision-making and probabilistic reasoning, to pedagogical for probability and logic courses.”

RobinCar2: ROBust INference for Covariate Adjustment in Randomized Clinical Trials. “Performs robust estimation and inference when using covariate adjustment and/or covariate-adaptive randomization in randomized controlled trials.” Though see this post.

Statistics as a valid form of Indigenous knowledge

Chris Andersen (2025, p. 31):

‘[…] talking about statistics as a valid form of Indigenous knowledge and as an important tool for Indigenous decolonization efforts raises strong feelings for many. We continue to take the position that Indigenous knowledge must include any form of knowledge that benefits Indigenous Peoples. We are not suggesting that no differences exist between Indigenous and “Western” forms of knowledge (to give one example of many, ceremonial knowledge can and does form the backbone of Indigenous governing systems, though it would rarely be confused with that of Western governing systems). Our point instead is simply that Indigenous Peoples are powerfully embedded in the last five centuries of colonial projects. As such, any tools that help us push back against such projects, and that assist us in “constituting ourselves anew,” are worth discussing. And we will argue further that statistics—literally the “language of state”—must be included as one of those tools. It is too important a vector in the fight for Indigenous control of our collective lives and territories not to.’

Andersen, C. (2025). The Statistical Field, Writ Indigenous. In C. Andersen, M. Walter, T. Kukutai, & C. Gabel, Indigenous Statistics (2nd ed., pp. 30–47). Routledge. https://doi.org/10.4324/9781003173342-3

Covariate Commons

Here’s a half‑baked idea, inspired by multiple conversations about randomised controlled trials versus quasi‑experiments at the Australian Evaluation Society 2025 conference.

Outcome measurement databases are now a familiar part of evaluation. One database I frequently use is the Child Outcomes Research Consortium (CORC) Directory of Outcome Measures. The Education Endowment Foundation (EEF) has developed similar resources.

Databases like these make it easier to map theories of change outcomes to measures, and they often include information on psychometric properties such as reliability, and pre‑post correlations, which are handy for informing power calculations.

While outcomes receive this collective attention, quasi‑experimental designs hinge just as much on the quality of their covariates. Pre‑intervention measures of outcomes are especially important, given how highly correlated they tend to be with outcomes. That’s the covariate we get for free once we fix on an outcome, but other characteristics of people and contexts are essential too – often a large number of them.

So, the half‑baked idea is this: evaluation should develop a covariates database, organised by topic area and outcome, to provide evaluators with a shared resource to aid the design of quasi‑experiments. A potential name for this shared resource is the Covariate Commons.

The selection of covariates should always be considered on a case‑by‑case basis, guided by the most likely confounding variables (those that correlate both with who engages with a programme and with outcomes). However, evaluators know that some variables appear repeatedly across studies and sometimes across topic areas (age, gender, social deprivation, to name three).

What do you think? Is there already an example of this for some topics? And who should be responsible for developing and maintaining it?

The green whistle test

I have some lived-experience to report and a brief reflection on an RCT. Content warning: it involves a brief description of an injury, but I lived happily ever after!

I recently dislocated my shoulder after slipping on a traverse climbing wall – ten minutes before the end of an introductory bouldering session. Thankfully, I wasn’t far off the ground. As far as these things go, it was a “good” dislocation: I feel fine and am starting physio imminently.

It wasn’t my first. Over a decade ago, I had a dislocation and was given nitrous oxide (gas and air) at A&E while they reattached my arm. This time, I was handed Penthrox – also known as the green whistle – a vape-sized handheld device that delivers methoxyflurane.

Subjectively, Penthrox felt much better than gas and air. During the procedure, I’d describe my mental state as “relaxed, pain-free, and a bit messed up but in a good way.” Once my shoulder was back in place, I would say I was “elated and very chatty” (emphasis on the “very”: apologies to everyone else waiting for an x-ray with me).

In contrast, gas and air didn’t do as much for the pain, though it did have the “messed up” component, which was helpfully distracting – at least as I remember it. There were plenty of confounding variables: injury severity (previously I needed an ambulance, this time not) and memory effects among them. So I wondered what the evidence says about the average differences between the two.

I came across a systematic review (Porter et al., 2017). Both gas and air and Penthrox outperformed placebo; however, no statistically significant difference was found between them in direct comparisons. But hang on – someone ran a study comparing Penthrox with placebo? I’m not sure I’d have met the inclusion criteria, but imagining being invited to take part in a placebo-controlled trial while hanging onto my arm, I suspect my response would have involved expressive language and a firm decline.

The justification for the placebo-control was: “Use of an active comparator, although preferred, would have posed considerable challenges to keep the study blind because of the unique mode of delivery and smell of methoxyflurane” (Coffey et al., 2014, p. 614).

An interesting case example to discuss.

References

Coffey, F., Wright, J., Hartshorn, S., Hunt, P., Locker, T., Mirza, K., & Dissmann, P. (2014). STOP!: A randomised, double-blind, placebo-controlled study of the efficacy and safety of methoxyflurane for the treatment of acute pain. Emergency Medicine Journal, 31(8), 613–618.

Porter, K. M., Siddiqui, M. K., Sharma, I., Dickerson, S., & Eberhardt, A. (2017). Management of trauma pain in the emergency setting: Low-dose methoxyflurane or nitrous oxide? A systematic review and indirect treatment comparison. Journal of Pain Research, 11, 11–21.

DAGs are not necessarily associated with causation

The bad old days: an association was found between X and Y; however, correlation does not imply causation. Check out this long list of covariates and how they were adjusted in a big ol’ regression model.

These enlightened times: X causes Y; here’s a causal DAG, which you critically appraise by determining which important confounding variables are absent; whether something that’s currently a confounder should, e.g., really be a mediator or vice versa; and whether the regression, weighting, or matching model reflects what you think is the true causal structure, having critically appraised the authors’ DAG.

The main benefit of causal DAGs is not in identifying confounders but rather not mistakenly conditioning on the wrong things. The hard work of non-experimental causal inference is in the topic theory. X causes Y in the DAG does not imply causation in the world without a solid tested theory.

Subverting randomisation

One of the reasons that people test covariate balance in trials is because they have a vague sense that they should check whether something has gone wrong with random assignment – but often without a clear idea of what that something is. One thing that could go wrong is that people involved in participant recruitment, such as clinicians, could be deliberately trying to subvert assignment so people receive the treatment clinicians think they need.

Brown et al. (2005) asked 25 clinicians and research nurses whether they tried to predict treatment allocation and, if they did, how, and why.

Reassuringly, most (21 out of 25) said they did not, citing that they were “aware that this is wrong”, wanted to “remain impartial to the treatments”, were “unable to, allocation is out of our hands”, or thought it would be too complicated to do so.

The 4 out of 25 who said that they did try to predict treatment allocation reported keeping a log of previous allocations:

“A few investigators stated that they hoped some patients would be randomized to specific treatments, giving such reasons as that patients would be able to travel more easily or that some would have a perceived benefit from a particular treatment.”

References

Brown, S., Thorpe, H., Hawkins, K., & Brown, J. (2005). Minimization—Reducing predictability for multi‐centre trials whilst retaining balance within centre. Statistics in Medicine, 24(24), 3715–3727.

Fei Wan (2025) on propensity score matching

Quasi-experimentalists will be familiar with King and Nielsen’s (2019) landmark paper, Why Propensity Scores Should Not Be Used for Matching. This specifically critiqued the use of propensity scores for matching; other common uses of propensity scores, such as inverse probability weighting, were not affected. An interesting paper has appeared by Fei Wan (2025) critiquing King and Nielsen’s findings.

Fei Wan’s targets include:

  1. The inappropriateness of using an average pairwise covariate distance between treated and closest comparison match to evaluate PSM. Distances are always nonnegative so can’t take account of positive and negative differences averaging out. It’s already known that two observations with the same propensity score are likely to have different covariate values, but it’s ok if they are random.
  2. King and Nielsen used a cherry-picking approach for model selection, trying 512 different specifications and selecting the one that yields the largest average treatment estimate. This is just poor analysis practice.

Fei Wan recommends machine learning approaches to estimate propensity scores, rather than the common use of logistic regression.

Lots more to digest in this…

References

King, G., & Nielsen, R. (2019). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 27(4), 435–454.

Wan, F. (2025). Propensity Score Matching: Should we use it in designing observational studies? BMC Medical Research Methodology, 25(1), 25.