Tag: RCTs

Subverting randomisation

One of the reasons that people test covariate balance in trials is because they have a vague sense that they should check whether something has gone wrong with random assignment – but often without a clear idea of what that something is. One thing that could go wrong is that people involved in participant recruitment, such as clinicians, could be deliberately trying to subvert assignment so people receive the treatment clinicians think they need.

Brown et al. (2005) asked 25 clinicians and research nurses whether they tried to predict treatment allocation and, if they did, how, and why.

Reassuringly, most (21 out of 25) said they did not, citing that they were “aware that this is wrong”, wanted to “remain impartial to the treatments”, were “unable to, allocation is out of our hands”, or thought it would be too complicated to do so.

The 4 out of 25 who said that they did try to predict treatment allocation reported keeping a log of previous allocations:

“A few investigators stated that they hoped some patients would be randomized to specific treatments, giving such reasons as that patients would be able to travel more easily or that some would have a perceived benefit from a particular treatment.”

References

Brown, S., Thorpe, H., Hawkins, K., & Brown, J. (2005). Minimization—Reducing predictability for multi‐centre trials whilst retaining balance within centre. Statistics in Medicine, 24(24), 3715–3727.

RCTs can and do use sensible control groups

If there are effective treatments for a condition, then a placebo-controlled randomised trial would generally be unethical (see below for exceptions discussed in the literature). But RCTs do not need to use placebo control.

There is a common misconception that an RCT should have a “pure” control (whatever that means) and if an active control of some description is used, e.g., another treatment, then it becomes A/B testing. This is false. A/B testing is just a type of trial and the term is more commonly used in marketing, e.g., in studies of variations of a website design.

RCTs can and do have active controls. Education trials and psychological therapy trials often use usual practice controls. Once effective Covid vaccines had been found, trials investigated the comparative efficacy and safety of different vaccines.

Example discussions

“Often, trials are conducted to compare a new intervention against current practice. The new intervention might be a small change, or a set of small changes to current practice; or it could be a whole new approach which is proving to be successful in a different country or context, or that has sound theoretical backing.” (Haynes, Goldacre, & Torgerson, 2012, p. 20)

“An RCT is not necessarily a test between doing something and doing nothing. Many interventions might be expected to do better than nothing at all. Instead, trials can be used to establish which of a number of policy intervention options is best.” (Haynes, Goldacre, & Torgerson, 2012, p. 21)

“Placebo controls are clearly inappropriate for conditions in which delay or omission of available treatments would increase mortality or irreversible morbidity in the population to be studied. For conditions in which forgoing therapy imposes no important risk, however, the participation of patients in placebo-controlled trials seems appropriate and ethical, as long as patients are fully informed.” (Temple & Ellenberg, 2000, p. 460)

“In cases where an available treatment is known to prevent serious harm, such as death or irreversible morbidity in the study population, it is generally inappropriate to use a placebo control. There are occasional exceptions, however, such as cases in which standard therapy has toxicity so severe that many patients have refused to receive it.” (European Medicines Agency, 2001, p. 16)

“Many placebo-controlled trials are conducted as add-on trials, where all patients receive a specified standard therapy or therapy left to the choice of the treating physician or institution.” (European Medicines Agency, 2001, p. 16)

References

European Medicines Agency (2001). ICH Topic E 10: Choice of Control Group in Clinical Trials.

Haynes, L., Goldacre, B., & Torgerson, D. (2012). Test, learn, adapt: developing public policy with randomised controlled trials. Cabinet Office and Behavioural Insights Team.

Temple, R., & Ellenberg, S. S. (2000). Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 1: ethical and scientific issues. Annals of Internal Medicine, 133(6), 455-463.

Carefully controlled nonrandomised studies give us subjective randomisation

“Consider a carefully controlled nonrandomized study – a study in which there are no obviously important prior variables that systematically differ in the E trials and the C trials. In such a study, there is a real sense in which a claim of ‘subjective randomization’ can be made. For example, if the study was composed of carefully matched pairs of trials, there might be a very defensible belief that within each matched pair each unit was equally likely to receive E as C in the sense that if you were shown the units without being told which received E, only half the time would you correctly guess which received E.”

Rubin, D. B. (1974, p. 698). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.


See also:

“Matching can be thought of as a technique for finding approximately ideal experimental data hidden within an observational data set.”

King, G., & Nielsen, R. (2019, p. 442). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 27(4), 435–454.

Sharp variance bounds for SATE: giving Mikhaeil and Green’s (2024) work a test drive

The standard ANCOVA-style way to estimate the sample average treatment effect (SATE) in an efficacy RCT yields an unbiased estimate of the SATE; however, the standard error is a little too big, which means the confidence interval is a little too wide. The reason for this is that the correct standard error depends on the correlation between potential outcomes, only one of which is realised for each participant depending on which experimental condition they ended up in, making that correlation impossible to calculate.

This problem was pointed out by Neyman (1923/1990). Aronow, Green, and Lee (2014) proved a result that puts bounds on the correlation based on the observed marginal distribution of outcomes; however, their approach only works for a model without any covariates.

I’m excited by a new preprint by Mikhaeil and Green (2024) that extends the approach to models with covariates and provides R code, in the {sharpvar} package.

I gave it a go with simulated data; knitted code and output over here.

Analysing individually randomized group treatment trials with complex clustering in nested and crossed designs

Interesting paper by Jonathan Moyer, Fan Li, and others on the best ways to analyse individually randomised trials in which (1) participants share professionals such as therapists and case workers, (2) participants may interact with multiple professionals over the course of the trial, and (3) the nesting structure may differ between experimental conditions.

Hedges’ g for multilevel models in R {lmeInfo}

This package looks useful (for {nlme} not {lme4}).

“Provides analytic derivatives and information matrices for fitted linear mixed effects (lme) models and generalized least squares (gls) models estimated using lme() (from package ‘nlme’) and gls() (from package ‘nlme’), respectively. The package includes functions for estimating the sampling variance-covariance of variance component parameters using the inverse Fisher information. The variance components include the parameters of the random effects structure (for lme models), the variance structure, and the correlation structure. The expected and average forms of the Fisher information matrix are used in the calculations, and models estimated by full maximum likelihood or restricted maximum likelihood are supported. The package also includes a function for estimating standardized mean difference effect sizes (Pustejovsky, Hedges, and Shadish (2014) <doi:10.3102/1076998614547577>) based on fitted lme or gls models.”

Why does everyone love a good RCT?

The individual treatment effect is defined as an individual’s potential outcome under treatment minus their potential outcome under control. This within-participant difference cannot be directly measured since only one of the two potential outcomes is realised depending on whether the participant was exposed to treatment or control.

Everyone loves a good randomised controlled trial because the mean outcome of people who were exposed to treatment minus the mean outcome of people who were exposed to control – a between-participant difference – is an unbiased estimator of the mean of within-participant individual treatment effects.

I’ve coded up a simulation in R over here to illustrate how they work. Note in particular the importance of confidence intervals!

On the term “randomista”

Sophie Webber and Carolyn Prouse (2018, p. 169, footnote 1) write:

“Randomistas is a slang term used by critics to describe proponents of the RCT methodology. It is almost certainly a gendered, derogatory term intended to flippantly dismiss experimental economists and their success, particularly Esther Duflo, one of the most successful experts on randomization.” [Emphasis original.]

References

Sophie Webber and Carolyn Prouse (2018). The New Gold Standard: The Rise of Randomized Control Trials and Experimental Development. Economic Geography, 94(2), 166–187.