Dodo bird verdicts

The Dodo bird verdict asserts that all effective psychological therapies are equally effective and work because of shared “common factors”. It is heavily debated.

Another Dodo bird verdict could also be said to apply to evaluation methods. I don’t believe that all methods are equally good; however, some do, and it might be handy to have a name for the position.

Here’s the section of Alice’s Adventures in Wonderland that inspired the term:

“What I was going to say,” said the Dodo in an offended tone, “was, that the best thing to get us dry would be a Caucus-race.”

“What is a Caucus-race?” said Alice; not that she wanted much to know, but the Dodo had paused as if it thought that somebody ought to speak, and no one else seemed inclined to say anything.

“Why,” said the Dodo, “the best way to explain it is to do it.” (And, as you might like to try the thing yourself, some winter day, I will tell you how the Dodo managed it.)

First it marked out a race-course, in a sort of circle, (“the exact shape doesn’t matter,” it said,) and then all the party were placed along the course, here and there. There was no “One, two, three, and away,” but they began running when they liked, and left off when they liked, so that it was not easy to know when the race was over. However, when they had been running half an hour or so, and were quite dry again, the Dodo suddenly called out “The race is over!” and they all crowded round it, panting, and asking, “But who has won?”

This question the Dodo could not answer without a great deal of thought, and it sat for a long time with one finger pressed upon its forehead (the position in which you usually see Shakespeare, in the pictures of him), while the rest waited in silence. At last the Dodo said, “Everybody has won, and all must have prizes.”

“Generative” causation

“For the believer in generative causality the existence of the statistics is but the first step in a long process of investigation which ends only when the nature of the things involved has been found out and the reasons for the statistics thus elicited. Science follows the generative rather than the successionist theory of causality.”

– Rom Harré (1972, p. 118)

Harre, R. (1972). The philosophies of science. Oxford University Press.

Sharp variance bounds for SATE: giving Mikhaeil and Green’s (2024) work a test drive

The standard ANCOVA-style way to estimate the sample average treatment effect (SATE) in an efficacy RCT yields an unbiased estimate of the SATE; however, the standard error is a little too big, which means the confidence interval is a little too wide. The reason for this is that the correct standard error depends on the correlation between potential outcomes, only one of which is realised for each participant depending on which experimental condition they ended up in, making that correlation impossible to calculate.

This problem was pointed out by Neyman (1923/1990). Aronow, Green, and Lee (2014) proved a result that puts bounds on the correlation based on the observed marginal distribution of outcomes; however, their approach only works for a model without any covariates.

I’m excited by a new preprint by Mikhaeil and Green (2024) that extends the approach to models with covariates and provides R code, in the {sharpvar} package.

I gave it a go with simulated data; knitted code and output over here.

Propensity score analysis with baseline measure of outcome

Mild headache in this new simulation study by Peter Austin of propensity score matching/weighting studies where you have a baseline measurement of the outcome variable – a useful thing to balance as it’s often highly correlated with outcome.

For ATE, the recommendations make sense: include the baseline in the propensity model and then, for what are often called doubly robust approaches, include baseline again as a covariate in the outcome model.

For ATT, the findings are not what I would have expected. Austin’s recommendation for best standard errors is to exclude the baseline from the propensity model and analyse change from baseline in the outcome model 😱:

“… recommendations for the analysis of RCTs with baseline measurements of the follow-up variable do not reflect perfectly what we observed in the context of the analysis of observational studies using propensity score methods. We found that the analysis of change from baseline using a propensity score that excluded the baseline value of the follow-up variable tended to result in the most precise estimates of treatment effect. However, we did observe that, when using weighting, the use of ANCOVA in conjunction with a propensity score model that included the baseline value of the follow-up variable tended to perform well.”

One to read properly to see what’s going on. Would be interested in your thoughts.

References

Austin, P. (2024). Propensity Score Analysis With Baseline and Follow-Up Measurements of the Outcome Variable. Pharmaceutical Statistics. Early view.

Analysing individually randomized group treatment trials with complex clustering in nested and crossed designs

Interesting paper by Jonathan Moyer, Fan Li, and others on the best ways to analyse individually randomised trials in which (1) participants share professionals such as therapists and case workers, (2) participants may interact with multiple professionals over the course of the trial, and (3) the nesting structure may differ between experimental conditions.

Sampling without replacement with specified marginal sampling probabilities

The {sampling} library by Yves Tillé and Alina Matei implements Brewer’s (1975) method – discussed above – as UPbrewer. Tillé’s (2023) review looks interesting too, cataloguing a bunch of incorrect methods and what to use instead.

References

Brewer, K.E.W. (1975), A simple procedure for sampling πpswor. Australian Journal of Statistics, 17, 166-172.

Tillé, Y. (2023). Remarks on some misconceptions about unequal probability sampling without replacement. Computer Science Review, 47, 100533.

Being brave with causal interpretations

One of the fabulous things about evaluation is that we are allowed to (or, rather, required to!) make causal claims from a wide variety of evidence, not only experiments. Causal estimands, defined in terms of potential outcomes, are the norm for quantitative impact evaluations. Causal DAGs are becoming more popular too and one of my favourite tools for thinking through quasi-experimental designs.

Not all fields have it so good.

Julia Rohrer has written a great blog post, following a series of journal papers, urging personality psychologists to be upfront with their causal claims. It’s challenging to persuade and I think interesting as a sociology of science case study. As Rohrer summarises, ‘Being vague about the desired interpretation may maintain some degree of plausible deniability and potentially offload the error to others (“oh, you shouldn’t have interpreted our numbers like that!1”). But honestly, that’s just a cowardly way to do science. Life is too short for that.’

Comparing {eefAnalytics} and {lmeInfo}

In multilevel analyses, it is often necessary to estimate a standardised mean difference (SMD) where the numerator is a covariate-adjusted mean difference and the denominator is the SD from a model without covariate adjustment.

I’m aware of two (frequentist, non-bootstrap) packages calculating an SMD and confidence interval in R. Here’s a comparison of them for simulated data.

Work in progress – different ways to calculate Hedges’ g in cluster RCTs

The problem is, you want to estimate g with a covariate-adjusted treatment effect in the numerator but unadjusted SD in the denominator and an uncertainty interval of some description.

Work in progress R code here – scroll to the bottom for a table comparing results. Use at own risk.

Here is also a comparison of {eefAnalytics} and {lmeInfo}.

A new checklist for designing scientific data visualizations, by Helena Klara

Interesting new checklist for designing scientific data visualizations, by Helena Klara Jambor. The first test is to ask 2 to 5 people to explain what they see first when looking at the visualisation. I like how this treats graphs similarly to how user interfaces are treated in HCI/UX research: checking whether a graph works as intended is an empirical problem.