Tag: methodology

ITT ATT

Intention to treat (ITT) analyses code a participant’s group according to the condition they were assigned to, regardless of whether they engaged with that condition or, e.g., swapped to another. Sometimes the average treatment effect on the treated (ATT) estimand is misunderstood to mean analysing people according to what intervention they actually engaged with. But the two concepts are independent of each other and depend on how the potential outcomes are defined.

For example, let Yᵢ(ai) denote the potential outcome for participant, i, were they to be assigned to the intervention, and Yᵢ(ac) denote the potential outcome were they to be assigned to some comparison condition. Let Gᵢ denote the group they were assigned to (note, not necessarily what they engaged with!). The following estimand is an ITT ATT:

E[Yᵢ(ai) – Yᵢ(ac) | Gᵢ = ai]

Intervals for individuals

A clear and concise guide to choosing between the standard error of measurement (SEM) interval and the standard error of estimation (SEE) interval – two intervals that are used when interpreting an individual’s score in psychological testing (Schmukle and Rohrer, 2025).

Some areas, such as the personality test industry, would benefit from more often using any kind of interval on their proclamations.

Schmukle, S. C., & Rohrer, J. M. (2025). Clarifying the Choice of Confidence Intervals in Psychological Testing: A Comment on Stanley and Spence (2024). Advances in Methods and Practices in Psychological Science.

Key study design considerations in state policy evaluation

A new preprint by Elizabeth Stone and colleagues explores methods for evaluating state-level policies in the US, offering insights relevant to UK policy evaluation. It covers selection of a range of approaches based on, for example, how a policy varies across the country, whether there is staggered roll out, and the availability of baseline data.

References

Stone, E. M., Schuler, M. S., Stuart, E. A., Rubinstein, M., Griswold, M., Stein, B. D., & Griffin, B. A. (2025). Choosing an analytic approach: Key study design considerations in state policy evaluation.

Hearing impairment and incident heart failure

Another potential example for a DAG or two. Huang et al. (2025) found an association between hearing impairment and subsequent heart failure, which they attributed to a common cause of systemic vascular disorders. I don’t know anything about either topic, but it sounds plausible – more so than, e.g., hearing loss causing heart failure. They also found that whether people wore a hearing aid did not moderate this association – which again makes sense, though it was unclear what the authors expected:

“Of note, both the participants who used hearing aids and those with poor hearing status had a similarly significant increase in the risk of incident HF, suggesting that while hearing aids can improve auditory function, they may not address the underlying vascular issues that contribute to the risk of HF.”

I think it’s missing an “unsurprisingly”, but again don’t know the topic.

But then the mediation test comes along and suggests that the association between hearing loss and heart failure is mediated by psychological factors such as stress. Now how to get that in a DAG with the common cause model.

Here’s the relevant paragraph:

“Second, we found that the positive association between poor hearing status and incident HF was significantly mediated by psychological factors. Because hearing problems can lead to difficulties in speech comprehension and poor engagement in social activities, people with hearing impairment are more likely to experience social isolation, psychological distress, anxiety and depression than people without hearing impairment. These psychological factors may increase the activity of the sympathetic nervous system and the hypothalamic-pituitary-adrenal axis, and enhance inflammation and oxidative stress, thereby accelerating atherosclerosis, increasing peripheral stress, and promoting the development of cardiac remodelling in the pre-HF period, thus significantly increasing the risk of HF.”

References

Huang, Y., Zhang, Y., Zhang, Y., Xiang, H., Ye, Z., Yang, S., Gan, X., Wu, Y., Zhang, Y., & Qin, X. (2025). Hearing impairment, psychological distress, and incident heart failure: A prospective cohort study. Heart, heartjnl-2024-325394.

RCTs can and do use sensible control groups

If there are effective treatments for a condition, then a placebo-controlled randomised trial would generally be unethical (see below for exceptions discussed in the literature). But RCTs do not need to use placebo control.

There is a common misconception that an RCT should have a “pure” control (whatever that means) and if an active control of some description is used, e.g., another treatment, then it becomes A/B testing. This is false. A/B testing is just a type of trial and the term is more commonly used in marketing, e.g., in studies of variations of a website design.

RCTs can and do have active controls. Education trials and psychological therapy trials often use usual practice controls. Once effective Covid vaccines had been found, trials investigated the comparative efficacy and safety of different vaccines.

Example discussions

“Often, trials are conducted to compare a new intervention against current practice. The new intervention might be a small change, or a set of small changes to current practice; or it could be a whole new approach which is proving to be successful in a different country or context, or that has sound theoretical backing.” (Haynes, Goldacre, & Torgerson, 2012, p. 20)

“An RCT is not necessarily a test between doing something and doing nothing. Many interventions might be expected to do better than nothing at all. Instead, trials can be used to establish which of a number of policy intervention options is best.” (Haynes, Goldacre, & Torgerson, 2012, p. 21)

“Placebo controls are clearly inappropriate for conditions in which delay or omission of available treatments would increase mortality or irreversible morbidity in the population to be studied. For conditions in which forgoing therapy imposes no important risk, however, the participation of patients in placebo-controlled trials seems appropriate and ethical, as long as patients are fully informed.” (Temple & Ellenberg, 2000, p. 460)

“In cases where an available treatment is known to prevent serious harm, such as death or irreversible morbidity in the study population, it is generally inappropriate to use a placebo control. There are occasional exceptions, however, such as cases in which standard therapy has toxicity so severe that many patients have refused to receive it.” (European Medicines Agency, 2001, p. 16)

“Many placebo-controlled trials are conducted as add-on trials, where all patients receive a specified standard therapy or therapy left to the choice of the treating physician or institution.” (European Medicines Agency, 2001, p. 16)

References

European Medicines Agency (2001). ICH Topic E 10: Choice of Control Group in Clinical Trials.

Haynes, L., Goldacre, B., & Torgerson, D. (2012). Test, learn, adapt: developing public policy with randomised controlled trials. Cabinet Office and Behavioural Insights Team.

Temple, R., & Ellenberg, S. S. (2000). Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 1: ethical and scientific issues. Annals of Internal Medicine, 133(6), 455-463.

Green Book review

Debated 2 April 2025 Westminster Hall. Hansard link.

Sarah Russell (Lab): “We believe that the review must be done with ambition and a willingness to challenge underlying customs and practices. In particular, it needs to learn from the successes and failures—largely failures, I would say—of previous Green Book reviews, in particular the 2020 review.”

“There is a consultancy industry around the Green Book, and that raises the cost of developing a successful business case. It also makes it very difficult for smaller local authorities to successfully put together a business case because the complexity of doing so is absolutely mind boggling. Frankly, the whole thing is just a bunch of piffle—we need to make this very simple and outcomes driven.”

Jeevun Sandher (Lab): “For too long, investment has been biased towards London and the south-east. Someone living in London gets £800 more infrastructure spending per year than the national average. That is because the cost-benefit analysis set out in the Green Book to evaluate projects has a hardwired London bias. First, the Green Book prices the benefits of projects in a way that benefits places with higher wages—namely, London. Secondly, it does not estimate the wider impact of investment on growth.”

Andrew Cooper (Lab): “The Green Book is part of a colourful spectrum of current Government guidance and standards documents, which alongside the Aqua Book, the Magenta Book, the Orange Book and 19 other publications and pieces of supplementary guidance forms the basis of how the Treasury appraises policies, programmes and projects. On top of that, several Departments have issued their own interpretation of the guidance as it applies to them, thankfully avoiding extending the colourful metaphors any further.

“That complexity—well over 1,000 pages of guidance—is at the root of the criticism of the small industry and almost mysticism around navigating the assessment process and of the poor outcomes to which it leads.”

“Local authorities tell me it is still common practice for response letters to open with sentences such as, “We note that the BCR for the proposed scheme is 1.8.” That belies the intent of the 2020 review. […] The review must end the arbitrary BCR thresholds across Government.”

200 years of diff-in-diffs

“The starting point of this paper was a 2 × 2 DiD design that researchers have been using for almost 200 years. The end point was a design with five treatment groups, 11 years of data, six covariates, three types of parallel trends assumptions, and four estimation techniques.”

Baker, A., Callaway, B., Cunningham, S., Goodman-Bacon, A., & Sant’Anna, P. H. C. (2025). Difference-in-Differences Designs: A Practitioner’s Guide (No. arXiv:2503.13323). arXiv. https://doi.org/10.48550/arXiv.2503.13323

Testimony

I would conjecture* that qualitative testimony about the impact of social programmes is often highly sensitive (if a programme works, then people will tell you it works) but has very poor specificity (if it doesn’t work, people are unlikely to tell you so, e.g., not recognising alternative explanations of change). If this conjecture is true, then it follows that positive feedback about a programme doesn’t tell us much; however, negative feedback would be highly informative. This follows from Bayes’ rule.

Play around with the probabilities here.

Sensitivity is the probability that someone will tell you the programme works, if it does actually work.

Specificity is the probability that someone will tell you the programme doesn’t work, if it doesn’t actually work.

Prevalence is the probability that programmes of the type you are asking about actually work, e.g., 0.5 would mean for a given type of programme you think the probability it does work is the same as the probability it doesn’t.

Positive predictive value (PPV) is the probability that the programme actually does work if someone tells you it does.

Negative predictive value (NPV) is the probability that the programme doesn’t actually work if someone tells you it doesn’t.

* This conjecture is at the bottom of the evidence hierarchy, whatever the opposite of a gold standard is. Balsa wood standard?