Tag: policy evaluation

A review of process tracing

I was interested to see a recent review of process tracing in health research (Johnson, Beach, & Al-Janabi, 2024). How are its users getting on…?

Of the 19 evaluations satisfying review criteria, only 2 got full marks:

These were praised for how they justified using process tracing, their complete and detailed description of methods, their use of multiple types of evidence, and how they considered and attempted to rule out alternative or counterfactual explanations of outcomes. Finally, they linked their findings back to the theory. I have added them to my reading stack.

The review’s conclusions are damning:

“… a majority of studies reported do not describe in-depth how they conduct their process trace, and fewer still support their claims consistently and comprehensively with verifiable and well organised evidence. This is somewhat counterintuitive given that one of the features of process tracing is to support the more robust depth of analysis to explore hypotheses and confirm or counter with supportive evidence. However, this is not just a feature of studies reporting process tracing in health. Applications of process tracing in policy studies and political science more generally have similar trends, with many studies providing little detail of their methodology or a mechanistic depiction of the theory.”

As ever, to be continued…

References

Alvarado, M., Penney, T. L., Unwin, N., Murphy, M. M., & Adams, J. (2021). Evidence of a health risk ‘signalling effect’ following the introduction of a sugar-sweetened beverage tax. Food Policy, 102.

Johnson, R., Beach, D., & Al-Janabi, H. (2024). How is process tracing applied in health research? A systematic scoping review. Social Science & Medicine, in press, journal pre-proof.

Ryu, S.-K., & Chung, S.-G. (2021). Korea’s Early COVID-19 Response: Findings and Implications. International Journal of Environmental Research and Public Health, 18(16), 8316.

Dodo bird verdicts

The Dodo bird verdict asserts that all effective psychological therapies are equally effective and work because of shared “common factors”. It is heavily debated.

Another Dodo bird verdict could also be said to apply to evaluation methods. I don’t believe that all methods are equally good; however, some do, and it might be handy to have a name for the position.

Here’s the section of Alice’s Adventures in Wonderland that inspired the term:

“What I was going to say,” said the Dodo in an offended tone, “was, that the best thing to get us dry would be a Caucus-race.”

“What is a Caucus-race?” said Alice; not that she wanted much to know, but the Dodo had paused as if it thought that somebody ought to speak, and no one else seemed inclined to say anything.

“Why,” said the Dodo, “the best way to explain it is to do it.” (And, as you might like to try the thing yourself, some winter day, I will tell you how the Dodo managed it.)

First it marked out a race-course, in a sort of circle, (“the exact shape doesn’t matter,” it said,) and then all the party were placed along the course, here and there. There was no “One, two, three, and away,” but they began running when they liked, and left off when they liked, so that it was not easy to know when the race was over. However, when they had been running half an hour or so, and were quite dry again, the Dodo suddenly called out “The race is over!” and they all crowded round it, panting, and asking, “But who has won?”

This question the Dodo could not answer without a great deal of thought, and it sat for a long time with one finger pressed upon its forehead (the position in which you usually see Shakespeare, in the pictures of him), while the rest waited in silence. At last the Dodo said, “Everybody has won, and all must have prizes.”

SEM and TBE

Theory-based evaluation always included quant methods. It’s a huge relief to see more work reminding the community of that… (Nielsen & Lemire, 2024, p. 50):

“Structural Equation Modelling (SEM) is a particular variant of Multivariate Analysis, which is widely used in the social sciences. It provides a flexible framework for developing and analysing complex relationships among multiple variables that allow researchers to test the validity of theory using empirical models (Beran & Violato, 2010). Its ability to test theoretical models makes it especially useful for theory-informed evaluations that apply quantitative data. It has also been applied in realist evaluations (Von Thiele Schwarz et al., 2017) and other types of theory-based evaluation (Lemire et al., 2023). As with statistical multivariate analysis in general, it is particularly useful for quantitatively testing CMOs and hypothesised causal processes.”

References

Nielsen, S. B., & Lemire, S. (2024). Nothing as practical as an analytical strategy in realist evaluation. In A. Manzano & E. Williams, Realist Evaluation: Principles and Practice (pp. 40–56). Routledge.

No evaluation finding will convince a bad programme developer that their programme does not work

No evaluation finding will convince a bad programme developer that their programme does not work and has no chance of ever working. At most, you may drive them to write a dozen poorly argued articles about epistemology.

With apologies to Asher Peres (2002, p. 45):

“No experimental evidence will convince a bad theorist that his statistical predictions are wrong. At most, you may drive him to bankruptcy if he is serious about betting.”

References

Peres, A. (2002). Quantum theory: concepts and methods. Kluwer Academic Publishers.

Sharp variance bounds for SATE: giving Mikhaeil and Green’s (2024) work a test drive

The standard ANCOVA-style way to estimate the sample average treatment effect (SATE) in an efficacy RCT yields an unbiased estimate of the SATE; however, the standard error is a little too big, which means the confidence interval is a little too wide. The reason for this is that the correct standard error depends on the correlation between potential outcomes, only one of which is realised for each participant depending on which experimental condition they ended up in, making that correlation impossible to calculate.

This problem was pointed out by Neyman (1923/1990). Aronow, Green, and Lee (2014) proved a result that puts bounds on the correlation based on the observed marginal distribution of outcomes; however, their approach only works for a model without any covariates.

I’m excited by a new preprint by Mikhaeil and Green (2024) that extends the approach to models with covariates and provides R code, in the {sharpvar} package.

I gave it a go with simulated data; knitted code and output over here.

NAO (2024), A planning and spending framework that enables long-term value for money

New National Audit Office (NAO) report with interesting analysis, examples of positive change such as the creation of the Evaluation Registry, and further recommendations regarding the role of evaluation in UK government planning and spending. For example:

“Data quality is a problem in government which makes monitoring both costs and value harder. […] Even where data are available, we see examples of a ‘good news culture’ in some major programmes and government bodies, that disincentivises transparent reporting of performance.” [2.27]

“Evaluation is sometimes perceived as apportioning blame, instead of as a neutral learning mechanism. If an evaluation shows a lack of progress, it might lead to a programme losing funding or being cancelled. This acts as a disincentive to evaluate.” [2.28]

“The previous government also committed to making the evaluation registry open to the public, although that has not yet occurred.” [2.66]

Theory-based surveys, regression, interaction terms, and power calculations

“Outside of realist evaluations, a very common approach to the analysis of quantitative data is multivariate [they mean multivariable] regression analysis (in its various forms). [Context-Mechanism-Outcome] configurations, prima facie, lend themselves to regression analysis. Through surveys, data could be collected and the outcome could be used as a dependent variable. […] Given that context and mechanism need to interact to produce an outcome, the two variables that proxy them could be interacted with one another in the regression model (as well as being included by themselves). A statistically significant parameter on the interaction variable would identify whether both the C and M are needed together to achieve a higher (or lower) outcome. Moreover, other factors which might be important for determining the outcome could be included (controlled for) in the regression to ensure that the statistical finding is not driven by omitted variables.” (pp. 14-15)

“Realists are not particularly interested in average outcomes, but in explaining differences in the nature or extent of outcomes for different sub-groups of participants and/or in different contexts. While this can be explored in a regression framework by including context as an independent variable, it does require ample observations across all of the different contexts to provide meaningful results.” (p. 15)

Westhorp, G., & Feeny, S. (2024). Using surveys in realist evaluation. Evaluation Journal of Australasia, 1035719X241292083.

See also challenging the binary.

Propensity score analysis with baseline measure of outcome

Mild headache in this new simulation study by Peter Austin of propensity score matching/weighting studies where you have a baseline measurement of the outcome variable – a useful thing to balance as it’s often highly correlated with outcome.

For ATE, the recommendations make sense: include the baseline in the propensity model and then, for what are often called doubly robust approaches, include baseline again as a covariate in the outcome model.

For ATT, the findings are not what I would have expected. Austin’s recommendation for best standard errors is to exclude the baseline from the propensity model and analyse change from baseline in the outcome model 😱:

“… recommendations for the analysis of RCTs with baseline measurements of the follow-up variable do not reflect perfectly what we observed in the context of the analysis of observational studies using propensity score methods. We found that the analysis of change from baseline using a propensity score that excluded the baseline value of the follow-up variable tended to result in the most precise estimates of treatment effect. However, we did observe that, when using weighting, the use of ANCOVA in conjunction with a propensity score model that included the baseline value of the follow-up variable tended to perform well.”

One to read properly to see what’s going on. Would be interested in your thoughts.

References

Austin, P. (2024). Propensity Score Analysis With Baseline and Follow-Up Measurements of the Outcome Variable. Pharmaceutical Statistics. Early view.

Analysing individually randomized group treatment trials with complex clustering in nested and crossed designs

Interesting paper by Jonathan Moyer, Fan Li, and others on the best ways to analyse individually randomised trials in which (1) participants share professionals such as therapists and case workers, (2) participants may interact with multiple professionals over the course of the trial, and (3) the nesting structure may differ between experimental conditions.

Being brave with causal interpretations

One of the fabulous things about evaluation is that we are allowed to (or, rather, required to!) make causal claims from a wide variety of evidence, not only experiments. Causal estimands, defined in terms of potential outcomes, are the norm for quantitative impact evaluations. Causal DAGs are becoming more popular too and one of my favourite tools for thinking through quasi-experimental designs.

Not all fields have it so good.

Julia Rohrer has written a great blog post, following a series of journal papers, urging personality psychologists to be upfront with their causal claims. It’s challenging to persuade and I think interesting as a sociology of science case study. As Rohrer summarises, ‘Being vague about the desired interpretation may maintain some degree of plausible deniability and potentially offload the error to others (“oh, you shouldn’t have interpreted our numbers like that!1”). But honestly, that’s just a cowardly way to do science. Life is too short for that.’

Let’s do statistics the other way around