Ioannidis and Psillos (2018) on mechanisms

“Mechanisms are causal pathways described in theoretical language that have certain functions; these descriptions can be enriched by offering more detailed or fine‐grained descriptions; the same mechanism can then be described at various levels using different theoretical vocabularies (e.g., cytological vs biochemical descriptions in the case of apoptosis); lastly, the descriptions of biomedically important mechanisms are often such that they contain specific causal information that can be used to make interventions for therapeutic purposes.” (Ioannidis & Psillos, 2018, pp. 1180–1)

Ioannidis, S., & Psillos, S. (2018). Mechanisms in practice: A methodological approach. Journal of Evaluation in Clinical Practice, 24(5), 1177–1183.

“No rigorous distinction is possible…”

“No rigorous distinction is possible between theory-driven evaluation (science), TBE [theory-based evaluation], theory-anchored evaluation, realistic evaluation, contribution analysis, evaluation based on a logic model, and a theory of change. I shall here refer pragmatically only to TBE as an umbrella term for all evaluation strategies and approaches where a core task is to construct and clarify a set of assumptions about why and how an intervention works explicitly in the form of a written or graphic description of a causal chain of events (the program theory).

“The term program theory is habitual and should not be restricted to programs only because a similar logic can be applied in evaluations of policies, projects, and many kinds of interventions (Rogers, 2007). In a similar vein, the term “theory” has rough edges. A theory used in TBE may refer to something more or less explicit and articulate, more or less abstract and formal, and more or less stakeholder based versus anchored in general social science theory.”

– Dahler-Larsen, P. (2018, p. 9). Theory-Based Evaluation Meets Ambiguity: The Role of Janus Variables. American Journal of Evaluation, 39(1), 6–23.

The interocular traumatic test

“The preceding paragraph illustrates a procedure that statisticians of all schools find important but elusive. It has been called the interocular traumatic test; you know what the data mean when the conclusion hits you between the eyes. The interocular traumatic test is simple, commands general agreement, and is often applicable; well-conducted experiments often come out that way. But the enthusiast’s interocular trauma may be the skeptic’s random error. A little arithmetic to verify the extent of the trauma can yield great peace of mind for little cost.” (Edwards, Lindman, & Savage, 1963, p.217, who attribute the “test” to J. Berkson, personal comm., 14 July 1958)

Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70(3), 193.

TBE QEDs

‘In TBE [theory-based evaluation] practice […] theory as represented is not specific enough to support causal conclusions in inference […]. For example, in contribution analysis “causal assumptions” refer to a “causal package” consisting of the program intervention and a set of contextual conditions that together may explain an observed change in the outcome […]. In realist evaluation, the causal mechanisms that are triggered by the intervention are specified in “configuration” with their context and the outcome. Often, however, the causal structure of the configuration is not clear […]. Moreover, the main TBE approaches to inference do not have standard practices, conventions, for treating bias in evidence […].

‘TBE practitioners may borrow from other methods to test theoretical assumptions […]. Sometimes TBE employs regression analysis or quasi-experimental propensity score matching in inference (our running example in this article of an actual TBE program evaluation does so).’

Schmidt, R. (2024). A graphical method for causal program attribution in theory-based evaluation. Evaluation, online first.

 

Is a theory of change different to a logic model? Depends

Evaluation purists: It’s only a Logic Model if it comes from the W. K. Kellogg Foundation region of Michigan, otherwise it’s just sparkling boxes and arrows.

Alternatively:

“A program theory is an explicit theory or model of how an intervention contributes to a set of specific outcomes through a series of intermediate results. The theory needs to include an explanation of how the program’s activities contribute to the results, not simply a list of activities followed by the results, with no explanation of how these are linked, apart from a mysterious arrow.” (Funnell & Rogers, 2011, p. 31)

“A program theory is usually displayed in a diagram called a logic model.” (Funnell & Rogers, 2011, p. 32)

“… sometimes the terms logic model and theory of change have been distinguished in particular ways that are different to the ways we are using the terms here. For example Heléne Clark, from ActKnowledge, and Andrea Anderson, from the Aspen Institute Roundtable on Community Change, who employ program theory extensively, have used these terms to make the distinction between ways of representing program theory that we have labeled pipeline logic models and outcomes chain logic models (Clark and Anderson, 2004). Mhairi Mackenzie and Avril Blamey (2005) have used theories of change to refer specifically to the type of logic model advocated by the Aspen Institute […]. The key message here is to define your terms carefully and ask others to do so as well. It cannot be assumed that you mean the same thing when you use the same term or that you mean something different when you use a different term.” (Funnell & Rogers, 2011, p. 26)

Funnell, S. C., & Rogers, P. J. (2011). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.

Ignorance of history in evaluation

“Despite occasional statements that program theory is a new approach, its roots go back more than fifty years. […] The history of program theory evaluation is not one of a steady increase in understanding. Instead, many of the key ideas have been well articulated and then ignored or forgotten in descriptions of the approach. It is not unusual to have statements that demonstrate a lack of knowledge of previous empirical and theoretical developments, such as a call for proposals from the Agency for Healthcare Research and Quality (2008) that claimed that “‘theory-based evaluation’ is a relatively new approach” (p. 14).”

Sue C. Funnell and Patricia J. Rogers (2011, pp. 15–16). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.

Something other than gross self-indulgence

“More than anything, I’m excited. I’m excited to see how life is going to be different for the queer, trans, and even cis kids too, growing up in a world that has more language for gender variance. I’m excited to find out what sort of lives they will lead, from the genderqueer activists in the audience at my last reading to the barista with the orange mohawk who handed me the cup of tea I’m clutching for dear life as I write alone in this café, trying to believe that writing this piece is something other than gross self-indulgence.

“The barista is wearing two name badges. One says their name; the other one says, in thick chalk capitals, I am not a girl. My pronouns are They/Them.”

– Laurie Penny (2015), How To Be A Genderqueer Feminist

On the term “randomista”

Sophie Webber and Carolyn Prouse (2018, p. 169, footnote 1) write:

Randomistas is a slang term used by critics to describe proponents of the RCT methodology. It is almost certainly a gendered, derogatory term intended to flippantly dismiss experimental economists and their success, particularly Esther Duflo, one of the most successful experts on randomization.” [Emphasis original.]

References

Sophie Webber and Carolyn Prouse (2018). The New Gold Standard: The Rise of Randomized Control Trials and Experimental Development. Economic Geography, 94(2), 166–187.

Philip Larkin on how to write poetry

Philip Larkin interview excerpt in Paris Review (Issue 84, Summer 1982):

Yes, I was afraid you’d ask about writing. Anything I say about writing poems is bound to be retrospective, because in fact I’ve written very little since moving into this house, or since High Windows, or since 1974, whichever way you like to put it. But when I did write them, well, it was in the evenings, after work, after washing up (I’m sorry: you would call this “doing the dishes”). It was a routine like any other. And really it worked very well: I don’t think you can write a poem for more than two hours. After that you’re going round in circles, and it’s much better to leave it for twenty-four hours, by which time your subconscious or whatever has solved the block and you’re ready to go on.

The best writing conditions I ever had were in Belfast, when I was working at the University there. Another top-floor flat, by the way. I wrote between eight and ten in the evenings, then went to the University bar till eleven, then played cards or talked with friends till one or two. The first part of the evening had the second part to look forward to, and I could enjoy the second part with a clear conscience because I’d done my two hours. I can’t seem to organize that now.