Tag: policy evaluation

Is a theory of change different to a logic model? Depends

Evaluation purists: It’s only a Logic Model if it comes from the W. K. Kellogg Foundation region of Michigan, otherwise it’s just sparkling boxes and arrows.

Alternatively:

“A program theory is an explicit theory or model of how an intervention contributes to a set of specific outcomes through a series of intermediate results. The theory needs to include an explanation of how the program’s activities contribute to the results, not simply a list of activities followed by the results, with no explanation of how these are linked, apart from a mysterious arrow.” (Funnell & Rogers, 2011, p. 31)

“A program theory is usually displayed in a diagram called a logic model.” (Funnell & Rogers, 2011, p. 32)

“… sometimes the terms logic model and theory of change have been distinguished in particular ways that are different to the ways we are using the terms here. For example Heléne Clark, from ActKnowledge, and Andrea Anderson, from the Aspen Institute Roundtable on Community Change, who employ program theory extensively, have used these terms to make the distinction between ways of representing program theory that we have labeled pipeline logic models and outcomes chain logic models (Clark and Anderson, 2004). Mhairi Mackenzie and Avril Blamey (2005) have used theories of change to refer specifically to the type of logic model advocated by the Aspen Institute […]. The key message here is to define your terms carefully and ask others to do so as well. It cannot be assumed that you mean the same thing when you use the same term or that you mean something different when you use a different term.” (Funnell & Rogers, 2011, p. 26)

Funnell, S. C., & Rogers, P. J. (2011). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.

Ignorance of history in evaluation

“Despite occasional statements that program theory is a new approach, its roots go back more than fifty years. […] The history of program theory evaluation is not one of a steady increase in understanding. Instead, many of the key ideas have been well articulated and then ignored or forgotten in descriptions of the approach. It is not unusual to have statements that demonstrate a lack of knowledge of previous empirical and theoretical developments, such as a call for proposals from the Agency for Healthcare Research and Quality (2008) that claimed that “‘theory-based evaluation’ is a relatively new approach” (p. 14).”

Sue C. Funnell and Patricia J. Rogers (2011, pp. 15–16). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.

On the term “randomista”

Sophie Webber and Carolyn Prouse (2018, p. 169, footnote 1) write:

“Randomistas is a slang term used by critics to describe proponents of the RCT methodology. It is almost certainly a gendered, derogatory term intended to flippantly dismiss experimental economists and their success, particularly Esther Duflo, one of the most successful experts on randomization.” [Emphasis original.]

References

Sophie Webber and Carolyn Prouse (2018). The New Gold Standard: The Rise of Randomized Control Trials and Experimental Development. Economic Geography, 94(2), 166–187.

Components and delivery formats of therapy for chronic insomnia

Interesting meta-analysis by Furukawa et al. (in press) of 241 trials, aiming to work out what components of therapy lead to better outcomes for people with chronic insomnia.

References

Furukawa Y., Sakata M., Yamamoto R., et al. Components and Delivery Formats of Cognitive Behavioral Therapy for Chronic Insomnia in Adults: A Systematic Review and Component Network Meta-Analysis. JAMA Psychiatry. Published online January 17, 2024.

Degtiar & Rose (2023) – A Review of Generalizability and Transportability

“This article presents a framework for addressing external validity bias, including a synthesis of approaches for generalizability and transportability, and the assumptions they require, as well as tests for the heterogeneity of treatment effects and differences between study and target populations.”

References

Degtiar, I., & Rose, S. (2023). A Review of Generalizability and Transportability. Annual Review of Statistics and Its Application, 10(1), 501–524.

Intent-to-fail treatments

Westen et al. (2004, p. 651) on “intent-fo-fail” conditions in randomised controlled trials comparing two interventions with each other:

‘Researchers should also exercise caution in labeling control treatments not constructed to maximize their efficacy ([…] what might be called intent-to-fail conditions) with brand names that are readily confused with genuine treatments and create sleeper effects in the literature. For example […] to test the efficacy of CBT for bulimia, Garner et al. (1993) developed a treatment they called supportive– expressive therapy, an abbreviated treatment described as nondirective and psychodynamically inspired, in which clinicians were forbidden to discuss the target symptoms with the patient and were instead instructed to reflect them back to the patient. Such a practice is not in fact characteristic of psychodynamic therapies for eating disorders (e.g., Bruch, 1973) and is analogous to a researcher creating a cognitive therapy comparison condition in which the therapist is instructed to say, “That’s irrational,” every time a patient tries to discuss the symptom.’

References

Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The Empirical Status of Empirically Supported Psychotherapies: Assumptions, Findings, and Reporting in Controlled Clinical Trials. Psychological Bulletin, 130(4), 631–663.

What works for whom?

It can be interesting to venture outside the general evaluation literature. Here’s Gordon Paul (1967, p. 111) on what works for whom:

“What is the appropriate question to be asked of outcome research? In all its complexity, the question towards which all outcome research should ultimately be directed is the following: What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances?”

References

Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31(2), 109–118.

 

What works for whom and in what contexts

“It is sometimes argued that we need rich qualitative data in order to find out not ‘what works’ but for whom and in what contexts. Anyone familiar with the design of experiments will agree. That question is answered by factorial designs with interaction terms… although main effects predominate in most educational datasets. If that last sentence seems like an unfamiliar concept, blame whoever taught you research methods and please make sure that your students are familiar with procedures that will underpin the thousands of controlled trials that education needs if it is to know that rather than simply guessing or asserting why.”

– Carol Taylor Fitz-Gibbon (2002). Knowing Why and Knowing That. Paper presented to the European Evaluation Conference in Seville, Spain. Emphasis and ellipsis in original.

The first definition of something named “theory-based evaluation”

“A theory-based evaluation of a program is one in which the selection of program features to evaluate is determined by an explicit conceptualization of the program in terms of a theory […] which attempts to explain how the program produces the desired effects. The theory might be psychological […] or social psychological […] or philosophical […]. The essential characteristic is that the theory points out a causal relationship between a process A and an outcome B.”

– Carol Taylor Fitz-Gibbon and Lynn Lyons Morris (1975)

References

Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.

Worthen, B. R. (1996). Editor’s Note: The Origins of Theory-Based Evaluation. Evaluation Practice, 17(2), 169–171. This comment traces the path back to Fitz-Gibbon and Morris (1975).