challenging TBE vs CIE binary – Page 2

Modelling theories of change – a look at Weiss (1972)

Periodically I listen to contemporary debates in evaluation and wonder how much progress there has been in the past fifty years. Here’s a picture from Carol Weiss (1972, Figure 3–3, p. 50; reprinted in 1996):

Weiss called it a model of a programme; now we would use the term programme theory or theory of change. About half of the next page (Weiss, 1972, p. 51) is filled with intriguing footnotes. The first notes the similarity between the diagram above and a statistical approach:

“As the similarity between Figure 3–3 and a path diagram suggests, path analysis is a useful way of estimating the strength of the linkages. When the necessary conditions for each path analysis are met, path coefficients will represent the magnitude of effects for each arrow.”

On path analysis, Weiss cites Blalock’s (1969) Theory construction from verbal to mathematical formulations. Structural equation modelling, path analysis plus confirmatory factor analysis, turned out to be a popular way to test theories in social science (see, e.g., MacCallum & Austin, 2000; Zyphur, Bonner, & Tay, 2023).

The same footnote also summarises the Automatic Interaction Detector (AID), an early example of a data-driven partitioning procedure:

“Another analytic device, particularly useful when one step in the chain is totally or largely conditional on the one before, is the Automatic Interaction Detector (AID). Unlike path analysis, this type of ‘branch’ analysis shows whether the program is producing strong effects for subsamples of the group. This is a matter of both theoretical and programmatic interest, because if there are differential effects, special programs can be developed to fit special groups.”

AID and its successors are data hungry and liable to overfitting data if misused; however, there is a rich literature on how to use them and stay out of trouble. Loh (2014) reviews fifty years of that literature, beginning with AID (p. 330):

“Despite their novelty, or perhaps owing to it, AID and THAID did not initially attract much interest in the statistics community. Einhorn (1972) showed by simulation that AID can severely overfit the data. Doyle (1973) pointed out that if two or more variables are highly correlated, at most one may appear in the tree structure. This problem of masking can lead to spurious conclusions about the relative importance of the variables. Bishop et al. (1975) criticized AID for ignoring the inherent sampling variability of the data. Around the same time though, the idea of recursive partitioning was gaining steam in the computer science and engineering communities as more efficient algorithms for carrying out the search for splits began to appear (Chou 1969; Henrichon & Fu, 1973; Meisel & Michalopoulos, 1977; Payne & Meisel, 1977; Sethi & Chatterjee, 1991).”

(Example newer algorithms are implemented in {rpart} in R.) Weiss’s footnote and these statistical approaches are clear examples of trying to tackle the “what works best for whom” problem, which for some reason (along with “context”) is now associated, like a Pavlovian response, with “realist” evaluation. We need to do better.

References

Blalock, H. M., Jr. (1969). Theory construction from verbal to mathematical formulations. Prentice-Hall Inc.

Loh, W. (2014). Fifty Years of Classification and Regression Trees. International Statistical Review, 82(3), 329–348.

MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.

Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.

Weiss, C. H. (1996). Excerpts from evaluation research: Methods of assessing program effectiveness. Evaluation Practice, 17(2), 173–175.

Zyphur, M. J., Bonner, C. V., & Tay, L. (2023). Structural equation modeling in organizational research: The state of our science and some proposals for its future. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 495-517.

SEM and TBE

Theory-based evaluation always included quant methods. It’s a huge relief to see more work reminding the community of that… (Nielsen & Lemire, 2024, p. 50):

“Structural Equation Modelling (SEM) is a particular variant of Multivariate Analysis, which is widely used in the social sciences. It provides a flexible framework for developing and analysing complex relationships among multiple variables that allow researchers to test the validity of theory using empirical models (Beran & Violato, 2010). Its ability to test theoretical models makes it especially useful for theory-informed evaluations that apply quantitative data. It has also been applied in realist evaluations (Von Thiele Schwarz et al., 2017) and other types of theory-based evaluation (Lemire et al., 2023). As with statistical multivariate analysis in general, it is particularly useful for quantitatively testing CMOs and hypothesised causal processes.”

References

Nielsen, S. B., & Lemire, S. (2024). Nothing as practical as an analytical strategy in realist evaluation. In A. Manzano & E. Williams, Realist Evaluation: Principles and Practice (pp. 40–56). Routledge.

Theory-based surveys, regression, interaction terms, and power calculations

“Outside of realist evaluations, a very common approach to the analysis of quantitative data is multivariate [they mean multivariable] regression analysis (in its various forms). [Context-Mechanism-Outcome] configurations, prima facie, lend themselves to regression analysis. Through surveys, data could be collected and the outcome could be used as a dependent variable. […] Given that context and mechanism need to interact to produce an outcome, the two variables that proxy them could be interacted with one another in the regression model (as well as being included by themselves). A statistically significant parameter on the interaction variable would identify whether both the C and M are needed together to achieve a higher (or lower) outcome. Moreover, other factors which might be important for determining the outcome could be included (controlled for) in the regression to ensure that the statistical finding is not driven by omitted variables.” (pp. 14-15)

“Realists are not particularly interested in average outcomes, but in explaining differences in the nature or extent of outcomes for different sub-groups of participants and/or in different contexts. While this can be explored in a regression framework by including context as an independent variable, it does require ample observations across all of the different contexts to provide meaningful results.” (p. 15)

Westhorp, G., & Feeny, S. (2024). Using surveys in realist evaluation. Evaluation Journal of Australasia, 1035719X241292083.

Let’s not replace “impact evaluation” with “contribution analysis”

Giel Ton has written an interesting blog post arguing that we should shift from talking about “impact evaluation” to “contribution analysis”, in the form devised by Mayne. Ton defines contribution analysis as following this process:

“make a good theory of change, identify key assumptions in this theory of change, and focus M&E and research on these key assumptions.”

My first thought was, this definition is remarkably broad! It’s the same as for any theory-based approach (or theory-driven – evaluation is awash with synonyms) where you start with a theory of change (ToC) and test and refine it. See, e.g., what Fitz-Gibbon and Morris (1975), Chen and Rossi (1980), and many others were proposing before Mayne. They all criticise “black box” approaches that lob methods at a programme before stopping to think what it might do and how, so I wondered what makes Ton’s (and/or Mayne’s) proposal different to these broad umbrella approaches that include all methods, mixed, blended, interwoven, shaken, or stirred – so long as a ToC is used throughout.

One recurring issue is people endlessly rocking up with yet another panacea: “Behold! ACME Programme™ will finally sort out your social problem!” Effect size, 0.1 SDs, if you’re lucky. A piece by Thomas Delahais (2023) helped clarify for me what’s different about the contribution analysis approach and how it helps address the panacea phenomenon: alternative explanations of change are treated as being as important as the new programme being investigated. That’s a fun challenge, for all evaluation approaches, qual, quant and RCTs included. For instance, we would design statistical analyses to tell us something about mechanisms that are involved in a range of activities in and around a new programme. We would explore how a new programme interacts with existing activities. These ideas sound very sensible to me – and are often done through implementation and process evaluation. But taking seriously the broader context and alternative explanations of change is much broader than contribution analysis. We might call the activity something like “evaluation”.

References

Chen, H.-T., & Rossi, P. H. (1980). The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science. Social Forces, 59, 106–122.

Delahais, T. (2023). Contribution Analysis. LIEPP Methods Brief, 44.

Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.

QCA versus regression trees

I have been pondering the relationship between qualitative comparative analysis (QCA) and regression trees – and obviously I’m not the first to do so. Here’s an interesting blog post by Rick Davies (2012), which includes a reanalysis of a QCA using a decision tree.

There’s also a 2014 EES talk:

Challenging the theory-based/counterfactual binary

Randomised controlled trials (RCTs) are an instance of the counterfactual kind of impact evaluation and contribution analysis of the theory-based kind. For example, the UK’s Magenta Book recommends using theory-based evaluation if you can’t find a comparison group (HM Treasury, 2020, p. 47). The founding texts on contribution analysis present it as using a non-counterfactual approach to causation (e.g., Mayne, 2019, pp. 173–4). In December 2023, I gave a talk to the UK Evaluation Society (UKES) exploring what happens if we challenge this theory-based/counterfactual binary. This post summarises what I said.

Good experiments and quasi-experiments are theory-based

Theory is required to select the variables used in RCTs and quasi-experimental designs (QEDs); that is, to decide what data to gather and include in analyses. We need to peek inside the “black box” of a programme to work out what we are evaluating and how. These variables include outcomes, moderators, mediators, competing exposures, and, in QEDs, confounders.

This diagram denotes a causal directed acyclic graph (causal DAG). Therapy is conjectured to lead to behavioural activation, which alleviates depression. Supportive friends are a “competing exposure” that can alleviate depression. Reading self-help texts is a confounder as it predicts both whether someone seeks therapy and their outcomes, so this (made up) model is of a quasi-experimental evaluation.

In QEDs, many different model structures are compatible with the data. The slogan “correlation doesn’t imply causation” can be reformulated as A causes B and B causes A are Markov equivalent if all we know is that A and B are correlated. The number of models that are Markov equivalent rises exponentially as the number of variables and statistical associations increases. Theory is needed to select between the models: the data alone cannot tell us.

Although RCTs give us an unbiased estimate of an average treatment effect, that is, the average of each individual’s unmeasurable treatment effect (difference between measured actual outcome and counterfactual outcome), they cannot tell us what that difference represents. To do that, we need a theory of the ingredients and processes in the two programmes being compared; for example, what are the similarities and differences between CBT and humanistic counselling or whatever “treatment as usual” is in practice.

Counterfactual evaluations do not need a comparison group

There is a long history of research on counterfactual reasoning in the absence of a control group. How should we determine the truth of a counterfactual such as “If Oswald had not killed Kennedy, no one else would have” (e.g., Adams, 1970)? Clearly we ponder statements like these without running violent RCTs. Another strand of research investigates how children, adults, and animals perform counterfactual reasoning in practice (e.g., Rafetseder, et al., 2010). This research on non-experimental counterfactual reasoning includes literature in evaluation, for instance by White (2010) and more recently Reichardt (2022).

Halpern (2016) introduces a formal framework for defining causal relationships and estimating counterfactual outcomes, regardless of the sources of evidence. The diagram below illustrates the simplest version of the framework where causes and effects are binary. Equations, annotating each node, define the causal relationships and allow counterfactual outcomes to be inferred.

This is one of the models I used to illustrate counterfactual inference without a comparison group. In the factual situation, Alex was feeling down, spoke to their counsellor, and then felt better. In this counterfactual scenario, we are exploring what would have happened if Alex hadn’t spoken to their counsellor. In this case, they would have spoken to their friend instead and again felt better. Although there is no difference between the factual and counterfactual outcome, Halpern’s framework allows us to infer that Alex speaking to the counsellor was an actual cause of Alex feeling better.

This provides a concrete illustration of why RCTs and QEDs are unnecessary to reason counterfactually; however, the causal model obviously must be correct for the correct conclusions to be drawn, so one might reasonably ask what sorts of evidence we can use to build these models and persuade people they are true. There is also hearty debate in the literature concerning what constitutes an “actual cause” and algorithms for determining this are fiddly to apply, even if it is assumed we have a true model of the causal relationships. I explored Halpern’s approach in a previous blog post.

Summary

The original definition of theory-based evaluation by Fitz-Gibbon and Morris (1975) included the full range of approaches, qualitative and quantitative – including RCTs and QEDs. Many others follow in the tradition. For instance, Weiss (1997, p. 512) cites path analysis (a kind of structural equation model) as being “conceptually compatible with TBE [theory-based evaluation] and has been used by evaluators”. Chen and Rossi (1980) explain how theory-based (what they term theory-driven) RCTs that include well-chosen covariates (competing exposures) yield more precise estimates of effects (reducing the probability of Type II error), even though those covariates are not needed to control Type I error. Counterfactual queries do not need a comparison group. They do need a model of how facts came about that can be modified to predict the counterfactual outcome.

Challenging the theory-based/counterfactual binary does not mean that all evaluations are the same. There can still be variation in the strength of evidence used to develop and test theories and how well theories withstand those tests. However, taking a more nuanced view of the differences and similarities between approaches leads to better evaluations.

References

Adams, E. W. (1970). Subjunctive and Indicative Conditionals. Foundations of Language, 6, 89–94.

Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302

Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.

Halpern, J. Y. (2016). Actual causality. The MIT press.

HM Treasury. (2020). Magenta Book.

Mayne, J. (2019). Revisiting contribution analysis. Canadian Journal of Program Evaluation, 34(2), 171–191.

Rafetseder, E., Cristi‐Vargas, R., & Perner, J. (2010). Counterfactual reasoning: Developing a sense of “nearest possible world”. Child Development, 81(1), 376-389.

Reichardt, C. S. (2022). The Counterfactual Definition of a Program Effect. American Journal of Evaluation, 43(2), 158–174.

Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501–524.

White, H. (2010). A contribution to current debates in impact evaluation. Evaluation, 16(2), 153–164.

Reclaiming the term “theory-based”

Excited to discover a triple of 2024 publications that use the broader conception of theory-based evaluation that includes trials and quasi-experiments, as the term was introduced by Fitz-Gibbon and Morris (1975) and used by, e.g., Chen and Rossi (1980), Coryn et al., (2011), Weiss (1997), Funnell and Rogers, P. J. (2011), Chen (2015), and many others.

I hope the next Magenta Book update has a more nuanced approach that includes RCTs and QEDs under theory-based, alongside, e.g., QCA and uses of Bayes’ rule to reason about qual evidence.

The new:

Bonell, C., Melendez-Torres, G. J., & Warren, E. (2024). Realist trials and systematic reviews: Rigorous, useful evidence to inform health policy. Cambridge University Press.

Matta, C., Lindvall, J., & Ryve, A. (2024). The Mechanistic Rewards of Data and Theory Integration for Theory-Based Evaluation. American Journal of Evaluation, 45(1), 110–132.

Schmidt, R. (2024). A graphical method for causal program attribution in theory-based evaluation. Evaluation, online first.

Key older texts:

Chen, H.-T., & Rossi, P. H. (1980). The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science. Social Forces, 59, 106–122.

Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302.

Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd edition). Sage Publications.

Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226.

Funnell, S. C., Rogers, P. J. (2011). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.

Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501–524.

TBE QEDs

‘In TBE [theory-based evaluation] practice […] theory as represented is not specific enough to support causal conclusions in inference […]. For example, in contribution analysis “causal assumptions” refer to a “causal package” consisting of the program intervention and a set of contextual conditions that together may explain an observed change in the outcome […]. In realist evaluation, the causal mechanisms that are triggered by the intervention are specified in “configuration” with their context and the outcome. Often, however, the causal structure of the configuration is not clear […]. Moreover, the main TBE approaches to inference do not have standard practices, conventions, for treating bias in evidence […].

‘TBE practitioners may borrow from other methods to test theoretical assumptions […]. Sometimes TBE employs regression analysis or quasi-experimental propensity score matching in inference (our running example in this article of an actual TBE program evaluation does so).’

Schmidt, R. (2024). A graphical method for causal program attribution in theory-based evaluation. Evaluation, online first.

Ignorance of history in evaluation

“Despite occasional statements that program theory is a new approach, its roots go back more than fifty years. […] The history of program theory evaluation is not one of a steady increase in understanding. Instead, many of the key ideas have been well articulated and then ignored or forgotten in descriptions of the approach. It is not unusual to have statements that demonstrate a lack of knowledge of previous empirical and theoretical developments, such as a call for proposals from the Agency for Healthcare Research and Quality (2008) that claimed that “‘theory-based evaluation’ is a relatively new approach” (p. 14).”

Sue C. Funnell and Patricia J. Rogers (2011, pp. 15–16). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.

What works for whom and in what contexts

“It is sometimes argued that we need rich qualitative data in order to find out not ‘what works’ but for whom and in what contexts. Anyone familiar with the design of experiments will agree. That question is answered by factorial designs with interaction terms… although main effects predominate in most educational datasets. If that last sentence seems like an unfamiliar concept, blame whoever taught you research methods and please make sure that your students are familiar with procedures that will underpin the thousands of controlled trials that education needs if it is to know that rather than simply guessing or asserting why.”

– Carol Taylor Fitz-Gibbon (2002). Knowing Why and Knowing That. Paper presented to the European Evaluation Conference in Seville, Spain. Emphasis and ellipsis in original.