Blog 🎤

Modelling theories of change – a look at Weiss (1972)

Periodically I listen to contemporary debates in evaluation and wonder how much progress there has been in the past fifty years. Here’s a picture from Carol Weiss (1972, Figure 3–3, p. 50; reprinted in 1996):

Weiss called it a model of a programme; now we would use the term programme theory or theory of change. About half of the next page (Weiss, 1972, p. 51) is filled with intriguing footnotes. The first notes the similarity between the diagram above and a statistical approach:

“As the similarity between Figure 3–3 and a path diagram suggests, path analysis is a useful way of estimating the strength of the linkages. When the necessary conditions for each path analysis are met, path coefficients will represent the magnitude of effects for each arrow.”

On path analysis, Weiss cites Blalock’s (1969) Theory construction from verbal to mathematical formulations. Structural equation modelling, path analysis plus confirmatory factor analysis, turned out to be a popular way to test theories in social science (see, e.g., MacCallum & Austin, 2000; Zyphur, Bonner, & Tay, 2023).

The same footnote also summarises the Automatic Interaction Detector (AID), an early example of a data-driven partitioning procedure:

“Another analytic device, particularly useful when one step in the chain is totally or largely conditional on the one before, is the Automatic Interaction Detector (AID). Unlike path analysis, this type of ‘branch’ analysis shows whether the program is producing strong effects for subsamples of the group. This is a matter of both theoretical and programmatic interest, because if there are differential effects, special programs can be developed to fit special groups.”

AID and its successors are data hungry and liable to overfitting data if misused; however, there is a rich literature on how to use them and stay out of trouble. Loh (2014) reviews fifty years of that literature, beginning with AID (p. 330):

“Despite their novelty, or perhaps owing to it, AID and THAID did not initially attract much interest in the statistics community. Einhorn (1972) showed by simulation that AID can severely overfit the data. Doyle (1973) pointed out that if two or more variables are highly correlated, at most one may appear in the tree structure. This problem of masking can lead to spurious conclusions about the relative importance of the variables. Bishop et al. (1975) criticized AID for ignoring the inherent sampling variability of the data. Around the same time though, the idea of recursive partitioning was gaining steam in the computer science and engineering communities as more efficient algorithms for carrying out the search for splits began to appear (Chou 1969; Henrichon & Fu, 1973; Meisel & Michalopoulos, 1977; Payne & Meisel, 1977; Sethi & Chatterjee, 1991).”

(Example newer algorithms are implemented in {rpart} in R.) Weiss’s footnote and these statistical approaches are clear examples of trying to tackle the “what works best for whom” problem, which for some reason (along with “context”) is now associated, like a Pavlovian response, with “realist” evaluation. We need to do better.

References

Blalock, H. M., Jr. (1969). Theory construction from verbal to mathematical formulations. Prentice-Hall Inc.

Loh, W. (2014). Fifty Years of Classification and Regression Trees. International Statistical Review, 82(3), 329–348.

MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.

Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.

Weiss, C. H. (1996). Excerpts from evaluation research: Methods of assessing program effectiveness. Evaluation Practice, 17(2), 173–175.

Zyphur, M. J., Bonner, C. V., & Tay, L. (2023). Structural equation modeling in organizational research: The state of our science and some proposals for its future. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 495-517.


Citational chains

“Within the academy, the word theory has a lot of capital. I have always been interested in how the word theory itself is distributed; how some materials are understood as theory and not others. […] As a student of theory, I learned that theory is used to refer to a rather narrow body of work. Some work becomes theory because it refers to other work that is known as theory. A citational chain is created around theory: you become a theorist by citing other theorists that cite other theorists.” (Ahmed, 2017, p. 8)

Reference

Ahmed, S. (2017). Living a feminist life. Duke University Press.

Quant methods aren’t positivist.

A paragraph from the archive (Fugard, 2020, p. 525–6):

“Theories in social science are underdetermined by evidence. They are produced by people who have biases and differing values, so two theorists may interpret the same evidence in different ways. Quantitative research is no exception since researchers have to choose what to measure and how to model it. For non-experimental data, many causal models are mathematically equivalent, so assumptions beyond the data are required (Pearl, 1995). Although there are criteria for preferring one theory over another, such as accuracy, simplicity, and scope, applying them is not always straightforward, even in natural sciences (Kuhn, 1977).”

References

Fugard, A. (2020). Should trans people be postmodernist in the streets but positivist in the spreadsheets? A reply to Sullivan. International Journal of Social Research Methodology, 23, 525–531.

Kuhn, T. S. (1977). Objectivity, value judgment, and theory choice. In The essential tension: Selected studies in scientific tradition and change (pp. 320–339). The University of Chicago Press.

Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4), 669–688.