“One day I asked a passionate and healthy young woman: ‘How can one find out if one is in love? It is also possible to want to take your mother in your arms and kiss her, isn’t it?’ She explained: ‘When you kiss your lover, you want him to die of your kisses and you want to disappear into him (she used another very characteristic word) etc.,’ – there is an infinity of expressions. The most elevated instincts always present themselves as murderous instincts. To begin with there is the little brother who calls the kitten: ‘Puss! Puss! My little darling! I love you! I am going to torture you and make you die!’; besides there is the well-known fact that lovers love to tease and torment each other, and moreover there is martyrdom and crimes of passion. Everywhere you see the pleasure involved in destroying and dying.” (Sabina Spielrein, 2001, p. 160)
Spielrein, S. (2001). Unedited extracts from a diary (1906/1907?). Journal of Analytical Psychology, 46, 155–171.
George: You asked how it works. To be with someone in this business. This is how. You each know what you know, and you know what you’ll do, and you never discuss certain things again. I watch her. I assume she watches me. If she’s in trouble, even of her own making, I will do everything in my power to extricate her. No matter what that means. You understand?
“A starting point for the QuIP [Qualitative Impact Protocol] is the premise that those who were intended to benefit from an intervention know a great deal about what has caused and affected changes in their (and their households’) lives in the recent past, and what has influenced their active decisions to start or stop doing certain activities. Relying on the narrative testimonies of intended beneficiaries removes the need for an independent counterfactual based on interviews with a control or comparison group. This is because comparisons between what happened and what would have happened otherwise are embedded in causal claims within the narrative – our thinking and language is laden with ways of doing this (e.g. through use of the conditional tense and the many ways we can answer ‘why’ questions). Collected with care, narrative accounts are full of explicit and latent counterfactuals; the task is to identify and interpret them. […] We refer to reliance on respondents to provide evidence of the causal chain puzzle themselves as self-reported attribution, and we distinguish it from statistically inferred attribution that generally relies on exposure variation, including comparing treatment and control groups.” (Copestake, Morsink, & Remnant, 2019, pp. 7–8.)
Copestake, J., Morsink, M., & Remnant, F. (2019). Attributing Development Impact. Practical Action Publishing Ltd. https://doi.org/10.3362/9781780447469
“There is relatively little written about sexual orientation in analytic philosophy journals. However, there is a ton of relevant work on sexual orientation in non-analytic or non-philosophy journals. Philosophers, as Eric Schwitzgebel has pointed out, don’t like to cite. Citation rates in philosophy are very low. Combine this with the fact that there’s a lot of social philosophy that has existed prior to the social turn, and we have a recipe for philosophical Columbusing—the act of pretending to discover (or mistakenly thinking that one has discovered) a philosophical insight that has already existed.”
“Consider a carefully controlled nonrandomized study – a study in which there are no obviously important prior variables that systematically differ in the E trials and the C trials. In such a study, there is a real sense in which a claim of ‘subjective randomization’ can be made. For example, if the study was composed of carefully matched pairs of trials, there might be a very defensible belief that within each matched pair each unit was equally likely to receive E as C in the sense that if you were shown the units without being told which received E, only half the time would you correctly guess which received E.”
“Let’s pause for a moment and ask a very basic and very important question: what is theory? We start by answering in the negative: the production of theory is not necessarily a scholastic exercise; the site of production of theory is not necessarily the ivory towers of academia; the producers of theory are not necessarily academically trained personnel in possession of qualifications that bestow on them the status of ‘officially approved theorists’.
“Positively, we would argue that the building blocks of the production of theory are the efforts of ordinary people to make sense of their social experience; the sites of production of theory are, then, everywhere such activity may take place; the producers of theory are – potentially – everyone who engages in reflection on their experiences so as to develop new and improved practices to problematic aspects of that experience.”
Periodically I listen to contemporary debates in evaluation and wonder how much progress there has been in the past fifty years. Here’s a picture from Carol Weiss (1972, Figure 3–3, p. 50; reprinted in 1996):
Weiss called it a model of a programme; now we would use the term programme theory or theory of change. About half of the next page (Weiss, 1972, p. 51) is filled with intriguing footnotes. The first notes the similarity between the diagram above and a statistical approach:
“As the similarity between Figure 3–3 and a path diagram suggests, path analysis is a useful way of estimating the strength of the linkages. When the necessary conditions for each path analysis are met, path coefficients will represent the magnitude of effects for each arrow.”
On path analysis, Weiss cites Blalock’s (1969) Theory construction from verbal to mathematical formulations. Structural equation modelling, path analysis plus confirmatory factor analysis, turned out to be a popular way to test theories in social science (see, e.g., MacCallum & Austin, 2000; Zyphur, Bonner, & Tay, 2023).
The same footnote also summarises the Automatic Interaction Detector (AID), an early example of a data-driven partitioning procedure:
“Another analytic device, particularly useful when one step in the chain is totally or largely conditional on the one before, is the Automatic Interaction Detector (AID). Unlike path analysis, this type of ‘branch’ analysis shows whether the program is producing strong effects for subsamples of the group. This is a matter of both theoretical and programmatic interest, because if there are differential effects, special programs can be developed to fit special groups.”
AID and its successors are data hungry and liable to overfitting data if misused; however, there is a rich literature on how to use them and stay out of trouble. Loh (2014) reviews fifty years of that literature, beginning with AID (p. 330):
“Despite their novelty, or perhaps owing to it, AID and THAID did not initially attract much interest in the statistics community. Einhorn (1972) showed by simulation that AID can severely overfit the data. Doyle (1973) pointed out that if two or more variables are highly correlated, at most one may appear in the tree structure. This problem of masking can lead to spurious conclusions about the relative importance of the variables. Bishop et al. (1975) criticized AID for ignoring the inherent sampling variability of the data. Around the same time though, the idea of recursive partitioning was gaining steam in the computer science and engineering communities as more efficient algorithms for carrying out the search for splits began to appear (Chou 1969; Henrichon & Fu, 1973; Meisel & Michalopoulos, 1977; Payne & Meisel, 1977; Sethi & Chatterjee, 1991).”
(Example newer algorithms are implemented in {rpart} in R.) Weiss’s footnote and these statistical approaches are clear examples of trying to tackle the “what works best for whom” problem, which for some reason (along with “context”) is now associated, like a Pavlovian response, with “realist” evaluation. We need to do better.
References
Blalock, H. M., Jr. (1969). Theory construction from verbal to mathematical formulations. Prentice-Hall Inc.
MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.
Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall, Inc.
Weiss, C. H. (1996). Excerpts from evaluation research: Methods of assessing program effectiveness. Evaluation Practice, 17(2), 173–175.
Zyphur, M. J., Bonner, C. V., & Tay, L. (2023). Structural equation modeling in organizational research: The state of our science and some proposals for its future. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 495-517.
“Within the academy, the word theory has a lot of capital. I have always been interested in how the word theory itself is distributed; how some materials are understood as theory and not others. […] As a student of theory, I learned that theory is used to refer to a rather narrow body of work. Some work becomes theory because it refers to other work that is known as theory. A citational chain is created around theory: you become a theorist by citing other theorists that cite other theorists.” (Ahmed, 2017, p. 8)
Reference
Ahmed, S. (2017). Living a feminist life. Duke University Press.
