quotations – Page 4 – Andi Fugard (∧⇒)

The problem with politics and evaluation

“The problem is that politics and evaluation are both intrinsically connected and intrinsically separated. We need independence if we’re to be competent, objective, and credible. However, independence can’t always be preserved without a fight; this doesn’t make us a lot of friends; and yet we need friends, partners, and teamwork to be effective in a political universe.”
—Eleanor Chelimsky (2008, p. 414). A Clash of Cultures: Improving the “Fit” Between Evaluative Independence and the Political Requirements of a Democratic Society. American Journal of Evaluation, 29(4), 400–415.

Time and potential outcomes

“For our purposes, the same physical object or person at a different time is a different unit.” (Guido Imbens and Donald Rubin, 2015, p. 4)

“I guess I wont send that note now, for the mind is such a new place, last night feels obsolete.” (Emily Dickinson, 1870)

References

Dickinson, E. (1870). Letter to Elizabeth Holland.

Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press.

“QCA exists in a nether region between qualitative and quantitative research”

‘Highlighting QCA’s reliance on two-element Boolean algebra and propositional logic brings the weaknesses of QCA into sharp relief. The distinction between QCA and conventional statistics is not a difference between “set-theoretic” methods and what QCA researchers call “non-set-theoretic” methods (Schneider and Wagemann 2012) […]. Conventional statistics are set-theoretic methods. […] What truly sets QCA apart from conventional statistics is the former’s embrace of the limitations of two-element Boolean algebra and propositional logic. QCA exists in a nether region between qualitative and quantitative research; it exhibits neither the depth of small-n qualitative research nor the power of quantitative research.’ (Clarke 2020, p. 553)

Clarke, K. A. (2020). Logical Constraints: The Limitations of QCA in Social Science Research. Political Analysis, 28(4), 552–568.

Theory-based surveys, regression, interaction terms, and power calculations

“Outside of realist evaluations, a very common approach to the analysis of quantitative data is multivariate [they mean multivariable] regression analysis (in its various forms). [Context-Mechanism-Outcome] configurations, prima facie, lend themselves to regression analysis. Through surveys, data could be collected and the outcome could be used as a dependent variable. […] Given that context and mechanism need to interact to produce an outcome, the two variables that proxy them could be interacted with one another in the regression model (as well as being included by themselves). A statistically significant parameter on the interaction variable would identify whether both the C and M are needed together to achieve a higher (or lower) outcome. Moreover, other factors which might be important for determining the outcome could be included (controlled for) in the regression to ensure that the statistical finding is not driven by omitted variables.” (pp. 14-15)

“Realists are not particularly interested in average outcomes, but in explaining differences in the nature or extent of outcomes for different sub-groups of participants and/or in different contexts. While this can be explored in a regression framework by including context as an independent variable, it does require ample observations across all of the different contexts to provide meaningful results.” (p. 15)

Westhorp, G., & Feeny, S. (2024). Using surveys in realist evaluation. Evaluation Journal of Australasia, 1035719X241292083.

Variables

“The view attributed to positivism, that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.”

Paley, J., & Lilford, R. (2011). Qualitative methods: An alternative view. BMJ, 342, d424.

Origin of jingle and jangle fallacies

‘To use an illustration given by Thorndike (1904, page 14), the expression “college student,” found so frequently in general discussions, covers a multitude of classes: male and female; part time, full time; extension students and those in residence; native, foreign, lower classmen, upper classmen, graduates; etc. In each connection the expression “college student” sounds the same, and thus we come to treat it as a single concept. Dr. Thorndike quotes Professor Aikins as describing this as the “jingle” fallacy because there is merely a verbal resemblance and no sufficient underlying factual similarity between the classes.

‘Equally contaminating to clear thinking is the use of two separate words or expressions covering in fact the same basic situation, but sounding different, as though they were in truth different. The doing of this latter the writer will call the “jangle” fallacy. “Achievement” and “intelligence” sound as though they were different; they have different “jangles,” and thus we treat them as though they were different in truth.’

– Truman Lee Kelley (1927, pp. 63-64)

References

Kelley, T. L. (1927). Interpretation of educational measurements. World Book Company.

The problem with tennis and Alan Turing’s solution

“Turing would rather suddenly develop a tremendous passion for some form of activity, or some study, and wish to devote a lot of his time and energy to it. On my first encounter with him at Bletchley Park he was obviously very much fascinated by chess problems. At BP he developed a real delight in playing tennis, and especially enjoyed playing doubles. He was very good up at the net, where his speed and good eye enabled him to make many effective interceptions. However, he was dissatisfied with his success rate: too often he intercepted a return from an opponent, but sent the ball into the net. Applying his remarkable thinking processes to a mundane problem, he reasoned as follows: ‘The problem is that, when intercepting, one has very little time to plan one’s stroke. The time available is a function of the tautness of the strings of my racquet. Therefore I must loosen the strings.’ And, being Alan Turing, he then carried out the necessary alterations to his racquet himself. At this point my recollection may be coloured by the great distance in time, but I seem to recall Turing turning up for his next game with a racquet somewhat resembling a fishing net. He was absolutely devastating, catching the ball in his racquet and delivering it wherever he chose—but plainly in two distinct operations and, therefore, illegally. He was soon persuaded to revert to a more orthodox racquet!” (Hilton, 2006, pp. 198–199)

References

Hilton, P. (2006). Living with Fish: Breaking Tunny in the Newmanry and the Testery. In J. B. Copeland (Ed.), Colossus (pp. 189–203). Oxford University Press.

Ioannidis and Psillos (2018) on mechanisms

“Mechanisms are causal pathways described in theoretical language that have certain functions; these descriptions can be enriched by offering more detailed or fine‐grained descriptions; the same mechanism can then be described at various levels using different theoretical vocabularies (e.g., cytological vs biochemical descriptions in the case of apoptosis); lastly, the descriptions of biomedically important mechanisms are often such that they contain specific causal information that can be used to make interventions for therapeutic purposes.” (Ioannidis & Psillos, 2018, pp. 1180–1)

Ioannidis, S., & Psillos, S. (2018). Mechanisms in practice: A methodological approach. Journal of Evaluation in Clinical Practice, 24(5), 1177–1183.

“No rigorous distinction is possible…”

“No rigorous distinction is possible between theory-driven evaluation (science), TBE [theory-based evaluation], theory-anchored evaluation, realistic evaluation, contribution analysis, evaluation based on a logic model, and a theory of change. I shall here refer pragmatically only to TBE as an umbrella term for all evaluation strategies and approaches where a core task is to construct and clarify a set of assumptions about why and how an intervention works explicitly in the form of a written or graphic description of a causal chain of events (the program theory).

“The term program theory is habitual and should not be restricted to programs only because a similar logic can be applied in evaluations of policies, projects, and many kinds of interventions (Rogers, 2007). In a similar vein, the term “theory” has rough edges. A theory used in TBE may refer to something more or less explicit and articulate, more or less abstract and formal, and more or less stakeholder based versus anchored in general social science theory.”

– Dahler-Larsen, P. (2018, p. 9). Theory-Based Evaluation Meets Ambiguity: The Role of Janus Variables. American Journal of Evaluation, 39(1), 6–23.