Tag: quotations

Carol Fitz-Gibbon (1938 – 2017), author of first description of theory-based evaluation, on importance of RCTs

“[…] I produced the first description of theory based evaluation […]. The point of theory based evaluation is to see, firstly, to what extent the theory is being implemented and, secondly, if the predicted outcomes then follow. It is particularly useful as an interim measure of implementation when the outcomes cannot be measured until much later. But most (if not all) theories in social science are only sets of persuasively stated hypotheses that provide a temporary source of guidance. In order to see if the hypotheses can become theories one must measure the extent to which the predicted outcomes are achieved. This requires randomised controlled trials. Even then the important point is to establish the direction and magnitude of the causal relation, not the theory. Many theories can often fit the same data.”

Fitz-Gibbon, C. T. (2002). Researching outcomes of educational interventions. BMJ, 324(7346), 1155.

Asking counterfactuals to think about controversial policies

“As an anecdote: one of us lives in a 1930s neighbourhood. The local municipality proposed converting a road connecting the neighbourhood to other neighbourhoods from a two-direction road to a one-way road. This would apply to motorized traffic only, not to bicycles. Many people protested because of the detour they would have to take by car. We asked the counterfactual question: suppose the city would have introduced the one-way road decades ago, would they then heavily support a policy to make it a two-direction road? Several people we asked did not know, the main reason for not supporting that change being that it would result in more traffic, noise and pollution, and a reduction in safety.”

– Van Wee et al. (2023, p. 84)

Reference

Van Wee, B., Annema, J. A., & Van Barneveld, S. (2023). Controversial policies: Growing support after implementation. A discussion paper. Transport Policy, 139, 79–86.

Distinguishing between = and <-

‘Perhaps because of the use of = for assignment in FORTRAN […] assignment is often read as “x equals E”. This causes great confusion. The first author learned to distinguish between = and := while giving a lecture in Marktoberdorf, Germany, in 1975. At one point, he wrote “:=” on the board but pronounced it “equals”. Immediately, the voice of Edsger W. Dijkstra boomed from the back of the room: “becomes!”. After a disconcerted pause, the first author said, “Thank you; if I make the same mistake again, please let me know.”, and went on. Once more during the lecture the mistake was made, followed by a booming “becomes” and a “Thank you”. The first author has never made that mistake again! The second author, having received his undergraduate education at Cornell, has never experienced this difficulty.’

– David Gries and Fred B. Schneider (1993, p. 17, footnote 6) [A logical approach to discrete math. Springer.]

R lets you use both = and <- for assignment, FYI.

What is a confounder?

‘We thus proposed that a pre-exposure covariate C be considered a confounder for the effect of A on Y if there exists a set of covariates X such that the effect of the exposure on the outcome is unconfounded conditional on (X, C) but for no proper subset of (X, C) is the effect of the exposure on the outcome unconfounded given the subset.’ (VanderWeele & Shpitser, 2013, p. 215)

VanderWeele, T. J., & Shpitser, I. (2013). On the definition of a confounder. The Annals of Statistics, 41(1), 196–220.

Superdeterminism

Bell (1981, C2-57):

“… it may be that it is not permissible to regard the experimental settings a and b in the analyzers as independent variables, as we did. We supposed them in particular to be independent of the supplementary [a.k.a. hidden] variables λ, in that a and b could be changed without changing the probability distribution ρ(λ). Now even if we have arranged that a and b are generated by apparently random radioactive devices, housed in separate boxes and thickly shielded, or by Swiss national lottery machines, or by elaborate computer programmes, or by apparently free willed experimental physicists, or by some combination of all of these, we cannot be sure that a and b are not significantly influenced by the same factors λ that influence A and B [measurement outcomes]. But this way of arranging quantum mechanical correlations would be even more mind boggling that one in which causal chains go faster than light. Apparently separate parts of the world would be deeply and conspiratorially entangled, and our apparent free will would be entangled with them.”

Hance and Hossenfelder (2022, p. 1382) on the assumption of statistical independence of supplementary/hidden variables and experimental settings:

“Types of hidden variables theories which violate statistical independence include those which are superdeterministic, retrocausal, and supermeasured. Some have dismissed them on metaphysical grounds, by associating a violation of statistical independence with the existence of ‘free will’ or ‘free choice’ and then arguing that these are not assumptions we should give up.

“It is, in hindsight, difficult to understand how this association came about. We believe it originated in the idea that a correlation between the hidden variables and the measurement setting would somehow prevent the experimentalist from choosing the setting to their liking. However, this is mistaking a correlation with a causation. And any serious philosophical discussion of free will acknowledges that human agency is of course constrained by the laws of nature anyway.”

References

Bell, J. S. (1981). Bertlmann’s socks and the nature of reality. Le Journal de Physique Colloques, 42(C2), C2-41-C2-62. Reprinted in Bell (2004).

Bell, J. S. (2004). Speakable and unspeakable in quantum mechanics: Collected papers on quantum philosophy (2nd ed.). Cambridge University Press.

Hance, J. R., & Hossenfelder, S. (2022). Bell’s theorem allows local theories of quantum mechanics. Nature Physics, 18(12), 1382.  [Preprint]

 

A cynical view of SEMs

It is all too common for a box and arrow diagram to be cobbled together in an afternoon and christened a “theory of change”. One formalised version of such a diagram is a structural equation model (SEM), the arrows of which are annotated with coefficients estimated using data. Here is John Fox (2002) on SEM and informal boxology:

“A cynical view of SEMs is that their popularity in the social sciences reflects the legitimacy that the models appear to lend to causal interpretation of observational data, when in fact such interpretation is no less problematic than for other kinds of regression models applied to observational data. A more charitable interpretation is that SEMs are close to the kind of informal thinking about causal relationships that is common in social-science theorizing, and that, therefore, these models facilitate translating such theories into data analysis.”

References

Fox, J. (2002). Structural Equation Models: Appendix to An R and S-PLUS Companion to Applied Regression. Last corrected 2006.

Appeal to consequences fallacy in understanding Bell’s theorem

Joan Vaccaro (2018, p. 11) on arguments against superdeterminism:

“An argument that has been advocated by leading physicists is that humans are necessarily independent of the universe that surrounds them because the practice of science requires the independence of the experimenter from the subject of study. For example, Bell et al. state that unless the experimenter and subject are independent, we would need to abandon ‘…the whole enterprise of discovering the laws of nature by experimentation’, and Zeilinger claims that if the experimenter and subject were not independent ‘…such a position would completely pull the rug out from underneath science.’ However, this argument contains a logical fallacy called an appeal to consequences. Specifically, arguing for experimenter–subject independence on the basis that the alternative has undesirable consequences does not prove that experimenters are independent of their subjects. Rather, the alternative may well be true, in which case we would need to deal with the consequences.”

References

Vaccaro, J. A. (2018). The quantum theory of time, the block universe, and human experience. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2123).

Beautiful friendships have been jeopardised

This is an amusing opening to a paper on face validity, by Mosier (1947):

“Face validity is a term that is bandied about in the field of test construction until it seems about to become a part of accepted terminology. The frequency of its use and the emotional reaction which it arouses-ranging almost from contempt to highest approbation-make it desirable to examine its meaning more closely. When a single term variously conveys high praise or strong condemnation, one suspects either ambiguity of meaning or contradictory postulates among those using the term. The tendency has been, I believe, to assume unaccepted premises rather than ambiguity, and beautiful friendships have been jeopardized when a chance remark about face validity has classed the speaker among the infidels.”

I think dozens of beautiful friendships have been jeopardized by loose talk about randomised controlled trials, theory-based evaluation, realism, and positivism, among many others. I’ve just seen yet another piece arguing that you wouldn’t evaluate a parachute with an RCT and I can’t even.

References

Mosier, C. I. (1947). A Critical Examination of the Concepts of Face Validity. Educational and Psychological Measurement, 7(2), 191–205.

Nietzsche on causa sui

‘The causa sui is the best self-contradiction that has yet been conceived, it is a sort of logical violation and unnaturalness; but the extravagant pride of man has managed to entangle itself profoundly and frightfully with this very folly. The desire for “freedom of will” in the superlative, metaphysical sense, such as still holds sway, unfortunately, in the minds of the half-educated, the desire to bear the entire and ultimate responsibility for one’s actions oneself, and to absolve God, the world, ancestors, chance, and society therefrom, involves nothing less than to be precisely this causa sui, and, with more than Munchausen daring, to pull oneself up into existence by the hair, out of the slough of nothingness.’
– Friedrich Nietzsche (written 1886), Beyond Good and Evil, translated from German by Helen Zimmern