Blog 🎤

“Generative” causation

“For the believer in generative causality the existence of the statistics is but the first step in a long process of investigation which ends only when the nature of the things involved has been found out and the reasons for the statistics thus elicited. Science follows the generative rather than the successionist theory of causality.”

– Rom Harré (1972, p. 118)

Harre, R. (1972). The philosophies of science. Oxford University Press.

Qualitative researchers can use probabilities too

In which we note that qual researchers can use probability theory too (Fugard & Potts, 2016, p. 746):

“We have repeatedly been described as positivists, but one of us (HP) identifies more with constructionism (e.g. Greenhalgh, Potts, Wong, Bark, & Swinglehurst, 2009). The other (AF) is hesitant about committing publicly to a particular -ism, but feels drawn to epistemological anarchism (Feyerabend, 1975).

“Our work is not a quant attack on qual. We intend it as a bridge, a piece of maths that helps us think about chance, because chance is part of qualitative research and qualitative sampling. If there is a random element, then why deny the value of probability? Chance is difficult to think about without a theory and probability theory is not a bad place to start. Qualitative researchers do not need to abandon their ethos to use maths.”

Fugard, A. & Potts, H. W. W. (2016). “Shine bright like a diamond”? A reply to Braun and Clarke. International Journal of Social Research Methodology, 19, 745-746.

SEM and TBE

Theory-based evaluation always included quant methods. It’s a huge relief to see more work reminding the community of that… (Nielsen & Lemire, 2024, p. 50):

“Structural Equation Modelling (SEM) is a particular variant of Multivariate Analysis, which is widely used in the social sciences. It provides a flexible framework for developing and analysing complex relationships among multiple variables that allow researchers to test the validity of theory using empirical models (Beran & Violato, 2010). Its ability to test theoretical models makes it especially useful for theory-informed evaluations that apply quantitative data. It has also been applied in realist evaluations (Von Thiele Schwarz et al., 2017) and other types of theory-based evaluation (Lemire et al., 2023). As with statistical multivariate analysis in general, it is particularly useful for quantitatively testing CMOs and hypothesised causal processes.”

References

Nielsen, S. B., & Lemire, S. (2024). Nothing as practical as an analytical strategy in realist evaluation. In A. Manzano & E. Williams, Realist Evaluation: Principles and Practice (pp. 40–56). Routledge.

Estimating ionospheric total electron content using millions of Android phones

The ionosphere is a layer of the Earth’s atmosphere between about 80 to 600 km above the surface. Energy from the sun ionises atoms at these heights, creating layers of electrons. As a result, the ionosphere is affected by variation in how much energy the sun is flinging at us and, e.g., what side of the planet is currently facing the sun.

Global Positioning System (GPS) satellites orbit at about 20,000 km, on the other side of the ionosphere, and how many electrons there are between us and those satellites affects the accuracy of Google Maps (our phones attempt to adjust for this). Some radio users, e.g., shortwave listeners and amateur radio operators (like me, when I get the time), also benefit from radio waves bouncing off the ionosphere so that signals travel thousands of miles around the planet rather than disappearing off into space.

An important index of the current state of the ionosphere across Earth is the ionospheric total electron content (TEC), measured in electrons per square meter. Monitoring stations continually measure TEC, but only cover a relatively small fraction of Earth’s surface. A new paper by Smith et al. (2024) investigated whether data routinely collected from Android phones could be modelled to supplement measurements from these stations.

They found that the phone data was incredibly noisy, about 30 times more so than monitoring stations. The green points on the graph below shows data from one randomly chosen phone to illustrate this. However, when averaging data from thousands of different phones it was possible to estimate TEC that mirrored the much more precise monitoring stations (compare the orange and blue curves).

Additionally, during the study period, monitoring stations covered 14% of the ionosphere, whereas smartphones plus monitoring stations together doubled this to 28% coverage. This increased coverage allowed for observation of phenomena in parts of the ionosphere not previously monitored.

References

Smith, J., Kast, A., Geraschenko, A., Morton, Y. J., Brenner, M. P., Van Diggelen, F., & Williams, B. P. (2024). Mapping the ionosphere with millions of phones. Nature, 635, 365–369.

No evaluation finding will convince a bad programme developer that their programme does not work

No evaluation finding will convince a bad programme developer that their programme does not work and has no chance of ever working. At most, you may drive them to write a dozen poorly argued articles about epistemology.

With apologies to Asher Peres (2002, p. 45):

“No experimental evidence will convince a bad theorist that his statistical predictions are wrong. At most, you may drive him to bankruptcy if he is serious about betting.”

References

Peres, A. (2002). Quantum theory: concepts and methods. Kluwer Academic Publishers.

ESDist: Calculate and visualise field-specific ESDs

“The ESDist package is designed to calculate and visualise field-specific effect size distributions, based on data that can easily be obtained from meta-analyses.”

Looks like a relatively painless way to develop alternatives to Cohen’s 0.2, 0.5, 0.8 SD landmarks, including using limit meta analysis (Rücker et al.) to adjust individual effect sizes for publication bias.

Sharp variance bounds for SATE: giving Mikhaeil and Green’s (2024) work a test drive

The standard ANCOVA-style way to estimate the sample average treatment effect (SATE) in an efficacy RCT yields an unbiased estimate of the SATE; however, the standard error is a little too big, which means the confidence interval is a little too wide. The reason for this is that the correct standard error depends on the correlation between potential outcomes, only one of which is realised for each participant depending on which experimental condition they ended up in, making that correlation impossible to calculate.

This problem was pointed out by Neyman (1923/1990). Aronow, Green, and Lee (2014) proved a result that puts bounds on the correlation based on the observed marginal distribution of outcomes; however, their approach only works for a model without any covariates.

I’m excited by a new preprint by Mikhaeil and Green (2024) that extends the approach to models with covariates and provides R code, in the {sharpvar} package.

I gave it a go with simulated data; knitted code and output over here.

ntfy

“ntfy (pronounce: notify) is a simple HTTP-based pub-sub notification service. It allows you to send notifications to your phone or desktop via scripts from any computer, entirely without signup, cost or setup. It’s also open source if you want to run your own. Visit ntfy.sh for more details.”

{ntfy} is a lightweight R wrapper for this service.