Personal website – any views expressed herein are my own unless otherwise obvious
Are you wondering how to extract estimates of the following estimands from linear models underlying mediation analyses…?
You might be interested in this example I put together, using {mediation} in R, showing the arithmetic!
“Seeking active parent consent can undermine secondary school students’ autonomy, and limit participation, particularly among disadvantaged students, so biasing research. Our analysis suggests that active student consent and passive parent/carer consent be standard practice for most research procedures in secondary schools. More intrusive data collection, such as blood and saliva samples, would require parent/carer active consent since such procedures would be defined as diagnostic procedures so being classed as an investigational product. However, we would argue that for questionnaire completion, observation or routine data, student consent and autonomy should have primacy with parents having the right and means to receive full information, ask questions and withdraw their children from research should they wish. This approach gives proper primacy to student autonomy while also respecting parent/carer autonomy.”
Also helpful thoughts on issues arising with consent for whole-class interventions.
“This book introduces a new way of thinking about research designs in the social sciences. Our hope is that this approach will make it easier to develop and to share strong research designs.
“At the heart of our approach is the MIDA framework, in which a research design is characterized by four elements: a model, an inquiry, a data strategy, and an answer strategy. We have to understand each of the four on their own and also how they interrelate.”
Uses {DeclareDesign} in R.
Do you often cry while having a warm feeling in the heart because you find something beautiful? If so, you might be intrigued by this study of the Geneva Sentimentality Scale!
It is all too common for a box and arrow diagram to be cobbled together in an afternoon and christened a “theory of change”. One formalised version of such a diagram is a structural equation model (SEM), the arrows of which are annotated with coefficients estimated using data. Here is John Fox (2002) on SEM and informal boxology:
“A cynical view of SEMs is that their popularity in the social sciences reflects the legitimacy that the models appear to lend to causal interpretation of observational data, when in fact such interpretation is no less problematic than for other kinds of regression models applied to observational data. A more charitable interpretation is that SEMs are close to the kind of informal thinking about causal relationships that is common in social-science theorizing, and that, therefore, these models facilitate translating such theories into data analysis.”
Fox, J. (2002). Structural Equation Models: Appendix to An R and S-PLUS Companion to Applied Regression. Last corrected 2006.
“We conducted a systematic review and multiple meta-analyses of the growth mindset intervention literature. Our goal was to answer two questions: (a) Do growth mindset interventions generally improve students’ academic achievement? and (b) Are growth mindset intervention effects due to instilling growth mindsets in students or are apparent effects due to shortcomings in study designs, analyses, and reporting? To answer these questions, we systematically reviewed the literature and conducted multiple meta-analyses imposing varying degrees of quality control. Our results indicated that apparent effects of growth mindset interventions are possibly due to inadequate study designs, reporting flaws, and bias. In particular, the systematic review yielded several concerning patterns of threats to internal validity.”
Here’s a pic:
Cryptographic hash functions are sometimes used to create pseudo IDs from identifiable info like NHS numbers. An advantage of this approach is that two sites that have no way to communicate with each other will generate the same pseudo ID, allowing data to be linked. A disadvantage is that, although in general hashes are impossible to inverse, a rainbow table lookup attack can be used to inverse the hash when the space of inputs is relatively small, as is the case for ID numbers. John Cook explores.
Baseline balance is important for both experiments and quasi-experiments, just not in the way researchers sometimes believe. Here are excerpts from three of my favourite discussions of the topic.
Don’t test for baseline imbalance in RCTs. Senn (1994, p. 1716):
“… the following are two incontrovertible facts about a randomized clinical trial:
1. over all randomizations the groups are balanced;
2. for a particular randomization they are unbalanced.
Now, no ‘[statistically] significant imbalance’ can cause 1 to be untrue and no lack of a significant balance can make 2 untrue. Therefore the only reason to employ such a test must be to examine the process of randomization itself. Thus a significant result should lead to the decision that the treatment groups have not been randomized…”
Random sequence generators will randomise the groups. What can go wrong is that people can subvert the randomisation -sometimes with the best of intentions – if you’re not careful to prevent them doing so.
Balancing covariates between treatment groups, e.g., using permuted blocks or minimisation, can increase the efficiency of covariate adjustment. Senn (1994, p. 1721):
“… what is the value of balance as regards validity of an inference? Answer: none. This does not mean that balance is not useful. Its use is that, having decided to condition, we minimize the standard errors of our treatment effects if our covariate is orthogonal to treatment. This is the true value of balance which is an issue of efficiency, not validity.”
Do examine baseline imbalance in quasi-experiments; however, not by using statistical tests. Sample descriptives, such as a difference in means, suffice. Imai et al. (2008, p. 497):
“… from a theoretical perspective, balance is a characteristic of the sample, not some hypothetical population, and so, strictly speaking, hypothesis tests are irrelevant…”
Using p-values from t-tests and similar can lead to erroneous decisions of balance. As you prune a dataset to improve balance, power to detect effects decreases. Imai et al. (2008, p. 497 again):
“Since the values of […] hypothesis tests are affected by factors other than balance, they cannot even be counted on to be monotone functions of balance. The t-test can indicate that balance is becoming better whereas the actual balance is growing worse, staying the same or improving. Although we choose the most commonly used t-test for illustration, the same problem applies to many other test statistics…”
If your matching has led to baseline balance, then you’re good, even if the matching model is misspecified. (Though not if you’re missing key covariates, of course.) Rosenbaum (2023, p. 29):
“So far as matching and stratification are concerned, the propensity score and other methods are a means to an end, not an end in themselves. If matching for a misspecified and misestimated propensity score balances x, then that is fine. If by bad luck, the true propensity score failed to balance x, then the match is inadequate and should be improved.”
Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171(2), 481–502.
Rosenbaum, P. R. (2023). Propensity score. In J. R. Zubizarreta, E. A. Stuart, D. S. Small, & P. R. Rosenbaum, Handbook of Matching and Weighting Adjustments for Causal Inference (pp. 21–38). Chapman and Hall/CRC.
Senn, S. (1994). Testing for baseline balance in clinical trials. Statistics in Medicine, 13, 1715–1726.
“Have you ever heard the adage that communication is only 7 percent verbal and 93 percent non-verbal, i.e. body language and vocal variety? You probably have, and if you have any sense at all, you have ignored it.” I have, in a presentations skills training course.
Philip Yaffe wades into the 1967 studies that produced this oft-cited 7% figure:
Subjects were asked to listen to a recording of a woman’s voice saying the word “maybe” three different ways to convey liking, neutrality, and disliking. They were also shown photos of the woman’s face conveying the same three emotions. They were then asked to guess the emotions heard in the recorded voice, seen in the photos, and both together. The result? The subjects correctly identified the emotions 50 percent more often from the photos than from the voice.
In the second study, subjects were asked to listen to nine recorded words, three meant to convey liking (honey, dear, thanks), three to convey neutrality (maybe, really, oh), and three to convey disliking (don’t, brute, terrible). Each word was pronounced three different ways. When asked to guess the emotions being conveyed, it turned out that the subjects were more influenced by the tone of voice than by the words themselves.
The conclusion:
The fact is Professor Mehrabian’s research had nothing to do with giving speeches, because it was based on the information that could be conveyed in a single word.
The original studies behind the figure look interesting for what they actually tried to do rather than the bullshit claims that are still being repeated in corporate training.
Mehrabian, A., & Wiener, M. (1967). Decoding of inconsistent communications. Journal of Personality and Social Psychology, 6(1), 109–114.
You can bullshit a bullshitter, except if they also have high cognitive ability, according to Littrell et al. (2021).
Littrell, S., Risko, E. F., & Fugelsang, J. A. (2021). ‘You can’t bullshit a bullshitter’ (or can you?): Bullshitting frequency predicts receptivity to various types of misleading information. British Journal of Social Psychology, 60(4), 1484–1505.