Tag: methodology

Variables

“The view attributed to positivism, that reality is fragmented into variables, is a straw man. No one believes it, and classification into types is something that all researchers, both quantitative and qualitative, do. Variables are a product of measurement procedures; they are not part of the structure of reality.”

Paley, J., & Lilford, R. (2011). Qualitative methods: An alternative view. BMJ, 342, d424.

Was turnout associated with the composition of Labour, Tory, and Reform votes?

Was turnout associated with the composition of Labour, Tory, and Reform votes? An excuse to test drive the {compos} package, by David Firth and Fiona Sammut, which implements the generalized Wedderburn logit model. See preprint by Firth and Sammut (2023), Analysis of composition on the original scale of measurement.

Knitted output over here.

Dipping into How to Think Like a Realist

Raw Pawson’s new book (Pawson, 2024) is an introduction to scientific practice, with social science as a corollary, drawing on the philosophy of science and Pawson’s experience conducting applied research. The book is on my reading stack. For now, I have taken a purposive sample of a few episodes (as Pawson calls the chapters) to get a sense of what’s in store. Here are some clues.

The main objective of the book (p. xvi)

“… is revealed in the structure of the text, which journeys across physical science and clinical science, before landing squarely in social science. The coverage here represents a commitment to the ‘unity of science’ (Oppenheim and Putnam, 1958). This heroic proposition claims that there are core explanatory principles which underpin science in all it guises, and the book makes tiny, tentative steps in tracing the common realist tenets. Perforce, I am also committed to the ‘unity of social science’.”

What about those context, mechanism, outcome triads that realist evaluators use? (p. 42)

“All scientific investigation utilises explanations relating mechanisms and contexts to empirical patterns.”

And (p. 48):

“With Harré, I have characterised generative causation in physical science as the analysis of mechanisms, contexts, and regularities (MCR). In clinical research the focus, quite properly, is on mechanisms, contexts, and outcomes (MCO). In social research, it might be wise to begin with the shorthand mechanisms, contexts, and change (MCC).”

(I wonder what the implications of the book are for realist evaluation as a distinct genre – will read those bits with interest.)

On social science and people who don’t follow science (p. xviii):

“Despite the habitual use of the appellation ‘social science’, many of my colleagues would reject any claim to follow science, dismiss any interest in causality, deny any need for objectivity, and scorn the possibility of generalisation. They are beyond hope. I don’t seek to convert them. But in following their chosen paths these various tribes – constructivists, post-modernists, emancipators, critics, essayists, relativists, and so on – have found time to say why causality, objectivity, and generality are false idols. So, in defending the science in social science, their criticisms also need to be overturned.”

More on what Pawson aims to do (pp. 251-252):

“What I’ve come up with here might well be entitled The Old Rules of Sociological Method. I have attempted to extract and justify some realist principles for conducting social research on the back of a generous portfolio of existing examples. Those illustrations reach across many research domains and a broad portfolio of practical methods. But they remain a pinprick; I could have called upon a thousand others. Accordingly, there is another way of perceiving my efforts. The book is no more and no less than an attempt to codify and formalise existing practices. I have tried to capture a tradition. So, just as Monsieur Jourdain spoke prose without knowing what it was, it may well be that you, dear reader, have been thinking like a realist without knowing it!”

I had to google Jourdain. He’s the main character in a comedy by Molière, Le Bourgeois gentilhomme (The Bourgeois Gentleman). The play satirises “the pretensions of the social climber whose affectations are absurd to everyone but himself”, which is a curious reference, dear reader.

References

Pawson, R. (2024). How to Think Like a realist: A methodology for social science. Edward Elgar Publishing Limited.

Choosing a sample size for a thematic analysis

About a decade ago, Henry Potts and I (Fugard & Potts, 2015) developed a method to guide decisions about sample size in thematic analyses (there’s also an app that does the calculations).

The core idea is straightforward: the rarer the phenomenon you want to investigate, the larger the sample you’ll need -after using purposive sampling to recruit the most relevant participants. For example, if you’re interested in the experiences of people engaging with therapy for depression, you would begin with participants who have had therapy for depression, not the general population, which reduces the total needed.

The paper outlining the approach has been cited over a thousand times, though only a small number of those citations apply the method as we intended. Here are five of them:

  • “Framing the necessary sample size in terms of the likelihood of capturing important ideas, a sample size of at least 15 per group would have a 90% probability for capturing ideas held by 25% of the population.” (Weller et al., 2017, p. 2.) Would add, that’s to ensure two participants provide material relevant to those themes.
  • “An estimation of the sample size was done prior to the second selection to ensure that the final number of participants would be large enough to yield a rich data set. Twenty-one interviews would be needed according to Fugard [and Potts]’s sample size calculation method for 80% power. This calculation assumed: a lowest prevalence of 60% of a theme worth discovering, 90% of the informants having something to say about the theme, and that the theme should be recognised at least ten times in the data.” (Malmborg et al., 2020, p. 170)
  • “In this study of the reasons for open online student dropout, appropriate sample size was determined based on Fugard and Potts’ (2015) thematic analysis sample size tool, which highlights the required sample size as a function of anticipated theme prevalence in a population. In line with this tool, a sample of 200 has a 99% probability of detecting five theme instances for a theme prevalent in 6% of the population; thus, 200 was set as the minimum sample target for this study.” (Greenland & Moore, 2022, p. 652)
  • “To evaluate the saturation of the codes, we used the Fugard and Potts method to predict saturation based on probability theory. This approach was appropriate for our data set, given our large, random sample of reviews and our predominantly deductive approach to data analysis. Our data set provided >80% power to identify 5 instances of themes mentioned by 1% of the population. We chose a cutoff of 1% to reflect the shallow nature of this data set, assuming that not all who experienced a code would describe it in their review, and 5 instances because this was typically the number of observations required to achieve repetition of content within the codes.” (Polhemus et al., 2022, p. 4.) To have exactly 80% power for prevalence 1% and five instances, they would have needed a sample size of 671.
  • “According to the calculation method of Fugard and Potts, the target study cohort size of 28 patients would provide 90% power to detect any theme of interest in at least 1 interview if the true prevalence of the patient experience captured by that theme was between 5 and 10% in the underlying PH1 population represented by the study cohort.” (Danese et al., 2023, p. 3) (Actually prevalence 7.9%.)

From these examples, it’s clear that the richness of the material and time taken to analyse it was an important and sensible constraint. For example, we would expect smaller samples for long interviews and larger samples for a sentence of free text in a web-based survey. Contrary to some arguments we have seen in the literature, it does also apply to reflexive thematic analysis too. Please do pop me an email if you’d like to give it a go but feel a bit stuck. I’d be happy to help.

References

Danese, D., Goss, D., Romano, C., & Gupta, C. (2023). Qualitative assessment of the patient experience of primary hyperoxaluria type 1: An observational study. BMC Nephrology, 24(1), 319.

Fugard, A. J. B. & Potts, H. W. W. (2015). Supporting thinking on sample sizes for thematic analyses: A quantitative tool. International Journal of Social Research Methodology, 18, 669–684. (There’s an app for that.)

Greenland, S. J., & Moore, C. (2022). Large qualitative sample and thematic analysis to redefine student dropout and retention strategy in open online education. British Journal of Educational Technology, 53, 647–667.

Malmborg, A., Brynte, L., Falk, G., Brynhildsen, J., Hammar, M., & Berterö, C. (2020). Sexual function changes attributed to hormonal contraception use – a qualitative study of women experiencing negative effects. The European Journal of Contraception & Reproductive Health Care, 25(3), 169–175.

Polhemus, A., Simblett, S., Dawe-Lane, E., Gilpin, G., Elliott, B., Jilka, S., Novak, J., Nica, R. I., Temesi, G., & Wykes, T. (2022). Health Tracking via Mobile Apps for Depression Self-management: Qualitative Content Analysis of User Reviews. JMIR Human Factors, 9(4), e40133.

Weller, S. C., Baer, R., Nash, A., & Perez, N. (2017). Discovering successful strategies for diabetic self-management: A qualitative comparative study. BMJ Open Diabetes Research and Care, 5, e000349.

LaLonde (1986) after Nearly Four Decades: Lessons Learned (Guido Imbens, Yiqing Xu)

“We show that modern methods, when applied in contexts with significant covariate overlap, yield robust estimates for the adjusted differences between the treatment and control groups. However, this does not mean that these estimates are valid. To assess their credibility, validation exercises (such as placebo tests) are essential, whereas goodness of fit tests alone are inadequate. Our findings highlight the importance of closely examining the assignment process, carefully inspecting overlap, and conducting validation exercises when analyzing causal effects with nonexperimental data.”

arXiv:2406.00827

The Turing Way handbook to reproducible, ethical and collaborative data science

“The Turing Way project is open source, open collaboration, and community-driven. We involve and support a diverse community of contributors to make data science accessible, comprehensible and effective for everyone. Our goal is to provide all the information that researchers and data scientists in academia, industry and the public sector need to ensure that the projects they work on are easy to reproduce and reuse.”

Yonder