Tag: methodology

“The true beginnings of scientific activity…” (Freud, 1915)

“We have often heard the demand that a science be built on clear and precisely defined basic concepts. In reality, no science, not even the most basic, starts out with such definitions. The true beginnings of scientific activity consist, rather, in the description of phenomena, which are then grouped, classified, and brought into relation with each other. Even when simply describing the material, we cannot avoid applying to it certain abstract ideas, acquired from somewhere or other but certainly not just from the new observations alone.”

—Freud (1915), Drives and their Fates

Linking statistics and qualitative methods

You’ll be aware of the gist. Quantitative statistical models are great for generalizing, also data suitable for the stats tends to be quicker to analyze than qualitative data. More qualitative methods, such as interviewing, tend to provide much richer information, but generalization is very tricky and often involves coding up so the data can be fitted using the stats. How else can the two (crudely defined here!) approaches to analysis talk to each other?

I like this a lot:

“In the social sciences we are often criticized by the ethnographers and the anthropologists who say that we do not link in with them sufficiently and that we simply produce a set of statistics which do not represent reality.”

“… by using league tables, we can find examples of places which are perhaps not outliers but where we want to look for the pathways of influence on why they are not outliers. For example, one particular Bangladeshi village would have been expected to have high levels of immunization, whereas it was down in the middle of the table with quite a large confidence interval. This seemed rather strange, but our colleagues were able to attribute this to a fundamentalist imam. […] Another example is a village at the top of the league table, which our colleagues could attribute to a very enthusiastic school-teacher.”

“… by connecting with the qualitative workers, by encouraging the fieldworkers to look further at particular villages and by saying to them that we were surprised that this place was good and that one was bad, we could get people to understand the potential for linking the sophisticated statistical methods with qualitative research.” (Ian Diamond and Fiona Steele, from a comment on a paper by Goldstein and Spiegelhalter, 1996, p. 429)

Also reminds me of a study by Turner and Sobolewska (2009) which split participants on their Systemizing and Empathizing Quotient scores. Participants were asked, “What is inside a mobile phone?” Here’s what someone with high EQ said:

“It flashes the lights, screen flashes, and the buttons lights up, and it vibrates. It comes to life on the inside and it comes to life on the outside, and you talk to the one side and someone is answering on the other side”

And someone with high SQ:

“Many things, circuit boards, chips, transceiver [laughs], battery [pause], a camera in some of them, a media player, buttons, lots of different things. [pause] Well there are lots and lots of different bits and pieces to the phone, there are mainly in … Eh, like inside the chip there are lots of little transistors, which is used, they build up to lots of different types of gates…”

(One possible criticism is that the SQ/EQ just found students of technical versus non-technical subjects… But the general idea is still lovely.)

Would be great to see more quantitative papers with little excerpts of stories. We tried in our paper on spontaneous shifts of interpretation on a probabilistic reasoning task (Fugard, Pfeifer, Mayerhofer & Kleiter, 2011, p. 642), but we only squeezed in a few sentences:

‘Participant 34 (who settled into a conjunction interpretation) said: “I only looked at the shape and the color, and then always out of 6; this was the quickest way.” Participant 37, who shifted from the conjunction to the conditional event, said: “In the beginning [I] always [responded] ‘out of 6,’ but then somewhere in the middle . . . Ah! It clicked and I got it. I was angry with myself that I was so stupid before.” Five participants spontaneously reported when they shifted during the task, for example, saying, “Ah, this is how it works.”’

References

Fugard, A. J. B., Pfeifer, N., Mayerhofer, B., & Kleiter, G. D. (2011).  How people interpret conditionals: Shifts towards the conditional event.  Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 635–648.

Goldstein, H. & Spiegelhalter, D. J. (1996). League tables and their limitations: statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society. Series A (Statistics in Society) 159, 385–443.

Turner, P. & Sobolewska, E. (2009). Mental models, magical thinking, and individual differences. Human Technology 5, 90–113.

Brands

Photo by Polina Tankilevitch

 

CBT versus counselling. Mental models versus mental rules. Bayesianism versus everything. There are lots of brand wars in psychology which stifle the development of theory and lead to tedious arguments. This does get recognised in print now and again. Here are some examples.

The first comes from a discussion of the ACT-R theory (Anderson, 2007, p. 19):

“… this book is not about ACT-R; rather I am using ACT-R as a tool to describe the mind. […] We may be proud of our ACT-R models and think they are better than others […] but we try not to lose track of the fact that they are just a way of describing what is really of interest.”

Later (p. 239) Anderson cites Herbert Simon, who complained that

“… ‘brand names’ tend to make difficult the analysis and comparison of […] mechanisms or the exchange of knowledge between research groups. […] Physicists did not divide quantum mechanics into the Heisenberg Brand, the Schrodinger Brand, and the Dirac brand.”

Another example, from sociology: a book on social class (Wright, 1997, p. 34):

“Readers who are highly skeptical of the Marxist tradition for whatever reasons might feel that there is no point in struggling through the numbers and graphs in the rest of this book. If the conceptual justifications for the categories are unredeemably flawed, it might be thought, the empirical results generated with those categories will be worthless. This would be, I think, a mistake. The empirical categories themselves can be interpreted in a Weberian or hybrid manner. […] As is usually the case in sociology, the empirical categories are underdetermined by the theoretical frameworks within which they are generated or interpreted.”

References

Anderson, J. R. (2007). How can the human mind occur in the physical universe? Oxford University Press

Wright, E. O. (1997). Class counts: student edition. Cambridge University Press.

“Usefulness” in psychometrics

“Consider, for example, instrumentalism. This view does not invoke truth but usefulness as the primary criterion for the adequacy of scientific theories and measurements. However, we are surely not seriously considering the idea that we have to rule out rivals to the hypothesis that intelligence tests are useful. The Wechsler Adult Intelligence Scale (WAIS; Wechsler, 1955) comes in a big heavy box, which is very useful to hit people on the head with, but the hypothesis that the WAIS is valid for inflicting physical injury is certainly not the kind of hypothesis we are interested in.”

Borsboom, D., Mellenbergh, G.J., & Van Heerden (2004, p. 1065).  The concept of validity. Psychological Review, 111, 1061-1071.

On methodology

“I did not make a deliberate decision to adopt a particular methodology: I had the good fortune to work alongside gifted colleagues from backgrounds in different disciplines, and their various techniques seemed to be producing results. With hindsight, I should describe how one learns from both experiments and intelligent software in terms of the distinction that philosophers draw between the correspondence and the coherence theories of truth. An assertion is true according to the first theory if it corresponds to some state of affairs in the world; true according to the second if it coheres with a set of assertions constituting a general body of knowledge. Experiments provide information about correspondence with the facts, but they exert a dangerous pull in the direction of empirical pedantry, where the only things that count are facts, no matter how limited their purview. Computer programs provide information about the coherence of a set of assumptions, but they exert a dangerous pull in the direction of systematic delusion, where all that counts is internal consistency, no matter how remote from reality. Give up one approach and you turn into a Gradgrind, the teacher in Dicken’s novel Hard Times, whose only concern is with the facts; give up the other and you become an architect for the Flat Earth Society. Those, at least, are the dangers.”

From the prologue to Mental Models by Philip Johnson-Laird

John Fox on SEM

From an Appendix to An R and S-PLUS Companion to Applied Regression:

“A cynical view of SEMs is that their popularity in the social sciences reflects the legitimacy that the models appear to lend to causal interpretation of observational data, when in fact such interpretation is no less problematic than for other kinds of regression models applied to observational data. A more charitable interpretation is that SEMs are close to the kind of informal thinking about causal relationships that is common in social-science theorizing, and that, therefore, these models facilitate translating such theories into data analysis.”