Blog 🎤

Worrying developments in NHS England mental health outcomes monitoring

Mental health service users hope that the therapeutic interventions they receive will help them feel better. Randomised controlled trials are one important way to test whether a therapy “works”; however, they don’t reveal how interventions are experienced in routine care. This has led to routine outcomes monitoring which uses questionnaires to ask service users and clinicians to rate symptoms and other relevant information before, during, and after treatment. Outcomes monitoring has been used by NHS services for some years, for example through the Child Outcomes Research Consortium and Improving Access to Psychological Therapies (IAPT). It is, however, controversial. Ros Mayo (2010, p. 63) for example argues that:

“The application of oversimplified questions requiring tick-box answers … are driven by short-term and superficial policies and management techniques, largely incorporated from industry and the financial sector and primarily concerned with speed, change, results, cost effectiveness – turnover and minimising human contact and time involvement… They have nothing to do with human engagement…”

And yet, outcomes monitoring could be better than bureaucracy. There is emerging evidence that providing regular progress feedback to clinicians improves outcomes, especially when questionnaires are completed by service users. Intuitively this seems to make sense: people could sometimes reveal more about how they feel on paper than they can orally face-to-face. Items used in IAPT include:

  • “How often have you been bothered by… not being able to stop or control worrying?”
  • “Heart palpitations bother me when I am around people”

They also ask directly about the care received, for example:

  • “Did staff listen to you and treat your concerns seriously?”
  • “Did you feel involved in making choices about you treatment and care?”

Responses to these items could help clinicians understand to what extent services are helping service users. Also using standardised questionnaires means that expected progress curves can be developed (for example see work by Lambert and colleagues), so clinicians can see, for example, if progress is slower than would be expected given initial assessment and, if warranted, try a different approach.

It’s early days for outcomes monitoring but the above examples suggest that it could be a promising approach. However, closer examination shows that there are clear problems with how questionnaires are being used in practice, and I think NHS services in England are being asked to implement actively damaging approaches to outcomes monitoring.

Problem 1. Use of an unreliable measure

Suppose you wish to develop a rating scale for quality of life or how distressed you are so that you can monitor progress over time with a summary “score”. As a bare minimum requirement, all of the questions for one topic should be related to each other: the items should be “internally consistent”. This questionnaire probably wouldn’t do very well:

THE GENERAL STUFF QUESTIONNAIRE

  1. How often do you sing in the shower?
  2. What height are you?
  3. How far do you live from the nearest park?
  4. What’s your favourite number?
  5. How often do you go dancing?

You might learn interesting things from some of the individual answers, but summing all the answers together is unlikely to be revealing. This questionnaire would fare better:

THE RELIABLE FEELINGS QUESTIONNAIRE

  1. How do you feel? (0 is terrible, 10 fantastic)
  2. How do you feel? (0 is terrible, 10 fantastic)
  3. How do you feel? (0 is terrible, 10 fantastic)
  4. How do you feel? (0 is terrible, 10 fantastic)
  5. And finally… how do you feel? (0 is terrible, 10 fantastic)

However, you might wonder if questions 2 to 5 add anything. There are many ways to test the internal consistency of questionnaires, using the answers that people give. One is to use a formula called Cronbach’s alpha which gives answers from 0 to 1. Higher, say around 0.8, is better. Too close to 1 suggests redundancy in questions, as would be likely for the Reliable Feelings Questionnaire above.

In England, it is now recommended to use a “Mental Health Clustering Tool” to evaluate outcomes (see section 7.1 of recent guidelines). This is a questionnaire completed by clinicians covering areas such as hallucinations, delusions, depression, and relationship difficulties. The questionnaire suffers from a very basic problem: it’s not internally consistent. This has been discovered by the very people who proposed the approach (see p.30 of their report): “As a general guideline, alpha values of 0.70 or above are indicative of a reasonable level of consistency”. Their results are: 0.44, 0.58, 0.63, 0.57 – conspicuously smaller than 0.70. The authors also refer to previous studies explaining that this would always be the case, due to “its original intended purpose of being a scale with independent items” (p. 30). So, by design, it’s closer to the General Stuff Questionnaire above: a mixed bag of independent questions with low reliability.

Problem 2. Proposals to link outcomes to payment

Given evidence that collecting regular feedback might improve the quality of care people receive, it may be a good idea that the IAPT programme includes regular progress monitoring. IAPT uses service user completed questionnaires, which could in principle provide information clinicians might not otherwise have learned. There is, however, another potential difficulty over and above that of the quality of questionnaires used, and that is how external influences such as “Payment by Results” (PbR) initiatives can change for the worse how data is gathered and used. And PbR initiatives are beginning to be used in practice. The IAPT webpage notes, “An outcome based payment and pricing system is being developed for IAPT services. This is unique as other systems of PbR are activity or needs based.” Initial pilot results were “encouraging,” says the web page, and another pilot is currently running.

The idea with this proposal is that the more improvement shown by service users, as partly determined by outcomes scores, the more money service providers would receive. This is a worry as linking measures to targets has a tendency to cause the measures to stop measuring what it is hoped that they measure. For instance targets on ambulance response times have led to statistically unlikely peaks at exactly the target, suggesting that times have been changed. A national phonics screen has a statistically unlikely peak just at the cutoff score, suggesting that teachers have rounded marks up where they fell just below the cutoff. The effect has been around for such a long time that it has a name, Goodhart’s law:

“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

Faced with funding cuts, how many NHS managers in overstretched services will be forced to “game” performance-based payment systems to ensure their service survives? It’s not hard to do so, for example people who drop out of therapy tend to do so because they didn’t think it was helping. It can be easy to justify not asking people who leave therapy to complete questionnaires. Those who stay in therapy may be the ones with higher scores (see Clark, 2011, p. 321). Therefore, the missing data from those whom therapy was not helping could lead to a false picture of how well a service is working for service users. It is difficult to see how any data gathered that has been subject to these difficulties could tell clinicians or service providers anything helpful about their services or the wellbeing of those who use them.

Concluding thoughts

I think it is possible that statistically reliable, high quality questionnaires could be helpful in clinical practice, if thoughtfully used and explained to service users. Perhaps such questionnaires could be thought of as a bit like blood pressure readings; few patients feel reduced to numbers when told their results, and it’s obvious that more information is required to formulate a treatment if something is awry. However, using unreliable measures – especially when the developers know they are unreliable – is unacceptable for a national mental health programme. Moreover, linking questionnaire scores to payment raises even more complex ethical issues: there is a risk that the bureaucratic burden of questionnaires for service users would increase; also poorer treatment and financial decisions could result because those decisions would be made on the basis of low quality, unreliable data. Mental health services need to do much better than that, for the sake of everyone’s wellbeing.

Thanks very much to Martha Pollard and Justine McMahon for helpful comments.

Andy Fugard is a Lecturer in Research Methods and Statistics at the Department of Clinical, Educational and Health Psychology, University College London. His research investigates psychological therapies for children and young people: are they effective in routine practice and what moderates effectiveness. He is also interested in policy around practice-based evidence and recently became a member of the NHS England/Monitor Quality and Cost Benchmarking Advisory Group.

Why can psychological therapy be helpful?

Research explaining how therapy might help is saturated with pretentious jargon, e.g., invoking “transference”, “extinction”, heightening access to “cognitive–emotional structures and processes”, “reconfiguring intersubjective relationship networks” (see over here for more).

Could simpler explanations be provided? Here are some quick thoughts, inspired by literature, discussing with people, and engaging myself as a client in therapy:

  • You know the therapist is there to listen to you – they’re paid to do so – so there’s less need to worry about their thoughts and feelings. One can and is encouraged to talk at length about oneself. This can feel liberating whereas in other settings it might feel selfish or self-indulgent.
  • The therapist keeps track of topics within and across sessions. This can be important for recognising patterns and maintaining focus, whilst allowing time to tell stories, meandering around past experiences, to see where they lead.
  • The therapist has knowledge (e.g., through literature, supervisory meetings, and conversations with other clients) of a range of people who may have had similar feelings and experiences. So although we’re all unique, it can also be helpful to know that others have faced and survived similar struggles – especially if we learn what they tried and what helped.
  • Drawing on this knowledge, the therapist can conjecture what might be going on. This, perhaps, works best if the conjectures are courageous (so a step or two away from what the clients says) – and tentative, so it’s possible to disagree.
  • There can be an opportunity for practice, for instance of activities or conversations which are distressing. Practicing is a good way to learn.
  • Related, there’s a regular structure and progress monitoring (verbally, with a diary, or using questionnaires). Self-reflection becomes routine and constrained in time, like (this might be a bit crude but bear with me) a psychological analogue of flossing one’s teeth.
  • (Idea from Clare) “… daring to talk about things never spoken of before with someone who demonstrates compassion and acceptance; helpful because allows us to face things in ourselves that scare us and develop less harsh ways of responding to ourselves”
  • The therapist has more distance from situations having an impact on someone than friends might have so, e.g., alternative explanations for interpersonal disputes can more easily be provided.
  • It’s easier for a therapist to be courageous in interactions and suggestions than for a friend as – if all goes wrong – it’s easier for the client to drop out of the therapeutic relationship without long-term consequences (e.g., there’s no loss of friendship).
  • Telling your story to a therapist gives you an audience who is missing all of the context of your life. Most of the context can feel obvious, until you start to tell your story. Storytelling requires explaining the context, making it explicit. For instance who are the people in your life? Why did you and others say and do the things they did? Perhaps this act of storytelling and making the context explicit also makes it easier to become aware of and find solutions.

“How I became an analyst” by Arthur Valenstein

Interesting multidisciplinary background — some excerpts from Valenstein (1995):

“When I was sixteen years old I built my own short-wave receiver and transmitter and became a ham radio operator. This bent towards electronics motivated me to enter the engineering school at Cornell University in 1931 with the intention of becoming an electrical engineer…”

“But those were depression years, and it seemed unlikely that I could make a sufficient livelihood as an electrical engineer.”

“… from early years I had been curious about people, how and why they were as they were. I was puzzled about myself as well, feeling myself to be something of an ‘outsider’ in school. As I learned later, this is one of the elements contributing to psychological-mindedness, a predisposition that is conducive to psychoanalytic inquiry.”

“I have always had one foot in hard science and one foot in literature and the humanities, and fortunately I don’t seem to have fallen between the two.”

“George Henry was carrying out a heavily funded research project on homosexuality. This opened a whole world to me that I had never known, especially the gay world, and I learned something about it, even getting to know some of its colloquial terms. Later Henry and his research assistant, who in retrospect I realize was homosexual, published several books on homosexuality from a descriptive point of view.”

“… I came to be in Boston, which I never left except for one year in neurology with Foster Kennedy (a colourful man, a Northern Ireland Orangeman of great sartorial splendour and the gift of marvellously eloquent, elegant speech) at Bellevue Hospital in New York, and my years in the military.”

“My initial exposure to the activities and ambience of the Hampstead Child Therapy Clinic [now the Anna Freud Centre] forty years ago, and my continued contact with it and with Anna Freud over many years, greatly influenced my identity as a psychoanalyst, both theoretically and clinically. Before my sabbatical in London in 1955, I had become interested not only in what nowadays seems to be called ‘cognitive developmental psychology’ and ‘attachment theory’, but also what might be termed ‘affect developmental psychology’.”

Reference

Valenstein, A. (1995). How I became an analyst. Bulletin of the Anna Freud Centre, 18, 283–291.

Advice to empiricists from Alan Grafen

Spotted in Grafen, A. (1987). Measuring sexual selection: why bother.

To field workers, I would say: whatever you do, don’t go out and measure a parameter in somebody else’s model. Few things could be more fruitless. Your role is not to be a theoretician’s technician. Instead you are at least the theoretician’s equal and would be better employed doing one of three things. First, study an as yet unmodelled problem. Show the world that this problem is interesting and important, and outline what you see as its main features. Let theoreticians then busy themselves, working out in more and more detail that you had indeed got it nearly all right in the first place. Meanwhile, you can have discovered another interesting problem. Make the theoreticians come to you. After all, your function now is to provide the material which will determine what theoreticians will find interesting in five years’ time and beyond.

The second useful role of the field worker is to cut the Gordian knots of the theoreticians. For example, a debate is raging overhead about whether or not sufficient variability can be maintained in heritable fitness to make it worth while for a female to pay costs to choose a male solely for the inherited fitness of her offspring. Ultimately, this is a question that in theory will depend on the plausibility of various parameter values and details of the models. An empirical demonstration in reasonably natural conditions that fitness is often inherited from father to offspring in the absence of paternal care would radically change the nature of the debate. No longer would the plausibility of the conditions for maintaining this heritability be the hot topic, but rather what forces are actually responsible for it, and how strong are they? The empiricist must try to stop theoreticians wasting their time on empirical questions, if it is thought they could spend their time more profitably.

The third role of the empiricist is to undermine the theoretician by casting doubt on his accepted premises. What empirical demonstration would most embarass this model? How can I make this theory seem truly pointless? And so on. Go for the jugular.

Grafen, A. (1987). Measuring sexual selection: why bother?
J.W. Bradbury, M.B. Andersson (Eds.), Sexual Selection: Testing the Alternatives, John Wiley, Chichester (1987), pp. 221-233.

Individuals versus aggregrates

“Winwood Reade is good upon the subject,” said Holmes. “He remarks that, while the individual man is an insoluble puzzle, in the aggregate he becomes a mathematical certainty. You can, for example, never foretell what any one man will do, but you can say with precision what an average number will be up to. Individuals vary, but percentages remain constant. So says the statistician.”

—The Sign of Four by Sir Arthur Conan Doyle (hat-tip MP)

“… something more besides [psycho]analysis…”

“When the ego has taken its defensive measures against an affect for the purpose of avoiding unpleasure, something more besides analysis is required to undo them, if the result is to be permanent. This child must learn to tolerate larger and larger quantities of unpleasure without immediately having recourse to his defense mechanisms. It must, however, be admitted that theoretically it is the business of education rather than of analysis to teach him this lesson.”
—Anna Freud (1966, pp. 64-65)

Reference

Freud, A. (1966). The ego and the mechanisms of defense (revised ed.). New York: International Universities Press.

Famous women statisticians (or those who should be famous)

One of my day jobs is teaching psychology students how to do data analysis. Occasionally I quote famous statisticians, for instance to illustrate ways of thinking about analysis, the subjective nature of modelling data, and other fun things. I mention the likes of William Gosset (Guinness and t-tests), George Box (all models are wrong), and Bruno de Finetti (probabilities don’t exist).

Most – often all – of my students are women. Most of my current collection of quotations are from men. This is a problem. So, I’m currently looking for examples of famous statisticians who are women (statistician broadly interpreted; including data scientists, economists, quantitative social scientists). Here’s my current list. Suggestions for others would be most welcome, especially if you have a quotation I can use (turns out that statisticians write in maths most of the the time so it can be hard to find nice quotes).

  • Daphne Koller (Professor in Stanford University; wide range, e.g., conditional independence models, feature selection)
  • Deirdre McCloskey (Professor, economist, writes on stats amongst many other topics)
  • Elizabeth A. Stuart (Professor, statistician, causal inference methodology in public health and education)
  • Fiona Steele (Professor in Stats at LSE, e.g., multilevel modelling)
  • Florence Nightingale (data visualisation and public health stats)
  • Gertrude Mary Cox (experimental design and analysis of experiments)
  • Helena Chmura Kraemer (Professor of Biostatistics in Psychiatry, Emerita, at Stanford)
  • Hilary Mason (“enthusiastic member of the larger conspiracy to evolve the emerging discipline of data science”)
  • Hilary Parker (data analyst at Etsy; PhD in biostatistics, genomics; useR)
  • Irini Moustaki (Professor in Social Statistics at LSE)
  • Jane Hillston (Professor of quantitative modelling at Edinburgh University; invented the stochastic process algebra, PEPA)
  • Jennifer Neville (e.g., data mining for relational data such as networks/graphs)
  • Juliet Popper Shaffer (work on corrections for multiple hypothesis testing)
  • Pat Dugard (e.g., randomisation stats for single case and small-N multiple baseline studies)
  • Rachel Schutt (Senior Vice President of Data Science at News Corp)
  • Stella Cunliffe (worked in Guinness and first woman president of RSS)
  • Susan A. Murphy (e.g., clinical trial design; methods for multi-stage decision making)
  • Victoria Stodden (e.g., reproducibility of models, codes)

Example Quotations:

“The newly mathematized statistics became a fetish in fields that wanted to be sciences. During the 1920s, when sociology was a young science, quantification was a way of claiming status, as it became also in economics, fresh from putting aside its old name of political economy, and in psychology, fresh from a separation from philosophy. In the 1920s and 1930s even the social anthropologists counted coconuts.”
– Deirdre McCloskey, The Trouble with Mathematics and Statistics in Economics

“The Cabinet Ministers, the army of their subordinates… have for the most part received a university education, but no education in statistical method. We legislate without knowing what we are doing. The War Office has some of the finest statistics in the world. What comes of them? Little or nothing. Why? Because the Heads do not know how to make anything of them. […] What we want is not so much (or at least not at present) an accumulation of facts, as to teach men who are to govern the country the use of statistical facts.”
– Florence Nightingale in a letter to Benjamin Jowett; quoted by Kopf, E. W. (1916). Florence Nightingale as statistician. Publications of the American Statistical Association, 15, 388–404.

“To understand God’s thoughts we must study statistics, for these are the measure of His purpose.”
– Florence Nightingale

“The statistician who supposes that his [sic] main contribution to the planning of an experiment will involve statistical theory, finds repeatedly that he makes his most valuable contribution simply by persuading the investigator to explain why he wishes to do the experiment, by persuading him to justify the experimental treatments, and to explain why it is that the experiment, when completed, will assist him in his research.”
– Gertrude Mary Cox

“It is no use, as statisticians, our being sniffy about the slapdash methods of many sociologists unless we are prepared to try to guide them into more scientifically acceptable thought. To do this, there must be interaction between them and us.”
– Stella V. Cunliffe (1976, p. 9). Interaction. Journal of the Royal Statistical Society. Series A (General), 139, 1–19.

FIGJAM-based practice

Alternative to evidence-based practice: FIGJAM-based practice.MM0726_Fig_Jam__66623_std
(F**k I’m Good, Just Ask Me.)

Evidence is for the bureaucrats.
Trust us, we’re experts.
Join the school of the FIGJAM.
Throw your positivist randomised trials on the fire.

“I used the FIGJAM approach and I felt better.”

Coming to a social enterprise near you soon.

Book review: High-quality psychotherapy research, by AreĂĄn and Kraemer (2013)

Randomised controlled trials (RCTs) are great, the gold standard of empirical research. The only thing better than RCTs are systematic reviews of lots and lots of RCTs. (So the story goes.) The reader may have noticed that RCTs evaluating CBT for psychosis have been vigorously debated for many months after a review was published in the British Journal of Psychiatry (Jauhar et al., 2014). Maybe not everyone agrees that RCTs are great (disclosure: I have analysed a couple), but I think it’s fair to say they are unavoidable whether you are trying to design or demolish them.

High-quality psychotherapy research by Patricia Arean and Helena Chmura Kraemer sets out to be a “practical, step-by-step guide” to designing and running RCTs. So why bother with an RCT? Observational trials, the authors explain, might involve studying participants who choose one of two or more interventions of interest by simply observing how they get on. This is problematic as differences in outcomes might be due to whatever factors led to them ending up receiving an intervention rather than the effect it had. RCTs use randomisation to overcome this problem so that people differ only in terms of the intervention received. That’s about it for the “why”: don’t expect debate on the epistemology.

The book’s strengths emerge as it develops: it catalogues issues that should worry study investigators and the authors draw on their own experience to offer hints. The Delphi consensus-building approach is introduced to solve the problem of developing an intervention manual and examples are given of how to word a letter asking for feedback on the proposed result. Randomisation techniques are introduced including horror stories of how they have gone wrong and invalidated RCTs. Ideas are provided for control groups, e.g., waiting list, usual care, and “gold standard” controls, and their strengths and drawbacks. The importance of not using pilot study results to determine sample size choices is explained. Guidance is provided on the people required; for example you need three or more therapists, at least two research assistants in case one takes ill, and a good statistician amongst other people. The Appendix includes a sample budget justification. All practical advice.

The text runs to under 200 pages so this could never be a comprehensive guide to all aspects of RCTs. What this book does do well is provide a systematic menu of options and ideas for things to consider. It might possibly give some ideas of what to demolish too, should you be so inclined, but this book is really only for those who are already sold on RCTs and want to get on with the seemingly painful task of designing and running one.

References

AreĂĄn, P. A., & Kraemer, H. C. (2013). High-quality psychotherapy research: from conception to piloting to national trials. Oxford University Press.

Jauhar, S., McKenna, P. J., Radua, J., Fung, E., Salvador, R., & Laws, K. R. (2014). Cognitive-behavioural therapy for the symptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias. The British Journal of Psychiatry, 204, 20–29. doi:10.1192/bjp.bp.112.116285