Organising effective policy research teams – the case of INR

Interesting clues from the US State Department’s INR, according to this week’s report by Dylan Matthews:

  1. Most INR analysts have a PhD – some have professorships – and have worked on their topic area for an average of 14 years. There are few generalists.
  2. The organisation is relatively small and analyses tend to be written by individuals rather than large teams.
  3. Analysts work with State Department policymakers and so have a clearer understanding of what information is most useful to them.

A comment about a question

How often have you been at an event where someone from the audience wants to ask a question and insists that they don’t need a microphone because they have a loud voice – only to be reminded that they still require one. Well, I’m at a national conference and I have a question (not a comment). It’s concise, should take a few seconds to say, and hopefully open enough that anyone in the panel can pick it up. Fortunately, once upon a time I took a course on live sound engineering, so I understand how you’re supposed to hold a mic. I’ll make this brief with no fuss.

A couple of mics are passed around the audience. Someone else asks a question before me. After a brief wait, it’s my turn. I hold the microphone as I have learned you should, the correct distance from my mouth and pointing towards it. I begin to speak.

What I hadn’t considered was that I was sitting near the back of a large hall, and the PA system speakers were positioned at the front. My mic technique had optimised the volume of my speech, which I heard back after a short and reverberated delay. This booming, reverberated clone of me made it difficult to concentrate. Additionally, it heightened my awareness of the Julian Clary dimension to my voice, triggering thoughts about cisgendered and heterosexual (cishet) norms of professionalism and which types of voices tend to be taken seriously. I felt that mine had squarely landed on the not-seriously end of the spectrum. All of these thoughts were happening at once; I was only about a sentence in.

So, I tried to adjust the volume by holding the microphone further from my mouth, then a little closer, and then further again. I noticed people dotted around the room, whom I’d been speaking to earlier over coffee, had turned to listen. The panel appeared to be straining to hear me. One panel member asked me to repeat the question, which, by then, I had reduced to a short sentence – simply wanting the moment to be over. That panel member gave a good answer. Another asked, “Can we just take another question, please?”

I’ve recounted this experience in various ways, sometimes aiming for a giggle (taking poetic license with the Clary elements and in-out mic adjustments). Other times, focusing more on the frustrations of dominant professional norms. And yet, I don’t have a conclusion. I’d be eager to hear your thoughts.

“Use the difficulty”

Michael Caine talking about rehearsing a play where other actors improvising had got carried away.

“They started throwing things, and he threw a chair, and it lodged in the doorway. And I went to open the door, and I got my head around, and I went [to the director], ‘I’m sorry, sir, I can’t get in.’

“He said, ‘What do you mean?’

“I said, ‘There’s a chair there.’

“He said to me, ‘Use the difficulty.’

“I said, ‘What do you mean?’

“He said, ‘Well, if it’s a comedy, fall over it. If it’s a drama, pick it up and smash it. Use the difficulty.’”

What happens when you set targets – examples

(Updated 25/5/2016)

1. Ambulance response times (see Bevan and Hood, 2006)

ambulance target
Statistically unlikely spike exactly at the target suggesting something has happened to the data.

2. Phonics test scores (see Dorothy Bishop’s blog post)

phonics
A higher score is better performance. There’s a dip just below the pass mark of 32 and then a big spike, suggesting scores have been changed.

3. Call centre response times (see Caulcutt, 2004)

“The Times reported in October 2003 that the telecommunications regulator, Oftel, intended to investigate the workings of one of the newly established directory enquiry companies. According to the report: “Sixty call centre workers at the 118 118 directory enquiries service will be sacked in an attempt to head off a scandal over staff who deliberately gave out wrong numbers to boost their pay”. Why did they do this? It appears that the motivation was provided by a bonus system that rewarded employees for dealing with calls in less than 40 seconds.”

4. Final high school exam scores

Matura scores in Poland, 2013. To pass you need 30% or above.

4nnbyru

(See here, spotted via @MaxCRoser.)

5. “‘G4S cheats’ made 1,000 FAKE 999 calls to boost performance figures”

“Staff at scandal-hit G4S boosted performance figures by making hundreds of fake calls to a 999 centre run by the firm.

“Five employees have been suspended after allegedly making more than 1,000 “test calls” – many reportedly at quiet times when they could be picked up quickly.

“Without them G4S would have missed key targets of answering 92% of calls within 10 seconds in November and December 2015, so incurring a financial penalty.”

(Mirror article)

 

In general, Goodhart’s (1975) law applies: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”

 

References

Bevan, G., & Hood, C. (2006). What’s measured is what matters: targets and gaming in the English public health care system. Public administration, 84(3), 517-538.

Caulcutt, R. (2004). Managing by fact. Significance, 1(1), 36-38.

Goodhart, C. A. E. (1975). Monetary relationships: A view from Threadneedle Street. In Papers in Monetary Economics, Vol 1, Reserve Bank of Australia.

GCHQ’s director’s Turing speech – a research team manual?

Just read the (4 Oct 2012) speech about Alan Turing, given by Iain Lobban, Director GCHQ, at the University of Leeds.

Fantastic stuff in there. Here are some excerpts.

On learning to solve problems

“… [Turing] reported to Bletchley Park as agreed and immediately started working with [Dilly] Knox [expert on the Enigma cypher …]. Knox’s influence on Turing at this time is immense. The older veteran cryptanalyst shared everything he knew about Enigma with Turing, who eventually used this knowledge to write the first four chapters of his treatise on Enigma […]

“…[Turing] was happy to learn from Dilly Knox, happy to use that knowledge as the foundation for what he would develop subsequently, and was diligent in recording what he had learned and how he developed that into new areas so that others could profit from his knowledge just as he had profited from that of Knox.”

Knox could only take Turing so far and his quest for experience-based understanding of the cryptanalysis of Enigma took Turing to France in January 1940…”

Team work

There are lots of different ways in which people can work as part of a team.  Turing’s way was to take in other people’s ideas, develop and build on them, and then pass the product on to other people to be the foundation for the next stage.  He took the idea of electromechanical processing of Enigma messages from the Poles but developed their idea into something radically different.  When Welchman later enhanced the Bombe with his diagonal board, Turing was among the first to congratulate him on this major improvement.  Turing was part of the team, and shared in the success of the team.”

Respecting diversity

“I strongly believe a Sigint agency needs the widest range of skills possible if it is to be successful, and to deny itself talent just because the person with the talent doesn’t conform to a social stereotype is to starve itself of what it needs to thrive.”

“I don’t want to pretend that GCHQ was an organisation with twenty-first century values in the twentieth century, but it was at the most tolerant end of the cultural spectrum.  In an organisation which valued the skills and characteristics that difference can bring, Turing’s homosexuality was less of a talking point than his insights into the complex crypt problems of the day.  When he was put on trial, Hugh Alexander, the Head of Cryptanalysis at GCHQ went, with official approval, to speak as a character witness on his behalf, saying in court that Turing was a national asset.”

Exploiting serendipity

“Geoffrey Tandy was posted to Bletchley by the Admiralty in a spirit of helpfulness: his posting officer had understood him to be an expert in cryptograms, a word still used in the Admiralty at that time to mean messages signalled in code.  In fact he was an expert in cryptogams: non-flowering plants like ferns, mosses and seaweeds.  But while this knowledge might not have appeared to be of much use, Tandy became expert in German naval Enigma and because of his work on seaweed was able to provide unique advice on the preservation of cryptologic documents rescued from the sea.”

The role of management

“Part of my job is to continue to foster that atmosphere: to attract the very best people and harness their talents, and not allow preconceptions and stereotypes to stifle innovation and agility.”