What a fab week at AES 2025. I delivered a whole day training on RCTs (Monday), the opening keynote (Wednesday), and an Ignite talk (Friday).
Here’s the gist of the keynote – much of the content consisted of examples to illustrate (skim through the slides):
- Quant people talk about the subjectivity of what we do more than you might expect (if you haven’t been behind the scenes) given how some present their methods and findings.
- Evaluators have been (obviously!) merrily mixing methods for years – but we can go further.
- Every evaluation is counterfactual if you want it to be – it’s a lens you apply, and one way of thinking about causation, however qual or quant (or other) your evidence is.
Pics thanks to Albie Colvin:
Lovely blog post by Eleanor Kerdo (ARTD):
“They began by challenging and unravelling the artificial binaries we make in evaluation and in our judgements.
“As the lucky person I am to love a non-binary person (my partner) I feel blessed to live a little outside of some of our gender binaries and our notion of family and roles. This greatly impacts my experience as an evaluator and how I navigate my work, parenting, and life. So I reflected a lot on that in the opening session.
“Andi started with Quant and Qual, and how this is less of a binary than we think: that it is not so simplistic. They moved on to the counterfactual and non-counterfactual ‘binary’, and how this can be tested beyond an RCT. They finished with a discussion about the subjective vs objective binary. This one was fun! Andi looked at the social construction of the speed of light over time and how this changed and ‘the defensible, yet subjective analytic choices that influence all research results’ (Silberzahn, et al 2018).
“Andi finished by noting also that there are gender and racial biases in data and a privilege hazard that needs to be understood.
“Andi challenged us to think as evaluators: can we mess up our toolbox, can we find different ways to conceptualise and think and use different methods to better serve our community?”