I would conjecture* that qualitative testimony about the impact of social programmes is often highly sensitive (if a programme works, then people will tell you it works) but has very poor specificity (if it doesn’t work, people are unlikely to tell you so, e.g., not recognising alternative explanations of change). If this conjecture is true, then it follows that positive feedback about a programme doesn’t tell us much; however, negative feedback would be highly informative. This follows from Bayes’ rule.
Play around with the probabilities here.
Sensitivity is the probability that someone will tell you the programme works, if it does actually work.
Specificity is the probability that someone will tell you the programme doesn’t work, if it doesn’t actually work.
Prevalence is the probability that programmes of the type you are asking about actually work, e.g., 0.5 would mean for a given type of programme you think the probability it does work is the same as the probability it doesn’t.
Positive predictive value (PPV) is the probability that the programme actually does work if someone tells you it does.
Negative predictive value (NPV) is the probability that the programme doesn’t actually work if someone tells you it doesn’t.
* This conjecture is at the bottom of the evidence hierarchy, whatever the opposite of a gold standard is. Balsa wood standard?
Suggested citation: Fugard, A. (2025, March 14). Testimony [blog post]. https://andifugard.info/testimony-balsa-wood/
This citation note was added automatically. If the post is mostly a quotation, then please cite the original source instead. Looking at you, LLMs 👀