counterfactual – Andi Fugard (∧⇒)

Counterfactual thinking – common and useful

Counterfactual isn’t a synonym for control group. Here’s a brief Open Encyclopedia of Cognitive Science entry on counterfactual thinking, which argues that it is “extremely common and profoundly useful in ordinary life.” Whether or not you believe that your impact evaluation is counterfactual, there’s a good chance that your participants are using counterfactual thinking to answer your questions.

Brigard, F. D. (2025). Counterfactual Thinking. In Open Encyclopedia of Cognitive Science. MIT Press.

Master’s in Public Policy (MPP) talk at Cambridge

This week I gave a talk to the MPP class. Slides here. I covered the usual things, e.g., how all policy evaluation is theory-based and counterfactuals don’t need a comparison group. A few new additions:

I claimed that Realistic Evaluation, i.e., the (Pawson & Tilley, 1997) book, is an intro to scientific methods and philosophy of science, rather than introducing a distinct form of evaluation. I think this is clearest to see in Pawson (2024), and it’s one of the most muddled sections of the Magenta Book.
The importance of theory for interpreting the meaning of covariates as well as merely selecting them. This is particularly important for intersectional understandings of selection into programmes, e.g., you need a theory to work out why, e.g., some combinations of gender and ethnicity, are more likely to be referred to a targetted programme. Is it driven by gender and/or racial stereotypes at time of selection, or broader societal discrimination before selection, for example?
An example from the Green Book (para 4.17) of a type of counterfactual reasoning, used when appraising different ways to realise a policy. I think the reasoning behind “deadweight” and “additionality” outcomes may use future less vivid conditionals. Still reading.

Self-reported versus statistically inferred attribution

“A starting point for the QuIP [Qualitative Impact Protocol] is the premise that those who were intended to benefit from an intervention know a great deal about what has caused and affected changes in their (and their households’) lives in the recent past, and what has influenced their active decisions to start or stop doing certain activities. Relying on the narrative testimonies of intended beneficiaries removes the need for an independent counterfactual based on interviews with a control or comparison group. This is because comparisons between what happened and what would have happened otherwise are embedded in causal claims within the narrative – our thinking and language is laden with ways of doing this (e.g. through use of the conditional tense and the many ways we can answer ‘why’ questions). Collected with care, narrative accounts are full of explicit and latent counterfactuals; the task is to identify and interpret them. […] We refer to reliance on respondents to provide evidence of the causal chain puzzle themselves as self-reported attribution, and we distinguish it from statistically inferred attribution that generally relies on exposure variation, including comparing treatment and control groups.” (Copestake, Morsink, & Remnant, 2019, pp. 7–8.)

Copestake, J., Morsink, M., & Remnant, F. (2019). Attributing Development Impact. Practical Action Publishing Ltd. https://doi.org/10.3362/9781780447469

Your control group can be sensible

There’s nothing in theories of experiments requiring that researchers use a ridiculous control group. An example of such a ridiculous (and unethical) control would include randomising people who are living in abject poverty either to a regular sufficient social security payment or leaving them in poverty. Hmmm, what group will have a better outcome…?

The first example Rubin gave in his classic “Estimating causal effects of treatments in randomized and nonrandomized studies” (Rubin, 1974, p. 689) had the following two groups:

“… we will restrict discussion to the very simple study consisting of 2N units (e.g., subjects), half having been exposed to an experimental (E) treatment (e.g., a compensatory reading program) and the other half having been exposed to a control (C) treatment (e.g., a regular reading program).”

Lind’s (1753) study of scurvy treatments compared six treatments, including cider, vinegar, and sea water, as well as the oranges and lemons. Potentially unclear what the theoretical rationale was for all of those, so not ideal I’ll admit.

It’s pretty common now to use “business as usual”, “treatment as usual”, “teaching as usual”, etc., controls and to avoid waitlist controls. But this is not a new idea.

Additionally, Rubin didn’t use Y(1) and Y(0) for the two potential outcomes, but Y(E) and Y(C). It’s a very subtle difference; however, I wonder if ditching the zero helps people’s understanding. My preference would be something even closer to the names of the actual programs, e.g., if we are evaluating whether the Acme Compensatory Reading Programme is better than regular reading, Y(Acme) and Y(Regular) to emphasise the contrast. (Also now we would use “participants” rather than “subjects”.)

References

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.

Lind, J. (1753). A treatise of the scurvy. In three parts. Containing an inquiry into the nature, causes and cure, of that disease. Together with a critical and chronological view of what has been published on the subject. Edinburgh: Printed by Sands, Murray and Cochran for A. Kincaid and A. Donaldson.

Counterfactual analysis as fatalism

‘Many counterfactual analyses are based, explicitly or implicitly, on an attitude that I term fatalism. This considers the various potential responses \(Y_{i}(u)\), when treatment \(i\) is applied to unit \(u\), as predetermined attributes of unit \(u\), waiting only to be uncovered by suitable experimentation. (It is implicit that the unit \(u\) and its properties and propensities exist independently of, and are unaffected by, any treatment that may be applied.) Note that because each unit label \(u\) is regarded as individual and unrepeatable, there is never any possibility of empirically testing this assumption of fatalism, which thus can be categorized as metaphysical.’

– Dawid, A. P. (2000, pp. 412-413) [Causal inference without counterfactuals. Journal of the American Statistical Association, 95, 407–424].

Counterfactual talk as nonsense

‘We all indulge, in anger and regret, in counterfactual talk: “If they had not operated, John would be alive today”; “If I had not said that, she would not have left me”; “If I had chosen a different publisher, my book on causality without counterfactuals would have sold 10,000 copies.” The more fortunate among us have someone to remind us that we are talking nonsense.’

– Shafer, G. (2000, p. 442) [Causal inference without counterfactuals: Comment. Journal of the American Statistical Association, 95, 438–442].

Time for counterfactuals

I have just discovered Scriven’s stimulating (if bleak) challenge to a counterfactual understanding of causation (see the debate recorded in Cook et al., 2010, p. 108):

“The classic example of this is the guy who has jumped off the top of a skyscraper and as he passes the 44th floor somebody shoots him through the head with a .357 magnum. Well, it’s clear enough that the shooter killed him but it’s clearly not true that he would not have died if the shooter hadn’t shot him; so the counterfactual condition does not apply, so it can’t be an essential part of the meaning of cause.”

I love this example because it illustrates a common form of programme effect and summarises the human condition – all in a couple of sentences! Let’s reshape it into an analogous example that extends the timeline by a couple of decades:

“A 60 year old guy chooses not to get a Covid vaccine. A few months later, he gets Covid and dies. Average male life expectancy is about 80 years.”

By the end of the first sentence, I reason that if he had got the vaccine, he probably wouldn’t have died. By the end of the second sentence, I am reminded of the finiteness of life. So, the vaccine didn’t prevent death, similarly to an absence of a gunshot in the skyscraper example. But something doesn’t feel quite right about this conclusion – how can we think about this using counterfactuals?

In a programme evaluation, it is common to gather data at a series of fixed time points, for instance a few weeks, months, and, if you are lucky, years after baseline. We are often happy to see improvement even if it doesn’t endure. For instance, if I take a painkiller, I don’t expect its effects to persist forevermore. If a vaccine extends life by two decades, that’s rather helpful. Programme effects are defined at each time point.

To make sense of the original example, we need to add in time. There are three key timepoints:

Jumping (T₀).
Mid-flight after the gunshot (T₁).
Hitting the ground (T₂).

When considering counterfactuals, the world may be different at each of these times, e.g., at T₀ the main character might have decided to take the lift.

Here are counterfactuals that make time explicit:

If the guy hadn’t jumped at T₀, then he wouldn’t have hit the ground at T₂.
If the guy hadn’t jumped at T₀, then he wouldn’t have been shot with the magnum and killed at T₁.
If the guy had jumped, but hadn’t been shot by the magnum, he would still have been alive at T₁ but not at T₂.

To assign truth values or probabilities to each of these requires a model of some description, e.g., a causal Bayesian network, which formalises your understanding of the intentions and actions of the characters in the text – something like the DAG below, with conditional probabilities filled in appropriately.

So for instance, the probability of being dead at T₂ given jumping at T₀ is high – if you haven’t added variables about parachutes. What happens mid-flight governs T₁ outcomes. Alternatively, you could just use informal intuition. Exercise to the reader: give it a go.

Using the Halpern-Pearl definitions of causality on this model (Halpern, 2016), jumping caused death at both T₁ and T₂. The shooting caused death at T₁ but not T₂. (R code here – proper explanation to be completed, but you could try this companion blog post and citation therein.)

Back then to the vaccine example, the counterfactuals rewrite to something like:

If the guy hadn’t been born at T₀, then he wouldn’t have died at T₂.
If the guy hadn’t been born at T₀, then he couldn’t have chosen not to get a vaccine and died at T₁.
If the guy had been born, but had decided to get the vaccine, he would still have been alive at T₁ aged 60, but possibly not at T₂ aged 80.

References

Cook, T. D., Scriven, M., Coryn, C. L. S., & Evergreen, S. D. H. (2010). Contemporary Thinking About Causation in Evaluation: A Dialogue With Tom Cook and Michael Scriven. American Journal of Evaluation, 31(1), 105–117.

Halpern, J. Y. (2016). Actual causality. The MIT press.

Actual causes: two examples using the updated Halpern-Pearl definition

Halpern (2015) provides three variants of the Halpern-Pearl definitions of actual causation. The formalism is concise and precise; however, it is challenging to use in practice, so I wrote an R script to do the sums. This post works through two examples.

The second (“updated”) definition of an actual cause asserts that \(\vec{A} = \vec{a}\) is a cause of \(\varphi\) in \((M,\vec{u})\) iff the following conditions hold:

AC1 \((M,\vec{u}) \models (\vec{A} =\vec{a}) \land \varphi\).

This says, if \(\vec{A} = \vec{a}\) is an actual cause of \(\varphi\) then they both hold in the actual world, \((M,\vec{u})\). Note, for this condition, we are just having a look at the model and not doing anything to it.

AC2 There is a partition of the endogenous variables in \(M\) into \(\vec{Z} \supseteq \vec{X}\) and \(\vec{W}\) and there are settings \(\vec{x’}\) and \(\vec{w}\) such that

(a) \((M,\vec{u}) \models [ \vec{X} \leftarrow \vec{x’}, \vec{W} \leftarrow \vec{w}] \neg \varphi\).

So, we’re trying to show that undoing the cause, i.e., setting \(\vec{X}\) to \(\vec{x’} \ne \vec{x}\), prevents the effect. We are allowed to modify \(\vec{W}\) however we want to show this, whilst leaving \(\vec{Z}-\vec{X}\) free to do whatever the model tells these variables to do.

(b) If \((M,\vec{u}) \models \vec{Z} = \vec{z^{\star}}\), for some \(\vec{z^{\star}}\), then for all \(\vec{W’} \subseteq \vec{W}\) and \(\vec{Z’} \subseteq \vec{Z}-\vec{X}\),
\((M,\vec{u}) \models [ \vec{X} \leftarrow \vec{x}, \vec{W’} \leftarrow \vec{w’}, \vec{Z’} \leftarrow \vec{z^{\star}}] \varphi\).

This says, trigger the cause (unlike AC1, we aren’t just looking to see if it holds) and check whether it leads to the effect under all subsets of \(\vec{Z}\) (as per actual world) that aren’t \(\vec{X}\) and all subsets of the modified \(\vec{W}\) that we found for AC2(a). Note how we are setting \(\vec{Z}\) for those subsets, rather than just observing it.

AC3 There is no \(\vec{A’} \subset \vec{A}\) such that \(\vec{A’} = \vec{a’}\) satisfies AC1 and AC2.

This says, there’s no superfluous stuff in \(\vec{A}\). You taking a painkiller and waving a magic wand doesn’t cause your headache to disappear, under AC3, if the painkiller works without the wand.

Example 1: an (actual) actual cause

Let’s give it a go with an overdetermined scenario (lightly edited from Halpern) that Alice and Bob both lob bricks at a glasshouse and smash the glass. Define

\(\mathit{AliceThrow} = 1\)
\(\mathit{BobThrow} = 1\)
\(\mathit{GlassBreaks} = \mathit{max}(\mathit{AliceThrow},\mathit{BobThrow})\)

So, if either Alice or Bob (or both) hit the glasshouse, then the glass breaks. Strictly speaking, I should have setup one or more exogenous variables, \(\vec{u}\), that define the context and then defined \(\mathit{AliceThrow}\) and \(\mathit{BobThrow}\) in terms of \(\vec{u}\), but it works fine to skip that step as I have here since I’m holding \(\vec{u}\) constant anyway.

Is \(\mathit{AliceThrow} = 1\) an actual cause of \(\mathit{GlassBreaks} = 1\)?

AC1 holds since \((M,\vec{u}) \models \mathit{AliceThrow} = 1 \land \mathit{GlassBreaks} = 1\). The first conjunct comes directly from one of the model equations and none of the functions change it. Spelling out the second conjunct,

\(\mathit{GlassBreaks} = \mathit{max}(\mathit{AliceThrow},\mathit{BobThrow})\)
\(= \mathit{max}(1, 1)\)
\(= 1\)

For AC2, we need to find a partition of the endogenous variables such that AC2(a) and AC2(b) hold. Try \(\vec{Z} = \{ \mathit{AliceThrow}, \mathit{GlassBreaks} \}\) and \(\vec{W}= \{ \mathit{BobThrow} \}\).

AC2(a) holds since \((M,\vec{u}) \models [ \mathit{AliceThrow} \leftarrow 0, \mathit{BobThrow} \leftarrow 0] \mathit{GlassBreaks} = 0\).

For AC2(b), we begin with \(\vec{Z} = \{ \mathit{AliceThrow}, \mathit{GlassBreaks} \}\) and the settings as per the unchanged model, so

\((M,\vec{u}) \models \mathit{AliceThrow} = 1 \land \mathit{GlassBreaks} = 1\).

We need to check that for all \(\vec{W’} \subseteq \vec{W}\) and \(\vec{Z’} \subseteq \vec{Z}-\vec{X}\),
\((M,\vec{u}) \models [ \vec{X} \leftarrow \vec{x}, \vec{W’} \leftarrow \vec{w’}, \vec{Z’} \leftarrow \vec{z^{\star}}] \varphi\).

Here are the combinations and \(\varphi \equiv \mathit{GlassBreaks} = 1\) holds for all of them:

\((M,\vec{u}) \models [ \mathit{AliceThrow} \leftarrow 1, \mathit{GlassBreaks} \leftarrow 1, \mathit{BobThrow} \leftarrow 0 ] \varphi\)
\((M,\vec{u}) \models [ \mathit{AliceThrow} \leftarrow 1, \mathit{BobThrow} \leftarrow 0 ] \varphi\)
\((M,\vec{u}) \models [ \mathit{AliceThrow} \leftarrow 1, \mathit{GlassBreaks} \leftarrow 1 ] \varphi\)
\((M,\vec{u}) \models [ \mathit{AliceThrow} \leftarrow 1 ] \varphi \)

(The third was rather trivially true; however, as far as I understand, has to be checked given the definition.)

AC3 is easy since the cause only has one variable, so there’s nothing superfluous.

Example 2: not an actual cause

Now let’s try an example that isn’t an actual cause: the glass breaking causes Alice to throw the brick. It’s obviously false; however, it wasn’t clear to me exactly where it would fail until I worked through this…

AC1 holds since in the actual world, \(\mathit{GlassBreaks} = 1\) and \(\mathit{AliceThrow} = 1\) hold.

Examining the function defintions, they don’t provide a way to link \(\mathit{AliceThrow}\) to a change in \(\mathit{GlassBreaks}\), so the only apparent way to do so is through \(\vec{W}\). Therefore, use the partition \(\vec{W} = \{\mathit{AliceThrow}\}\) and \(\vec{Z} = \{\mathit{GlassBreaks}, \mathit{BobThrow}\}\).

Now for AC2(a), we can easily get \(\mathit{AliceThrow} = 0\) as required, since we can do what we like with \(\vec{W}\). It doesn’t help when we move onto AC2(b) since we have to hold \(\mathit{AliceThrow} = 0\), which is the negation of what we want. The same is the case for the other partition including \(\mathit{AliceThrow}\) in \(\vec{W}\), i.e., \(\vec{W} = \{ \mathit{AliceThrow}, \mathit{BobThrow} \}\).

So, the broken glass does not cause Alice to throw a brick. The setup we needed to get through AC2(a) set us up to fail AC2(b).

References

Halpern, J. Y. (2015). A Modification of the Halpern-Pearl Definition of Causality. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015), 3022–3033.

Asking counterfactuals to think about controversial policies

“As an anecdote: one of us lives in a 1930s neighbourhood. The local municipality proposed converting a road connecting the neighbourhood to other neighbourhoods from a two-direction road to a one-way road. This would apply to motorized traffic only, not to bicycles. Many people protested because of the detour they would have to take by car. We asked the counterfactual question: suppose the city would have introduced the one-way road decades ago, would they then heavily support a policy to make it a two-direction road? Several people we asked did not know, the main reason for not supporting that change being that it would result in more traffic, noise and pollution, and a reduction in safety.”

– Van Wee et al. (2023, p. 84)

Reference

Van Wee, B., Annema, J. A., & Van Barneveld, S. (2023). Controversial policies: Growing support after implementation. A discussion paper. Transport Policy, 139, 79–86.

What is a counterfactual?

What’s a counterfactual? Philosophers love the example, “If Oswald hadn’t killed Kennedy, someone else would have”. More generally, Y would be y had X been x in situation U = u (Judea Pearl’s, 2011, rendering).

References

Pearl, J. (2011). The structural theory of causation. In P. McKay Illari, F. Russo, & J. Williamson (Eds.), Causality in the Sciences (pp. 697–727). Oxford University Press.