There is a simple formula for delivering evaluation findings in time to inform policymaking: \(T = S + R + D + I + A\).
Suppose that, for a programme to be considered effective, it must demonstrate a practically significant causal impact at least \(I\) months after the end of programme activities.
The programme duration is \(D\) months from the point at which a participant is enrolled in the evaluation.
Given recruitment rates, participants are enrolled over a period of \(R\) months to ensure sufficient numbers to achieve the required statistical power. The final participant therefore completes all programme activities \(R + D\) months after recruitment begins.
Study setup takes \(S\) months. This includes developing a theory of change, study design, developing surveys and topic guides, finalising the evaluation protocol, ethical approval, and recruiting study sites.
Analysis and reporting takes \(A\) months. This includes time waiting for administrative data requests to be fulfilled or for fieldwork to conclude, data management, statistical disclosure control and output clearance, peer review, and revisions.
The total study time is therefore \(T = S + R + D + I + A\) months. (You could add more steps, but it wrecks the anagram.)
For example, suppose the timeline is:
| Setup | 3 months |
| Recruitment period | 6 months |
| Duration of the programme | 3 months |
| Impact time point | 6 months |
| Analysis and reporting | 3 months |
The total duration \(T = 21\) months. So start the study 1 June 2026, it will report by 1 March 2028. And this evaluation only estimates impact at 6 months.
Suppose you want findings earlier than this as you need to inform an urgent policy decision. What could you do? Some options:
1. Analyse existing data for a similar programme, e.g., using meta-analysis of studies in a systematic review or quasi-experimental analysis of admin data. You can also gather implementation and process evidence to evaluate whether the current programme is being implemented correctly, even if you don’t have time to estimate impacts (use the review or secondary analysis of earlier similar programmes for that).
2. Investigate shorter-term outcomes, relying on existing evidence from similar programmes to argue that effects will endure. For example, it could be that you are really interested in impact at 10 years; however, the literature suggests that an outcome you can measure at 1 year is strongly correlated with the longer-term impact and the theory of change suggests it is an important link in the causal chain.
3. Shorten the programme duration, risking smaller impact and an underpowered study, so you may also need to boost the sample size, e.g., by lengthening the recruitment period.
4. Increase recruitment rates, e.g., by including more study sites or contacting more potential participants (depending on the nature of the programme).
5. Shorten the recruitment period, risking an underpowered study so you will likely have a wide confidence interval that straddles zero. Light a candle to guide interpretation.
6. Combine 4 and 5 to achieve the required sample size faster.
7. Run a simpler study, reducing setup, analysis, and reporting time. For example, you might measure only one or two key outcomes and conduct highly focused implementation and process evaluation.
8. Combine analysis of existing data with running a new evaluation: use option 1 (analyse existing data) to inform your policy design now, and also run the \(T\)-month evaluation to help out your future policy colleagues when they review the literature in a few years’ time.
9. Use time and relative dimensions in space-adjusted analysis to deliver the originally planned evaluation in time to feed into the policy decisions.
Option 9 seems to be what policymakers expect. My preference would be to accept our spacetime structure and go for option 8 where possible (with the help of 2) or 1 if not. In any case, ensuring that evaluations can complete in time to inform policy requires long-term planning that is resilient to changes of PM, Ministers, and governments.
Thanks Edisa for inspiring a mild but crucial tweak to the variable names!