Did this policy actually cause the result?

A programme starts. A measured outcome improves. The sequence is worth investigating, but it does not by itself measure the programme’s effect.

The missing comparison is the counterfactual: what would have happened to the same outcome without the intervention. The World Bank’s impact-evaluation guide explains why a simple before-and-after comparison can confuse a programme’s effect with other changes over time. 1

A calculation that shows the problem

Entirely hypothetical example: A school programme begins in one group of schools. Absence falls from 20% to 12%. In a comparison group without the programme, absence falls from 18% to 14% over the same period.

The first group improved by 8 percentage points. But a comparison-adjusted change is −8 − (−4) = −4 percentage points. This is the arithmetic behind a simple difference-in-differences comparison, not proof that any real Indian programme reduced absence by that amount.

That interpretation depends on assumptions. In particular, absent the programme, the groups’ outcomes must have followed sufficiently comparable trends, and other changes must not differentially explain the result. The World Bank guide discusses examining earlier trends while warning that such checks cannot conclusively prove the assumption. 2

Check who received the programme

A comparison can be distorted if participating schools were selected because they had unusual resources, unusually severe problems or a different expected trajectory. Similar averages before the programme do not eliminate every difference.

Ask whether the comparison group was affected indirectly, whether another intervention occurred at the same time and whether data collection changed. Also check whether the outcome was chosen before analysis or selected afterward because it looked favourable.

These questions do not make evaluation impossible. They identify which explanations a credible study must address.

Administrative success is not automatically causal success

A department can accurately report that it built facilities or paid benefits without establishing the size of their effect on health, education or income. The government’s output-outcome framework itself distinguishes direct outputs from intended results. 3

Equally, an outcome can fail to improve even when an intervention prevented a worse decline. The absence of a simple upward trend does not settle causation either. A credible comparison matters in both directions.

How to read a strong causal claim

Look for the exact intervention, outcome, population and period. Then find the comparison method, assumptions, uncertainty and alternative explanations. Check whether the published conclusion is narrower than the headline.

Do not turn “associated with” into “caused by”. Do not turn a study of one population into a guarantee for every district. A transparent observational study may still be useful when its limits are stated; pretending those limits do not exist makes it less useful.

A practical verdict can remain provisional: “The outcome improved after the programme, but this evidence does not isolate its effect.” That sentence preserves the observed improvement without inventing a causal estimate.