Methodological Note: Difference in Differences to Measure Impact

When assessing the impact of a public program, randomization may not be feasible due to logistical difficulties, ethical or political issues, or because the intervention has already taken place.

When this occurs, the treatment (program recipient) and control (non-recipient) groups will likely be very different from each other.

This problem can be fixed if we have data on the variables of interest for various points in time, especially before and after data. This information could help measure the impact of the program by implementing the so-called Difference in Differences methodology.

This methodology consists in applying a double difference, i.e. comparing the changes in time in the variable of interest between the treatment group and the control group.

To ensure the validity of this methodology, the treatment and control groups do not need to be similar in their observable characteristics. Unlike randomized experiments, where the balance condition is essential when measuring impact, in the difference in differences method, the necessary condition to evaluate impact is the so-called parallel trend assumption. This assumption establishes that in the absence of treatment, the two groups would have followed the same trend in the results of interest.

Suppose we have a group of companies that receive a grant for innovation promotion (treatment) and another group that does not receive such grant (control). When averaging the observable variables for each group, namely: year of incorporation, number of employees and turnover, the means of both groups, prior to intervention, should not be statistically equal. However, it is particularly relevant that the trend of each of these result variables would have followed a similar trajectory had treatment not occurred. As this assumption cannot be demonstrated, since the treatment has already occurred, we can determine whether in the years leading up to the intervention the trend of both groups behaved similarly.

Once this assumption is verified, we are able to measure the impact by applying this methodology. First, the difference (change) must be calculated, for both the treatment and the control groups, between the value of the variable of interest on the follow-up line (e.g. one year after the program) and the baseline (e.g. one year prior to the program). Second, since the change in the control group’s variable of interest represents the change that the control group would have experienced in the treatment group in the absence of the program, the second difference is materialized between the estimated changes for the treatment and control groups. The result is the average effect of treatment. (See Figure 1)

Figure 1. Difference in Differences Model

The difference in differences methodology has been applied in multiple research topics.

• Di Tella and Schargrodsky (2004) use the difference in differences model to measure the impact of increased police presence on Jewish and Muslim public buildings in Argentina following the terrorist attack on the Amia building in 1994. The authors found that the streets that benefit from police presence in these buildings experience fewer car thefts per month (-75%) compared with streets without this protection.
• Lichand et al. (2016) use this methodology to measure the impact of an anti-corruption program in Brazil, which seeks to enforce the rules on transfers assigned to municipalities. Their findings include that in the municipalities audited by the anti-corruption program, cases of over-invoicing and unofficial payments were significantly reduced, as well as manipulation in obtaining healthcare-related transfers, compared to unaudited municipalities. However, increased control resulted in a deterioration of healthcare indicators, namely: infrastructure, availability of medicines, difficulty meeting local needs, among others.
• Gertler and Boyce (2003) measure the impact of PROGRESA, a conditional transfer program for children’s healthcare and education in Mexico. These authors use the difference in differences method to measure the impact of this program on healthcare outcomes. They found that in areas where PROGRESA beneficiary families live, there is a greater number of preventive care visits to public clinics (up to +60%) than in areas where there are no beneficiaries of this program.

If you are interested in learning more about this methodology, theory and applications, as well as other quasi-experimental impact measurement methodologies, we invite you to participate in one of our MOOCs .

We invite you to take our MOOC on Public Management Impact Assessment (Intermediate) (2nd ed.) which will be available on April 16. Sign up

References:

Angrist, J., & Pischke, J.-S. (2015). Mastering Metrics. New Jersey: Princeton University Press.

Gertler, P., Martínez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2011). Impact Evaluation in Practice. Washington D.C.: World Bank.