3 key aspects for a rigorous impact evaluation

Ideally, for an impact evaluation, all units should comply with their initial assignment, i.e., all those assigned to the treatment group should actually be treated, while none of the control units are. In these cases, an evaluation is said to have “perfect compliance,” which means, provided that group assignment has been random, that the difference in the variable of interest observed between the two groups after the intervention will be the impact attributable to the program or policy under evaluation.    

However, unlike other fields of study, this tends to be difficult to achieve in social programs. First, this is due to the fact that policy implementers tend to only have the ability to propose (not force) participation in a program, and thus, the decision to participate will depend on the need and the motivation of each unit. Similarly, part of the control group might gain access to the program by having some kind of relationship with the implementer, as a result of political pressure, due to an error, etc. In short, imperfect compliance occurs when units assigned to treatment do not receive the program, or when control units end up receiving it.

Therefore, non-compliance is a recurrent factor and it is important to take it into account when assessing the impact of any initiative, in order to avoid inaccurate results or conclusions about their effectiveness. In this connection, we present three examples about this issue when performing impact analysis:

1. Imperfect compliance changes the interpretation of the of simple mean difference estimator between treatment and control
   
   Under imperfect compliance this difference cannot be taken as the average impact of the intervention, an estimator commonly known as “average treatment effect” (ATE), since these do not compare only the units under the program vs. non-recipients. On the other hand, this indicator becomes “intention to treat” (ITT) and is interpreted as the impact of offering participation in a program.
   
   However, this estimator can be informative, especially when imperfect compliance focuses on the treatment group, a more common scenario when an assessment includes this condition. In this particular case, ITT takes into account that a program’s success depends not only on the quality of implementation, but also the capacity to encourage participation in and adherence to such program.
   
   
2. Excluding non-compliers from the impact analysis would lead to biased results
   
   Taking only compliers, i.e. treatment units that were treated and control units that were not treated, is not recommended, because this would likely lead to comparing two groups that ceased to be equivalent, which would produce unreliable results.
   
   For example, in the case of a college scholarship program for young people with limited budgets, members of the treatment group who accept the scholarships (compliers) may have a greater motivation to study, perhaps because they come from families who value education more. Thus, comparing enrolment levels taking only this subset (motivated people) as treatment group and the entire control group (including motivated and non-motivated youngsters), would lead to overestimating the impact of this intervention.
   
   
3. Even with imperfect compliance it is possible to come close to the average impact of the intervention
   
   ITT reference is taken as a starting point, i.e. the average impact of offering participation in a program. While there is certainty that this estimator does not represent the average impact of participating in the program under perfect non-compliance, it is taken as an expected minimum value, taking into account that non-compliance will tend to dilute the average effect due to the mere fact that some treated units were not affected or some control units were. In other words, if the value of the impact variable is expected to increase thanks to the program, in the absence of non-compliance, the average value in the treatment group should be higher, and the average value in the control group should be lower.
   
   Based on the foregoing, calculating an average impact requires obtaining a number higher than that obtained with the ITT that ensures comparability of groups (see point 2). To this end, compliance rates of the treatment group and non-compliance rates of the control group are included. Thus, if non-compliance is concentrated only in treatment, ITT is divided between the rate of compliance, an estimator known as “treatment on the treated” (TOT). Also, if there is non-compliance both in treatment and control, ITT is divided by the subtraction of the rate of compliance of treatment and non-compliance of control, an indicator known as “local average treatment effect” (LATE). These estimates allow us to fulfill two goals: taking into account non-compliance and preserving the original composition of the sample.

Imperfect compliance is a fairly frequent phenomenon in impact evaluations of social programs and is a consequence of the inherent difficulties when using an experimental methodology outside a laboratory, where it is much more difficult to control the conditions under which an experiment takes place. However, this should not be taken as an impediment that renders the results of an evaluation invalid, or that prevents us from drawing lessons. On the other hand, non-compliance should simply be incorporated into the analysis in order to obtain better estimators to help make better decisions.