Lessons We Can Draw From “Non-significant” Results

Determining the effect of a program through an impact assessment involves running a statistical test to calculate the probability that the effect, or the difference between treatment and control groups, is a random result. If such probability is low enough, then the difference between the groups is real (or statistically significant) and therefore, the program has a— positive or negative—impact. When the probability does not meet that condition, the program result is null, i.e. there is no statistically significant difference between the treatment and control groups.

Null results alone do not produce a concrete response, and the intervention may not actually have any effect, but we cannot rule out the possibility that the impact assessment did not had the statistical power to detect it. Whatever the case, there is still information behind a null result (associated with the quality of implementation, the take-up of beneficiaries, statistical power, etc.) that can produce valuable lessons for decision-making in public management, as long as we have the necessary tools to extract them. In order to draw this information, we can perform some activities in addition to the impact assessment, which may help explain the null results, if any:

• Performing a design evaluation: A null result can come from an intervention that is not aligned with the context or theory of change, through which the impacts are expected to be generated. Conducting a design evaluation before program implementation will ensure that the assumptions and mechanisms on which the theory of change is based are appropriate and in line with the context, in order to produce the desired impacts on the target population.
• Surveying implementation information: The poor quality of the implementation and/or the limited level of participation of beneficiaries may also result in null impacts because, faced with these barriers, activities or mechanisms defined in the theory of change are not performed. Monitoring the implementation through an information survey is key to verifying that all activities are being performed and beneficiaries are receiving the program. Supplementing this information with a qualitative assessment can also add value and help explain the results behind the impact. The data obtained from both inputs will help verify the implementation and provide answers in case of null results.
• Calculating the statistical power before and after the intervention: The statistical power helps determine the minimum effect that can be detected with a given sample. These calculations are usually made before starting the assessment to confirm which effects can be detected, given a number of beneficiary units. It is useful to perform the exercise again at the end of the implementation using the actual data of the target population, to adjust the initial power calculations. If the effect obtained is lower than the minimum detectable effect of such exercise, we can conclude that it was not possible to detect the impact due to its low statistical power. In addition, the effect of a program may vary between groups within the sample (e.g. between women and men), and therefore, verifying the calculations for these groups can also shed light on heterogeneous effects of the intervention.

Sometimes, when performing impact assessments, we are unknowingly biased against null results, and thus, we expect the assessed programs to produce the desired impacts, and underestimate the information and lessons that can be drawn from non-significant results, which require further reflection on the intervention, mechanisms and results. While it is true that these activities require additional time, resources and efforts, they will also ensure that, whatever the result, it will be possible to obtain useful and informative inputs for public management.