Daniel Ortega
Director de evaluación de impacto en CAF -banco de desarrollo de América Latina
Promoting the use of scientific knowledge in making decisions about public resources is an objective shared by many international organizations and development agencies. It is a complex task for many reasons. To begin with, the political class is not always motivated to use resources efficiently; however, there are openings and opportunities leading to courses of action that have the potential to make a real difference. Here we want to emphasize two strategies that are especially relevant for development agencies: promoting public institutions that learn, and bolstering north-south collaboration in the accessible dissemination of available global evidence.
Support for institutional capacities
At CAF we have worked with more than 200 public institutions in Latin America to support them in the process of designing and implementing impact evaluations. Through our competitive international appeal to public institutions, we have received more than 600 applications in the last 5 years from local, regional and national agencies and governments. Particularly outstanding in all this is the many and various abilities of the region’s institutions to embark on the development of complex and rigorous evaluation programs.
Our processes, and even the incentives derived from academic interest of certain projects, typically lead us to select the institutions with the most promising projects: those with the best data, the most precisely defined implementation processes, the greatest academic support, the best trained staff, and the most aligned leadership. We usually support the institutions that will allow us to carry out impact assessments with the best causal identification strategies. This is necessary, since it contributes to the base of evidence available to the global community, which is one of the major objectives we are pursuing.
However, there is a problem with this rationale. This process tends to leave the weakest institutions behind, and the vast majority of public institutions in Latin America have little to no learning capacity. In other words, they have little understanding of the theory of change to their policies, no systems to monitor the implementation of the policy or to generate reports on the processes and their challenges, no clear idea about the intermediate and final indicators that would be more informative, and no process or bandwidth to digest and learn from any information or understanding that may come about as part of the implementation. And so our selection processes, although effective in producing the best impact assessments, leave out a group of institutions that need a lot of support to improve their abilities.
We are in a position take on many more public institutions that recognize the value of keeping quality records of their activities, that devote some resources to visualizing and analyzing their data, and that are willing to share and learn from such data even if the impact assessments are not provided to beneficiaries through random allocation. The potential gains from small changes in our institutions’ efforts to learn can be very large.
In this regard, reducing inequality between institutions tends to increase efficiency. For example, in education or health, if local governments share competences by providing services with the national government, more capable local institutions can facilitate the administration of national services, through, for example, the generation of better statistics for diagnosing local problems.
One part of this program is therefore to bridge the gap between public institutions and data analysts; there are untapped gains from this exchange. An initiative, led by CAF’s Socioeconomic Research Committee called “Hands on DATA”, brings together public institutions with data analytics needs with teams of data scientists to collaborate on projects that create prototypes of solutions to very specific public policy problems. These collaborations are cost-effective and can lead to institutional relationships that exceed the life of the original project, benefiting everyone.
We can therefore pay more attention to institutions that are lagging behind in their potential to generate quality causal evidence, since it is possible that they handle as much if not more resources than those that are more sophisticated.
Collaboration in evidence translation
Translating evidence is not just about rendering it into another language, but also about making scientific evidence accessible to people that have no technical training.
Many institutions, such as Campbell Collaboration and 3iE, have made significant strides in generating evidence synthesis products. While these institutions have played an important role in this regard, more effort is needed on a global scale. This is especially crucial for the developing world, where a significant part of quality empirical research is currently taking place, but where there is less public debate about the scientific merits of the various alternatives of public policy.
The work of the “What Works” network in the UK is the kind of work that by its very nature is globally relevant. A cycle that includes the production of new evidence, synthesis, translation, dissemination, followed by implementation and re-evaluation, somewhat describes the ideal cycle of policy learning, but translation plays a crucial role because it is the only way to facilitate adoption by those who cannot easily attain access to the evidence. Some developing countries, or some institutions in these countries, may be interested in finding the best available evidence for their policy alternatives, but find such evidence inaccessible or difficult to contextualize.
It is possible to consider a broad collaboration that follows the lead of the best evidence portals and that ensures that these are accessible to all audiences in all languages, that they are updated regularly, and that they are flexible enough to be transparent to the user in terms of specific evidence that may be locally relevant and for which policies there is not much quality evidence to form an opinion beforehand. Some developed countries have made significant progress. On the other hand, the developing world lags behind even though the marginal cost of making these resources available to them is minimal. International organizations can play an important role in promoting this type of collaboration.
Influencing policy decision making from the supply side is not without its challenges; it requires a special connection with the relevant stakeholders, and may also require significant innovation in communication strategies. In the last year, it has been apparent that public opinion of vaccination has been important for reaping its full benefits. The rapid production of evidence, dissemination and adoption by the most important policy stakeholders globally has not been enough; there are still significant groups of people around the world who choose not to get vaccinated, despite the best available scientific evidence. Many of us thought that the typical user of scientific evidence would be whoever makes decisions about programs or policies that affect the population; however, this year suggests that in some areas the very beneficiaries of the policies have to be persuaded of the state of our knowledge.
