[in reply to a paper and blog post by Dani Rodrik, 3.6.2008, here]
I just finished reading your paper. Other than the external versus internal validity issue, you point out that top journals may not have an incentive to publish the results of two identical randomized evaluations in different settings. Maybe this would have an adverse impact on the academic careers of the randomistas?
There are other practical issues that make it difficult for developing country organisations to carry out a randomized trial at ground level. First, a randomized evaluation is expensive (consulting fees and out-of-pocket expenses of a large team of field researchers plus faculty from an overseas college working in a project location to name the obvious). Few developing country organisations, let alone NGOs in these countries, can afford this unless they are propped up by corporate or donor agency funds (or by the funds of the university where the core team of economists come from).
Second, a randomized evaluation is time consuming and takes a year or more to come up with the final numbers. Donor agency funded projects have shorter timelines to showcase results to both the recipient government and to the government to which it reports to. So these agencies tend to stay away from randomised trials of micro-interventions. (I understand that the World Bank’s OED is now increasing its use of such trials but I haven’t come across a visibly increasing number of World Bank funded trials in India, surely a good place to begin?)
As for NGOs, apart from the prohibitive costs, does it really pay to conduct such a lengthy exercise all for the sake of some numbers? I would wager it does not as NGO funds are hardly dependent on any evaluation. In most cases, they are funded because of their track record of community work, relationship with community heads and local government officials and perceived integrity (many NGOs in India are fly-by-night operations).
For government agencies, the financial aspects are not important but the political implications of randomised trials are. While a case can be made that an evaluation that ‘proves’ the successful effect of an intervention can be used to garner political support for more of that intervention, what happens when an evaluation shows up a generously funded government intervention to have had no or very little effect on the indicators that it was trying to influence? I can safely say that key ministries in the government of India do not want to be put under that sort of scanner. (Could one imagine the Ministry of Health, one of the more notorious ministries in India and recently implicated by the World Bank for massive corruption in drug procurement, open its doors to randomised trials on a large scale?)
Next, take a look at Banerjee and Duflo’s Abdul Latif Jameel Poverty Action Lab (PAL) that has carried out, and is working on, several randomised trials in India. They rope in local NGOs to provide ‘buy-in’ with the local community and a team of Indian field coordinators, typically local boys and girls, to do the spade work of collecting the data. Though I don’t know how these projects are funded by PAL, I am quite sure that these NGOs (e.g. Pratham or Sewa) could not bear the full costs of such evaluations through a period of one year or more. In that case, the funds for the projects are coming out of research grants or from the funds of some usually US university or from corporate firms (or some combination of all three). So it is the researchers who are financially supporting the others to finish the research.
Also, PAL has a good working relationship with a think-tank in Chennai (Institute for Financial Management and Research) that is financially supported by ICICI Bank, one of India’s largest banks. ICICI’s interest is to gain insights into the country’s promising microfinance and rural credit market which it believes can be better understood by randomised evaluations. IFMR is now a place for regular workshops on randomised trials conducted by the PAL team and researchers from other US universities (e.g. Dean Karlan of Yale who also works on microfinance and credit products).
In short, randomised trials will not naturally be taken up by governments, NGOs or donor agencies working in developing countries just because it is ‘internally valid’ because of time, cost and political considerations. In that case, they will be driven by funding from corporate (towards business interests like the pharmaceutical industry) and research funds (towards the publications of young assistant professors) – as is the case with PAL and IFMR.