A rusting gold standard: Failures in an Indonesian RCT, and the implications for poverty reduction

Abstract

Randomised Control Trials (RCT) are both widely used in development, reaching hundreds of millions through RCT informed policy, and highly regarded, receiving a Nobel Prize in Economics. Proponents, largely academic economists, position RCTs as a scientific and ideologically neutral way to get to the heart of ‘what works’ in development. However, this new and radical micro-experimental approach to poverty reduction has sat uneasily within the broader development sphere, including critical geographers. We contribute to this debate by empirically examining the practical manifestation of the methodology within ‘PNPM Generasi’ – an innovative Indonesian cash transfer program evaluated through the largest RCT to date. Drawing primarily on field interviews, our examination finds three departures from the gold standard proclamations of the RCT methodology in practice. First, we find that randomising aid delivery breached ethical guidelines and compromised the effectiveness of the program being studied, largely as a result of RCTs being run in secret from local communities and politicians. Second, we find applying the RCT methodology on the ground was technically complex and financially costly, requiring not only an excessive use of scarce program funds, but also undermining the validity of the evaluation results. Finally, we find RCT results were not widely used in practice. We demonstrate these findings are not restricted to our specific case, but rather, reflect systemic deficiencies with RCT evaluations. Our results suggest that RCTs may have value if applied judiciously, and as part of multi-pronged approaches, but we caution against their growing monopoly influence on poverty reduction.