On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics

Abstract

Estimation of empirical relationships is prone to bias. Economists have carefully studied sources of bias in structural and quasi-experimental approaches, but the randomized control trial (RCT) has only begun to receive such scrutiny. In this paper, we argue that several lessons from medicine, derived from analysis of thousands of RCTs establishing a clear link between certain practices and biased estimates, can be used to reduce the risk of bias in economics RCTs. We identify the subset of these lessons applicable to economics and use them to assess risk of bias in estimates from economics RCTs published between 2001 and 2011. In comparison to medical studies, we find most economics studies do not report important details on study design necessary to assess risk of bias. Many report practices that suggest risk of bias, though this does not necessarily mean bias resulted. We conclude with suggestions on how to remedy these issues.

All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development

“…And when take-up is too low, researchers take a number of measures to 'force' it, thereby turning the evaluated programme into something that is far from a 'normal' programme (Bernard et al, 2012;Morvant-Roux et al, 2014;Quentin and Guérin, 2013). Spillover effects and attrition effects represent yet another unsolved issue, despite many efforts (Eble et al, 2014). 8 These 8.…”

“…problems, long recognized in medicine, have since been evidenced by many concrete examples, yet RCT promoters do not show any more reflexivity in their research protocols (Shaffer, 2011). That said, not all RCTs are equally undermined by such implementation distortions: the methodological properties of the RCTs conducted in development are systematically inferior to those conducted in medicine (in North and South alike) and those conducted to evaluate social policies in developed countries (Eble et al, 2014). The question of external validity is the most discussed in the literature.…”

From wealth to health: Evaluating microfinance as a complex intervention

“…Critical articles have been written on the use of RCTs in development (Barrett and Carter, 2010;Cartwright, 2007;Davies et al, 2014;Eble et al, 2013;Ogilvie et al, 2009;Shaffer, 2011;Toroyan et al, 2000;Worrall, 2007). Most of these critiques focus on the lack of generalizability of RCT results (Cartwright, 2011), but others reflect on flaws with respect to their internal validity and the difficulties of replicating clinical-type experiments in a broader social context (Deaton, 2010;Shaffer, 2011).…”

“…Most of these critiques focus on the lack of generalizability of RCT results (Cartwright, 2011), but others reflect on flaws with respect to their internal validity and the difficulties of replicating clinical-type experiments in a broader social context (Deaton, 2010;Shaffer, 2011). It has also been recognized that the economics literature on RCTs has, in the past, ignored reports on mistakes in the design, conduct and reporting of medical trials (Eble et al, 2013), some of which is related to the more balanced view within public health research that RCTs are not necessarily the 'gold standard' (Barrett and Carter, 2010;Cartwright, 2007;Deaton, 2010). Likewise there are serious ethical implications to consider in RCTs of socio-economic interventions (Barrett and Carter, 2010;Eble et al, 2013;Thomson et al, 2004).…”

“…It has also been recognized that the economics literature on RCTs has, in the past, ignored reports on mistakes in the design, conduct and reporting of medical trials (Eble et al, 2013), some of which is related to the more balanced view within public health research that RCTs are not necessarily the 'gold standard' (Barrett and Carter, 2010;Cartwright, 2007;Deaton, 2010). Likewise there are serious ethical implications to consider in RCTs of socio-economic interventions (Barrett and Carter, 2010;Eble et al, 2013;Thomson et al, 2004).…”