The International Initiative for Impact Evaluation (3ie) is an international grant-making NGO promoting evidence-informed development policies and programmes. We are the global leader in funding, producing and synthesising high-quality evidence of what works, for whom, why and at what cost. We believe that high-quality and policyrelevant evidence will make development more effective and improve people's lives. 3ie evidence gap maps 3ie evidence gap maps (EGMs) are thematic collections of information about impact evaluations or systematic reviews that measure the effects of international development policies and programmes. The maps present a visual overview of existing and ongoing studies in a sector or sub-sector in terms of the types of programmes evaluated and the outcomes measured. EGM reports provide all the supporting documentation for the maps themselves, including the background information for the theme of the map, the methods and results, including the protocols and the analysis of the results. 3ie EGMs are available through an online interactive platform on the 3ie website that allows users to explore the studies and reviews that are included. About this evidence gap map reportThis report summarises the methods and findings of an EGM on interventions that seek to improve state-society relations, which was developed by 3ie with funding from USAID and under contract with NORC at the University of Chicago. The online map can be found here.All of the content is the sole responsibility of the authors and does not represent the opinions of 3ie, its donors, Board of Commissioners, NORC, USAID or the US government. Any errors and omissions are the sole responsibility of the authors. Any comments or queries should be directed to the corresponding author, Daniel Phillips dphillips@3ieimpact.org. AcknowledgementsWe gratefully acknowledge funding of this project from the United States Agency for International Development (USAID) and the support of NORC at the University of Chicago for contract administration. We are grateful to colleagues who provided inputs at various stages of the map: Edoardo Masset, Annette Brown, Birte Snilstveit, Kristen Rankin, Jorge Hombrados, Francis Rathinam, Deepthy Menon and Diana Milena Lopez Avila. We would also like to thank the members of our advisory board: Eric Kramon, Jennifer Stuttle, Clare Mcloughlin, George Ingram and Leonard Wantchekon. We are grateful to staff at USAID and NORC at the University of Chicago who helped inform the map's development. Thanks also to all the authors who shared completed and ongoing papers and helped improve the map, as well as all those who helped by providing comments, in particular our external peer reviewer. All errors are authors' own.ii Summary BackgroundThe role of the state, the effectiveness of its institutions and its legitimacy in the eyes of its citizens are central to determining a country's prospects for stability and development. The critical importance of the state-society relationship to the global development agenda is undersco...

No abstract

Executive SummaryBackground: Many systematic reviews incorporate nonrandomised studies of effects, sometimes called quasi-experiments or natural experiments. However, the extent to which nonrandomised studies produce unbiased effect estimates is unclear in expectation or in practice. The usual way that systematic reviews quantify bias is through "risk of bias assessment" and indirect comparison of findings across studies using meta-analysis. A more direct, practical way to quantify the bias in nonrandomised studies is through "internal replication research", which compares the findings from nonrandomised studies with estimates from a benchmark randomised controlled trial conducted in the same population. Despite the existence of many risks of bias tools, none are conceptualised to assess comprehensively nonrandomised approaches with selection on unobservables, such as regression discontinuity designs (RDDs). The few that are conceptualised with these studies in mind do not draw on the extensive literature on internal replications (within-study comparisons) of randomised trials.Objectives: Our research objectives were as follows:Objective 1: to undertake a systematic review of nonrandomised internal study replications of international development interventions.Objective 2: to develop a risk of bias tool for RDDs, an increasingly common method used in social and economic programme evaluation.Methods: We used the following methods to achieve our objectives.Objective 1: we searched systematically for nonrandomised internal study replications of benchmark randomised experiments of social and economic interventions in low-and middle-income countries (L&MICs). We assessed the risk of bias in benchmark randomised experiments and synthesised evidence on the relative bias effect sizes produced by benchmark and nonrandomised comparison arms.Objective 2: We used document review and expert consultation to develop further a risk of bias tool for quasi-experimental studies of interventions (ROBINS-I) for RDDs.---

The Extractive Industries Transparency Initiative (EITI) is internationally recognised as a leading anti-corruption scheme, which promotes transparency, accountability and good governance of public oil, gas, and mining revenues. This article provides the first rigorous quantitative investigation of the impact of EITI on corruption in Zambia. Using a case-comparison approach, called the Synthetic Control Method (SCM), we find that the implementation of EITI provoked a significant decrease in corruption in Zambia (with the corruption-reducing effect of EITI being, though, much stronger at the earlier stages of implementation)

No abstract

Background: Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. Objectives: We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. Research Design: We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. Results: We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least “some concerns” about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. Conclusions: The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.

No abstract

No abstract