About 3ieThe International Initiative for Impact Evaluation (3ie) is an international grant-making NGO promoting evidence-informed development policies and programmes. We are the global leader in funding and producing high-quality evidence of what works, how, why and at what cost. We believe that better and policy-relevant evidence will make development more effective and improve people's lives. 3ie scoping papers 3ie thematic window grant programmes typically start with a consultative process that includes a scoping study that identifies the current state of impact evaluation evidence in a particular sector. Scoping studies lay out the landscape of what is known and help identify priority policy questions. Although thematic windows are conducted in response to donor demand, this consultation process gives developing country stakeholders a voice in setting key priorities and identifying research questions. About this scoping paperThis scoping paper provides an independent analysis of the evidence base of evaluations in humanitarian assistance and identifies key gaps and priorities in need of rigorous evidence. It is part of background scoping research and consultation undertaken to assess the scope and methods for impact evaluation in the humanitarian sector. The working paper, What methods may be used in impact evaluations of humanitarian assistance? examines the extent to which impact evaluation methods can provide evidence to help improve the effectiveness and efficiency in humanitarian action. All of the content is the sole responsibility of the authors and does not represent the opinions of 3ie, its donors or its Board of Commissioners. Any errors and omissions are also the sole responsibility of the authors. Any comments or queries should be directed to the corresponding author Mike Clarke at m.clarke@qub.ac.uk AcknowledgementsThis paper is part of a larger study supported by UKaid through the Department for International Development and USAID. The scoping paper examines the scope of evidence and need for evidence in humanitarian assistance. The scoping paper has been prepared by Evidence Aid with support from 3ie. 3ie, along with assistance from the SIPRI and humanitarian assistance experts, led the work on the methods paper.Jyotsna Puri provided overall leadership and management of these papers with support from Deo-Gracias Houndolo and Peter Giesen. Bharat Dhody provided research assistance. ii Executive summaryThis paper investigates the current landscape of evidence, with particular emphasis on evidence from impact evaluation, in the humanitarian sector. This is in an attempt to identify areas in which actionable evidence is available and those where more evidence is needed so as to direct research to where it will be most valuable.The study incorporates a wide array of methods to assess available evidence including an online survey of 395 participants who are knowledgeable with regards to the humanitarian sector, semi-structured interviews with 53 experts from the humanitarian sector, extensive lit...

The proposed Comprehensive Framework for Disaster Evaluation Typologies outlines the different typologies of disaster evaluations that were identified in this study and brings them together into a single framework. This unique, unifying framework has relevance at an international level and is expected to benefit the disaster, humanitarian, and development sectors. The next step is to undertake a validation process that will include international leaders with experience in evaluation, in general, and disasters specifically. This work promotes an environment for constructive dialogue on evaluations in the disaster setting to strengthen the evidence base for interventions across the disaster spectrum. It remains a work in progress. Wong DF , Spencer C , Boyd L , Burkle FM Jr. , Archer F . Disaster metrics: a comprehensive framework for disaster evaluation typologies. Prehosp Disaster Med. 2017;32(5):501-514.

Introduction:The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.Research Problem:Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.Methods:A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.Results:A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.Conclusion:For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.

No abstract