Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field

Abstract

Policy-makers and managers have always used a wide range of sources of evidence in making decisions about policy and the organization of services. However, they are under increasing pressure to adopt a more systematic approach to the utilization of the complex evidence base. Decision-makers must address complicated questions about the nature and significance of the problem to be addressed; the nature of proposed interventions; their differential impact; cost-effectiveness; acceptability and so on. This means that Cochrane-style reviews alone are not sufficient. Rather, they require access to syntheses of high-quality evidence that include research and non-research sources, and both qualitative and quantitative research findings. There is no single, agreed framework for synthesizing such diverse forms of evidence and many of the approaches potentially applicable to such an endeavour were devised for either qualitative or quantitative synthesis and/or for analysing primary data. This paper describes the key stages in reviewing and synthesizing qualitative and quantitative evidence for decision-making and looks at various strategies that could offer a way forward. We identify four basic approaches: narrative (including traditional 'literature reviews' and more methodologically explicit approaches such as 'thematic analysis', 'narrative synthesis', 'realist synthesis' and 'meta-narrative mapping'), qualitative (which convert all available evidence into qualitative form using techniques such as 'meta-ethnography' and 'qualitative cross-case analysis'), quantitative (which convert all evidence into quantitative form using techniques such as 'quantitative case survey' or 'content analysis') and Bayesian meta-analysis and decision analysis (which can convert qualitative evidence such as preferences about different outcomes into quantitative form or 'weights' to use in quantitative synthesis). The choice of approach will be contingent on the aim of the review and nature of the available evidence, and often more than one approach will be required.

Systematic reviews: from ‘bare bones’ reviews to policy relevance

“…Examples of these approaches include content analysis, thematic analysis without theory generation, meta-summary, qualitative case survey and Bayesian meta-analysis (Dixon-Woods et al 2005, Mays et al 2005, Noyes and Lewin 2011. These approaches often transform qualitative data into quantitative through coding and counting.…”

“…An increasing number of reviews are also adopting less traditional approaches to evidence synthesis, including different types of qualitative and mixed methods synthesis (Noyes 2010). A review of this literature, as well as the experience of 3ie's systematic reviews program and approach to impact evaluation (White 2009), suggests there are two key components to adapting the traditional approach to systematic reviews and producing more useful and higher quality reviews in international development, namely the use of theory (Petticrew and Roberts 2006, Gardner et al 2010, Tugwell et al 2010, Anderson et al 2011) and the need to include a broader range of evidence (Dixon-Woods et al 2005, Mays et al 2005, Oliver et al 2005, Popay 2005, Baxter et al 2010.…”

“…Poor indexing, obscure titles and lack of structured abstracts mean that searching for qualitative studies is more challenging than searching for effectiveness studies (Mays et al 2005). As a result, authors often have to obtain and review a high number of full-text articles to determine which articles meet the inclusion criteria, which, of course, has time and resource implications (Atkins et al 2008).…”

Methods of defining best practice for population health approaches with obesity prevention as an example

“…External validity, or generalizability, appears to be a secondary concern in traditional approaches to critical appraisal that rely on RCT design (Mays et al 2005). In contrast, assessment of external validity is a critical component of population health approaches used in public health, as it takes into consideration the contextual factors that provide information allowing for judgments on applicability beyond the study population.…”

“…A few published reports describe expanded approaches for conducting systematic reviews to identify best practice for population health approaches, which include the community guide to preventive services (Briss et al 2000) and a review of methods used to synthesize evidence for effectiveness in health promotion (Jackson et al 2001). In addition, those researchers concerned with making systematic reviews more relevant to policy makers have proposed even broader approaches to the process of synthesizing evidence (Atwood et al 1997;Lomas, et al 2005;Mays et al 2005;Pawson et al 2005), highlighting disparities in how researchers and policy makers define and describe evidence.…”