The impact of Cochrane Reviews: a mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research

Abstract

BackgroundThe last few decades have seen a growing emphasis on evidence-informed decision-making in health care. Systematic reviews, such as those produced by Cochrane, have been a key component of this movement. The National Institute for Health Research (NIHR) Systematic Review Programme currently supports 20 Cochrane Review Groups (CRGs) in the UK and it is important that this funding represents value for money.Aims and objectivesThe overall aim was to identify the impacts and likely impacts on health care, patient outcomes and value for money of Cochrane Reviews published by 20 NIHR-funded CRGs during the years 2007–11.DesignWe sent questionnaires to CRGs and review authors, undertook interviews with guideline developers (GDs) and used bibliometrics and documentary review to get an overview of CRG impact and to evaluate the impact of a sample of 60 Cochrane Reviews. The evaluation was guided by a framework with four categories (knowledge production, research targeting, informing policy development and impact on practice/services).ResultsA total of 3187 new and updated reviews were published on the Cochrane Database of Systematic Reviews between 2007 and 2011, 1502 (47%) of which were produced by the 20 CRGs funded by the NIHR. We found 40 examples where reviews appeared to have influenced primary research and reviews had contributed to the creation of new knowledge and stimulated debate. Twenty-seven of the 60 reviews had 100 or more citations in Google Scholar™ (Google, CA, USA). Overall, 483 systematic reviews had been cited in 247 sets of guidance. This included 62 sets of international guidance, 175 sets of national guidance (87 from the UK) and 10 examples of local guidance. Evidence from the interviews suggested that Cochrane Reviews often play an instrumental role in informing guidance, although reviews being a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and GDs were barriers to their use. Cochrane Reviews appeared to have led to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments. However, whether or not these changes were directly as a result of the Cochrane Review and not the result of subsequent clinical guidance was difficult to judge. Potential benefits of Cochrane Reviews included economic benefits through budget savings or the release of funds, improvements in clinical quality, the reduction in the use of unproven or unnecessary procedures and improvements in patient and carer experiences.ConclusionsThis study identified a number of impacts and likely impacts of Cochrane Reviews. The clearest impacts of Cochrane Reviews are on research targeting and health-care policy, with less evidence of a direct impact on clinical practice and the organisation and delivery of NHS services. Although it is important for researchers to consider how they might increase the influence of their work, such impacts are difficult to measure. More work is required to develop suitable methods for defining and quantifying the impact of research.FundingThe NIHR Health Technology Assessment programme.

Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study

“…Although RCTs included in Cochrane reviews do not represent all RCTs, they cover a large and important body of evidence. 31 32 We think that research on such a large group of trials would not have been possible without Cochrane reviews.…”

“…They synthesise findings from RCTs and are used to aid decision making and develop practice guidelines. 31 32 The risk of bias of each included study is systematically assessed in duplicate by trained reviewers who reach consensus. Reviewers use the Cochrane risk of bias tool, which assesses several methodological items that are considered crucial to evaluating the validity of an RCT.…”

Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence

“…Systematic reviews have been used by policy-makers, researchers, and health service providers to inform decision-making [1]. Traditionally, systematic reviews have given preference to quantitative evidence (mainly from randomized controlled trials (RCTs) and to clinical effectiveness questions).…”

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

“…Systematic reviews are considered among the best available sources of research evidence and are increasingly relied on to inform decision-making [1]. The past 40 years have seen increasingly rapid methodological advances in the field of systematic reviews and research synthesis.…”