Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10–67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations’ websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

Systematic mapping was developed in social sciences in response to a lack of empirical data when answering questions using systematic review methods, and a need for a method to describe the literature across a broad subject of interest. Systematic mapping does not attempt to answer a specific question as do systematic reviews, but instead collates, describes and catalogues available evidence (e.g. primary, secondary, theoretical, economic) relating to a topic or question of interest. The included studies can be used to identify evidence for policy-relevant questions, knowledge gaps (to help direct future primary research) and knowledge clusters (sub-sets of evidence that may be suitable for secondary research, for example systematic review). Evidence synthesis in environmental sciences faces similar challenges to those found in social sciences. Here we describe the translation of systematic mapping methodology from social sciences for use in environmental sciences. We provide the first process-based methodology for systematic maps, describing the stages involved: establishing the review team and engaging stakeholders; setting the scope and question; setting inclusion criteria for studies; scoping stage; protocol development and publication; searching for evidence; screening evidence; coding; production of a systematic map database; critical appraisal (optional); describing and visualising the findings; report production and supporting information. We discuss the similarities and differences in methodology between systematic review and systematic mapping and provide guidance for those choosing which type of synthesis is most suitable for their requirements. Furthermore, we discuss the merits and uses of systematic mapping and make recommendations for improving this evolving methodology in environmental sciences.

Reliable synthesis of the various rapidly expanding bodies of evidence is vital for the process of evidence-informed decision-making in environmental policy, practice and research. With the rise of evidence-base medicine and increasing numbers of published systematic reviews, criteria for assessing the quality of reporting have been developed. First QUOROM (Lancet 354:1896-1900, 1999) and then PRISMA (Ann Intern Med 151:264, 2009) were developed as reporting guidelines and standards to ensure medical meta-analyses and systematic reviews are reported to a high level of detail. PRISMA is now widely used by a range of journals as a pre-submission checklist. However, due to its development for systematic reviews in healthcare, PRISMA has limited applicability for reviews in conservation and environmental management. We highlight 12 key problems with the application of PRISMA to this field, including an overemphasis on meta-analysis and no consideration for other synthesis methods. We introduce ROSES (RepOrting standards for Systematic Evidence Syntheses), a pro forma and flow diagram designed specifically for systematic reviews and systematic maps in the field of conservation and environmental management. We describe how ROSES solves the problems with PRISMA. We outline the key benefits of our approach to designing ROSES, in particular the level of detail and inclusion of rich guidance statements. We also introduce the extraction of meta-data that describe key aspects of the conduct of the review. Collated together, this summary record can help to facilitate rapid review and appraisal of the conduct of a systematic review or map, potentially speeding up the peer-review process. We present the results of initial road testing of ROSES with systematic review experts, and propose a plan for future development of ROSES.

No abstract

Review articles can provide valuable summaries of the ever-increasing volume of primary research in conservation biology. Where findings may influence important resource-allocation decisions in policy or practice, there is a need for a high degree of reliability when reviewing evidence. However, traditional literature reviews are susceptible to a number of biases during the identification, selection, and synthesis of included studies (e.g., publication bias, selection bias, and vote counting). Systematic reviews, pioneered in medicine and translated into conservation in 2006, address these issues through a strict methodology that aims to maximize transparency, objectivity, and repeatability. Systematic reviews will always be the gold standard for reliable synthesis of evidence. However, traditional literature reviews remain popular and will continue to be valuable where systematic reviews are not feasible. Where traditional reviews are used, lessons can be taken from systematic reviews and applied to traditional reviews in order to increase their reliability. Certain key aspects of systematic review methods that can be used in a context-specific manner in traditional reviews include focusing on mitigating bias; increasing transparency, consistency, and objectivity, and critically appraising the evidence and avoiding vote counting. In situations where conducting a full systematic review is not feasible, the proposed approach to reviewing evidence in a more systematic way can substantially improve the reliability of review findings, providing a time- and resource-efficient means of maximizing the value of traditional reviews. These methods are aimed particularly at those conducting literature reviews where systematic review is not feasible, for example, for graduate students, single reviewers, or small organizations.

Background: The loss of carbon (C) from agricultural soils has been, in part, attributed to tillage, a common practice providing a number of benefits to farmers. The promotion of less intensive tillage practices and no tillage (NT) (the absence of mechanical soil disturbance) aims to mitigate negative impacts on soil quality and to preserve soil organic carbon (SOC). Several reviews and meta-analyses have shown both beneficial and null effects on SOC due to no tillage relative to conventional tillage, hence there is a need for a comprehensive systematic review to answer the question: what is the impact of reduced tillage intensity on SOC? Methods:We systematically reviewed relevant research in boreo-temperate regions using, as a basis, evidence identified within a recently completed systematic map on the impacts of farming on SOC. We performed an update of the original searches to include studies published since the map search. We screened all evidence for relevance according to predetermined inclusion criteria. Studies were appraised and subject to data extraction. Meta-analyses were performed to investigate the impact of reducing tillage [from high (HT) to intermediate intensity (IT), HT to NT, and from IT to NT] for SOC concentration and SOC stock in the upper soil and at lower depths.Results: A total of 351 studies were included in the systematic review: 18% from an update of research published in the 2 years since the systematic map. SOC concentration was significantly higher in NT relative to both IT [1.18 g/ kg ± 0.34 (SE)] and HT [2.09 g/kg ± 0.34 (SE)] in the upper soil layer (0-15 cm). IT was also found to be significant higher [1.30 g/kg ± 0.22 (SE)] in SOC concentration than HT for the upper soil layer (0-15 cm). At lower depths, only IT SOC compared with HT at 15-30 cm showed a significant difference; being 0.89 g/kg [± 0.20 (SE)] lower in intermediate intensity tillage. For stock data NT had significantly higher SOC stocks down to 30 cm than either HT [4.61 Mg/ ha ± 1.95 (SE)] or IT [3.85 Mg/ha ± 1.64 (SE)]. No other comparisons were significant. Conclusions:The transition of tilled croplands to NT and conservation tillage has been credited with substantial potential to mitigate climate change via C storage. Based on our results, C stock increase under NT compared to HT was in the upper soil (0-30 cm) around 4.6 Mg/ha (0.78-8.43 Mg/ha, 95% CI) over ≥ 10 years, while no effect was detected in the full soil profile. The results support those from several previous studies and reviews that NT and IT increase SOC in the topsoil. Higher SOC stocks or concentrations in the upper soil not only promote a more productive soil with higher biological activity but also provide resilience to extreme weather conditions. The effect of tillage practices on total SOC stocks will be further evaluated in a forthcoming project accounting for soil bulk densities and crop yields. Our findings can hopefully be used to guide policies for sustainable management of agricultural soils.

Systematic reviews and systematic maps represent powerful tools to identify, collect, evaluate and summarise primary research pertinent to a specific research question or topic in a highly standardised and reproducible manner. Even though they are seen as the "gold standard" when synthesising primary research, systematic reviews and maps are typically resource-intensive and complex activities. Thus, managing the conduct and reporting of such reviews can become a time consuming and challenging task. This paper introduces the open access online tool CADIMA, which was developed through a collaboration between the Julius Kühn-Institut and the Collaboration for Environmental Evidence, in order to increase the efficiency of the evidence synthesis process and facilitate reporting of all activities to maximise methodological rigour. Furthermore, we analyse how CADIMA compares with other available tools by providing a comprehensive summary of existing software designed for the purposes of systematic review management. We show that CADIMA is the only available open access tool that is designed to: (1) assist throughout the systematic review/map process; (2) be suited to reviews broader than medical sciences; (3) allow for offline data extraction; and, (4) support working as a review team.