The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

Abstract

Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10–67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations’ websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

Systematic Reviews for Occupational Safety and Health Questions: Resources for Evidence Synthesis

“…However, it should also be kept in mind that there is only limited research available that has empirically determined the effect of any shortcuts or deviations from systematic review standards (Tsertsvadze et al, 2015). Only a few studies have investigated the incremental value of specific systematic review methods (Dickersin, Scherer, and Lefebvre, 1994;Moher et al, 2000;Bushman and Wells, 2001;Horsley, Dingwall, and Sampson, 2011;Giustini and Boulos, 2013;Selph, Ginsburg, and Chou, 2014;and Haddaway et al, 2015) or have tested the validity of the end product by comparing the conclusions reached in rapid reviews versus systematic reviews (Watt et al, 2008a;Watt et al, 2008b;and Hartling et al, 2015a). Hence, we know very little about the impact of deviating from full systematic review methodology.…”

Systematic searching for environmental evidence using multiple tools and sources

“…they order results using an unknown and often changing algorithm, [14]) and are restricted in their scope or in the number of results that can be viewed by the user (Google Scholar). Google Scholar has been shown not to be suitable as a standalone resource in systematic reviews but it remains a valuable tool for supplementing bibliographic searches [6,19] and to obtain full-text PDF of articles. BASE Bielefeld academic search engine (https://www.base-search.net) is developed by the University of Bielefeld (Germany) and gives access to a wide range of information, including academic articles, audio files, maps, theses, newspaper articles, and datasets.…”

“…Search strings needs to be tailored to the search engine of each electronic bibliographic source to be searched (e.g. [19]). To build up the string, the team should rely on the syntax that is available in the help pages of the bibliographic sources, including the use of Boolean operators, where applicable.…”

Google Scholar como una fuente de evaluación científica: una revisión bibliográfica sobre errores de la base de datos

“…One such study that merits our attention is that published by Haddaway et al (2015), who, after investigating the usefulness of Google Scholar as a database in systematic reviews and grey literature, calculated a total rate of duplicate records due to parsing errors of around 5%, because of the following factors:…”