Gray literature: An important resource in systematic reviews

Abstract

Systematic reviews aide the analysis and dissemination of evidence, using rigorous and transparent methods to generate empirically attained answers to focused research questions. Identifying all evidence relevant to the research questions is an essential component, and challenge, of systematic reviews. Gray literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Gray literature can include academic papers, including theses and dissertations, research and committee reports, government reports, conference papers, and ongoing research, among others. It may provide data not found within commercially published literature, providing an important forum for disseminating studies with null or negative results that might not otherwise be disseminated. Gray literature may thusly reduce publication bias, increase reviews' comprehensiveness and timeliness, and foster a balanced picture of available evidence. Gray literature's diverse formats and audiences can present a significant challenge in a systematic search for evidence. However, the benefits of including gray literature may far outweigh the cost in time and resource needed to search for it, and it is important for it to be included in a systematic review or review of evidence. A carefully thought out gray literature search strategy may be an invaluable component of a systematic review. This narrative review provides guidance about the benefits of including gray literature in a systematic review, and sources for searching through gray literature. An illustrative example of a search for evidence within gray literature sources is presented to highlight the potential contributions of such a search to a systematic review. Benefits and challenges of gray literature search methods are discussed, and recommendations made.

Explanatory note on literature searching conducted in the context of GMO applications for (renewed) market authorisation and annual post‐market environmental monitoring reports on GMOs authorised in the EU market

“…Search engines may identify additional relevant publications compared to those found from searches of electronic bibliographic databases, because they search the full text of publications (Paez, 2017). Web-based search engines, such as Google Scholar, search the internet for a wide range of content in different formats including articles, books, theses, reports and grey literature.…”

“…Search engines may identify additional relevant publications compared to those found from searches of electronic bibliographic databases, because they search the full text of publications (Paez, 2017). Search engine algorithms are often adapted to searchers' search behaviour and both retrieval and ranking of results may be affected by the location, the device used to search, the business model of the search engine and commercial purposes (Livoreil et al, 2017;Paez, 2017). In addition, search strategies may be limited to a maximum number of characters, meaning that it can be laborious to create a sensitive search strategy (Bramer et al, 2017).…”

Information seeking by geoscientists: an update on Bichteler and Ward (1989)

“…While geoscientists may be particularly dependent on gray literature, it is also important in other fields of science. For example, Paez (2017) emphasizes its importance to ensuring a balanced picture of the evidence in systematic reviews in medicine.…”

Inadequate diversity of information resources searched in US-affiliated systematic reviews and meta-analyses: 2005–2016

“…Although including both published and unpublished data is important for validity, in practice, neither is searching the unpublished data easy [17–19] nor are guidelines suggesting resources to be searched for unpublished data consistent among each other [20]. Most popular biomedical search engines mine only published studies, and little consensus exists regarding which resources to look into for unpublished data [9,21].…”

“…Additional repositories of trial reports can include regulatory body reports (FDA databases), grant databases, and manufacturer web sites [8]. Sources of gray literature that are not clinical trials include conference abstracts, dissertations, book chapters, policy documents, and specialized gray literature databases, among others [19]. Although studies have documented the underuse of registries [2,25–28], factors that may lead to more widespread inclusion of such resources in search strategies have not been studied.…”