On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software Engineering

Abstract

The inclusion of grey literature (GL) is important to remove publication bias while gathering available evidence regarding a certain topic. The number of systematic literature reviews (SLRs) in Software Engineering (SE) is increasing but we do not know about the extent of GL usage in these SLRs. Moreover, Google Scholar is rapidly becoming a search engine of choice for many researchers but the extent to which it can find the primary studies is not known. Objective: This tertiary study is an attempt to i) measure the usage of GL in SLRs in SE. Furthermore this study proposes strategies for categorizing GL and a quality checklist to use for GL in future SLRs; ii) explore if it is feasible to use only Google Scholar for finding scholarly articles for academic research. Method: We have conducted a systematic mapping study to measure the extent of GL usage in SE SLRs as well as to measure the feasibility of finding primary studies using Google Scholar. Results and conclusions: a) Grey Literature: 76.09% SLRs (105 out of 138) in SE have included one or more GL studies as primary studies. Among total primary studies across all SLRs (6307), 582 are classified as GL, making the frequency of GL citing as 9.23%. The intensity of GL use indicate that each SLR contains 5 primary studies on average (total intensity of GL use being 5.54). The ranking of GL tells us that conference papers are the most used form 43.3% followed by technical reports 28.52%. Universities, research institutes, labs and scientific societies together make up 67.7% of GL used, indicating that these are useful sources for searching GL. We additionally propose strategies for categorizing GL and criteria for evaluating GL quality, which can become a basis for more detailed guidelines for including GL in future SLRs. b) Google Scholar Results: The results show that Google Scholar was able to retrieve 96% of primary studies of these SLRs. Most of the primary studies that were not found using Google Scholar were from grey sources.

Usability-In-Place—Remote Usability Testing Methods for Homebound Older Adults: Rapid Literature Review

“…Both searches are summarized in Table 1. We began with Google Scholar to take advantage of its relevance-based sorting feature and broader inclusion of diverse disciplines, academic and practice-based publications, and grey literature [8]. However, we conducted further searches because of the known limitations of Google Scholar, such as its lack of transparency and lack of specialization [9].…”

What drives citations of frontier application publications?

“…With 389 million records, GS is currently the most comprehensive academic search engine ( Gusenbauer, 2019 ). Moreover, more than 90% of the primary literature for systematic literature review is accessible from GS ( Yasin et al, 2020 ). Further, the use of GS permits the presentation of a wholistic picture of knowledge dissemination in economics and related fields ( Bosquet and Combes, 2013 ).…”

Query Processing of Geosocial Data in Location-Based Social Networks

“…Metrics (("geosocial networking" OR "geosocial network*" OR "location-based social network*") AND "metric*") 2 Privacy (("geosocial networking" OR "geosocial network*" OR "location-based social network*") AND "privacy") 33 Therefore, the trend provided in Figure 1 and Table 1 shows us that geosocial networking is a popular topic that attracts the interest of the scientific community. The main addressed research issues within this topic are the recommendations of geosocial data that facilitate users to find relevant places and friends, the privacy of the users' sensitive geosocial data, and the query processing that allows extracting meaningful data from geosocial databases.…”