Grey Literature in Software Engineering: A critical review

Kamei, Fernando; Wiese, Igor; Lima, Crescêncio; Polato, Ivanilton; Nepomuceno, Vilmar; Ferreira, Waldemar; Ribeiro, Márcio; Pena, Carolline; Cartaxo, Bruno; Pinto, Gustavo; Soares, Sérgio

doi:10.1016/j.infsof.2021.106609

Cited by 29 publications

(19 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…"information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing, i.e., where publishing is not the primary activity of the producing body." This definition was developed by librarians and information scientists and was agreed at the Third International Conference on Grey Literature in 1997 2 . It was subsequently expanded at the New York conference in 2004.…”

Section: Defining Grey Literaturementioning

confidence: 99%

“…Recent reviews of the use of grey literature have confirmed that SE systematic reviews do include grey literature. For example, Kamei et al [2] found 126 SRs out of a total of 446 that included references to grey literature. Unfortunately, they included book and book chapters in their definition of grey literature, as well as social media sources.…”

Section: Defining Grey Literaturementioning

confidence: 99%

“…In addition to the above issues, SE researchers have pointed out that there is an issue concerning the transience of online material [2], because there is no guarantee that the cited blogs or private communications will remain accessible in the long-term. For example, Garousi and Mäntylä [16] cited 46 internet articles and white papers using URL addresses, but, as of 25th May 2021, Kitchenham and Madeyski independently confirmed that only 19 were still accessible.…”

Section: Problems With Analysing Blogsmentioning

confidence: 99%

“…For example, a special issue of Information and Software Technology [1] was based on grey literature and multivocal reviews. In addition, there are several mapping studies that have reviewed the use of grey literature in systematic reviews (see [2], [3], and [4]).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

How Should Software Engineering Secondary Studies Include Grey Material?

Kitchenham

Madeyski

Budgen

2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Recent papers have proposed the use of grey literature (GL) and multivocal reviews. These papers have raised issues about the practices used for systematic reviews (SRs) in software engineering (SE) and suggested that there should be changes to the current SR guidelines. Objective: To investigate whether current SR guidelines need to be changed to support GL and multivocal reviews. Method: We discuss the definitions of GL and the importance of GL and of industry-based field studies in SE SRs. We identify properties of SRs that constrain the material used in SRs: a) the nature of primary studies; b) the requirements of SRs to be auditable, traceable, and reproducible; and explain why these requirements restrict the use of blogs in SRs. Results: SR guidelines have always considered GL as a possible source of primary studies and have never supported exclusion of field studies that incorporate the practitioners' viewpoint. However, the concept of GL, which was meant to refer to documents that were not formally published, is now being extended to information from sources such as blogs/tweets/Q&A posts. Thus, it might seem that SRs do not make full use of GL because they do not include such information. However, the unit of analysis for an SR is the primary study. Thus, it is not the source but the type of information that is important. Any report describing a rigorous empirical evaluation is a candidate primary study. Whether it is actually included in an SR depends on the SR eligibility criteria. However, any study that cannot be guaranteed to be publicly available in the long term should not be used as a primary study in an SR. This does not prevent such information from being aggregated in surveys of social media and used in the context of evidence-based software engineering (EBSE). Conclusions: Current guidelines for SRs do not require extensions, but their scope needs to be better defined. SE researchers require guidelines for analysing social media posts (e.g., blogs, tweets, vlogs), but these should be based on qualitative primary (not secondary) study guidelines. SE researchers can use mixed-methods SRs and/or the fourth step of EBSE to incorporate findings from social media surveys with those from SRs and to develop industry-relevant recommendations.

show abstract

Section: Defining Grey Literaturementioning

confidence: 99%

Section: Defining Grey Literaturementioning

confidence: 99%

Section: Problems With Analysing Blogsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

How Should Software Engineering Secondary Studies Include Grey Material?

Kitchenham

Madeyski

Budgen

2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Examples of grey literature are, e.g., presentations, videos, news articles, blogs, tweets, websites, emails, and so on. Kamei et al propose using different shades of grey to categorise them [64]. Recently, there have been concerns about performing guidelines updates frequently, and it is reasoned that updates should be made only if necessary and with a good justification [65].…”

Section: Systematic Reviewmentioning

confidence: 99%

Attraction Basins in Metaheuristics: A Systematic Mapping Study

2021

View full text Add to dashboard Cite

Context: In this study, we report on a Systematic Mapping Study (SMS) for attraction basins in the domain of metaheuristics. Objective: To identify research trends, potential issues, and proposed solutions on attraction basins in the field of metaheuristics. Research goals were inspired by the previous paper, published in 2021, where attraction basins were used to measure exploration and exploitation. Method: We conducted the SMS in the following steps: Defining research questions, conducting the search in the ISI Web of Science and Scopus databases, full-text screening, iterative forward and backward snowballing (with ongoing full-text screening), classifying, and data extraction. Results: Attraction basins within discrete domains are understood far better than those within continuous domains. Attraction basins on dynamic problems have hardly been investigated. Multi-objective problems are investigated poorly in both domains, although slightly more often within a continuous domain. There is a lack of parallel and scalable algorithms to compute attraction basins and a general framework that would unite all different definitions/implementations used for attraction basins. Conclusions: Findings regarding attraction basins in the field of metaheuristics reveal that the concept alone is poorly exploited, as well as identify open issues where researchers may improve their research.

show abstract