On the reliability of mapping studies in software engineering

Abstract

a b s t r a c tBackground: Systematic literature reviews and systematic mapping studies are becoming increasingly common in software engineering, and hence it becomes even more important to better understand the reliability of such studies. Objective: This paper presents a study of two systematic mapping studies to evaluate the reliability of mapping studies and point out some challenges related to this type of study in software engineering. Method: The research is based on an in-depth case study of two published mapping studies on software product line testing. Results: We found that despite the fact that the two studies are addressing the same topic, there are quite a number of differences when it comes to papers included and in terms of classification of the papers included in the two mapping studies. Conclusions: From this we conclude that although mapping studies are important, their reliability cannot simply be taken for granted. Based on the findings we also provide four conjectures that further research has to address to make secondary studies (systematic mapping studies and systematic literature reviews) even more valuable to both researchers and practitioners.

Evaluating strategies for study selection in systematic literature studies

“…Though, this will only be true if the sample on which the selection criteria are being applied is a true representative of the total population (c.f. [8]) and if there are certain consistent trends in results i.e. losing a few relevant articles will not alter the conclusions of the study altogether.…”

“…Unless we have more evidence to substantiate Lesson 4, we should avoid following such less inclusive strategies. Following the most inclusive strategy as a general principle, will reduce the likelihood of overlooking relevant articles and thus improve the consistency [8] of secondary studies.…”

“…Starting with a large set of potentially relevant studies, reviewers rely on several steps of selection, first by reading titles and abstracts, followed by full-text reading for quality assessment [3]. Thus, the reliability of a secondary study is highly dependent on the repeatability of the selection process [8].…”

Practical similarities and differences between Systematic Literature Reviews and Systematic Mappings: a tertiary study

“…In general, the purpose of a secondary study is to provide an overview of a research area and identify research gaps in this area [20]. SLRs and SMs, as secondary studies, enable the identification and aggregation of available evidence to answer research questions [21].…”

Exploring Model-to-Model Transformations for RIA Architectures by means of a Systematic Mapping Study

“…The main difference between SMSs and SLRs is in scope and in the analysis procedures. The scope is often broader in a SMS than in a SLR, and a deeper synthesis is not expected in a SMS [17]. Thus, we performed this SMS by considering the guidelines provided in [8,15,16].…”