An empirical assessment of a systematic search process for systematic reviews

Abstract

Background: Systematic Literature Reviews (SLRs) have been gaining significant attention from Software Engineering (SE) researchers since 2004. Several researches have also working on improving the scientific and technological infrastructure available to support SLRs in SE.Objective: The study reported in this paper aims to validate the QGS-based search process for SLR, i.e. whether a more effective and/or productive search can be achieved by following such a systematic process.Method: We used a dual-case study, in which each case includes two observations of SE literature search for the same SLR but using and not using the QGS-based approach.Results: The overall sensitivity and precision of each observation were calculated for the search cases that implemented different search design.Conclusions: A systematic search process (QGS-based search in this paper) appears to gain higher sensitivity and precision in the two cases of SLR. Such a search process may help capture more relevant studies as well as save reviewers' time spent in literature search activities. Our observations also proof that an integrated search strategy is recommended for SLRs in SE to avoid the possible limitations of applying single manual or automated search.

Systematic literature studies

“…The approach was based on the concept of quasi-gold standard for retrieving and identifying relevant studies, and it was concluded to serve the purpose and hence it can be used as a supplement to the guidelines for SLRs in EBSE. In a follow up validation study [13], a dual-case study was performed, and the proposed approach seemed to be more efficient than the EBSE process in capturing relevant studies and in saving reviewers' time. Further, the authors recommended an integrated search strategy to avoid limitations of applying a manual strategy or an automated search strategy.…”

“…Database searches and snowballing are by no means the only options. The use of personal knowledge or contacts [8], or mixed methods [13] has also been discussed in the literature. The focus here is, however, on the first step of two recommended methods to identify relevant literature.…”

Empirical studies of embedded software development using agile methods

“…Among them, "Real-time", "Medical devices", "Multimedia" and "Middleware" were excluded as the "embedded" factor is not their inherent attribute in our mind. In addition, "Sensor" was excluded mainly for its "precision" [3]. We were glad to find that the appraisal of the search string, which is present in Table 1, was not bad.…”

“…Development" OR DSDM OR "Dynamic Systems Development Method" OR (FDD AND software) OR "Feature Driven Development" OR (XP AND (software OR development)) OR "Extreme Programming" OR (Lean AND (software OR development)) OR Xbreed OR Scrum OR RUP OR "Rational Unified Process" OR (Crystal AND software) OR EVO OR "Evolutionary Project Management") AND (Embedded OR mobile OR automotive) Note: Considering the controversy about "RUP", we add it according to [2]. The search string was appraised based on the method proposed in [3] and the result is shown in Table 1: …”

FAST2: An intelligent assistant for finding relevant papers

“…2. Compared different approaches, such as search-query based methods (Zhang et al, 2011;Umemoto et al, 2016), reference-based methods (Jalali and Wohlin, 2012;Felizardo et al, 2016;Wohlin, 2014), supervised learning (Cohen et al, 2006;Adeva et al, 2014), semisupervised learning (Liu et al, 2016), unsupervised learning (Malheiros et al, 2007), and found that active learning is the most efficient in reducing the cost of primary study selection. 3.…”