Refining the systematic literature review process—two participant-observer case studies

Abstract

Systematic literature reviews (SLRs) are a major tool for supporting evidencebased software engineering. Adapting the procedures involved in such a review to meet the needs of software engineering and its literature remains an ongoing process. As part of this process of refinement, we undertook two case studies which aimed 1) to compare the use of targeted manual searches with broad automated searches and 2) to compare different methods of reaching a consensus on quality. For Case 1, we compared a tertiary study of systematic literature reviews published between which used a manual search of selected journals and conferences and a replication of that study based on a broad automated search. We found that broad automated searches find more studies than manual restricted searches, but they may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers, or they are assessing research trends in research methodologies. For Case 2, we analyzed the process used to evaluate the quality of SLRs. We conclude that if quality evaluation of primary studies is a critical component of a specific SLR, assessments should be based on three independent evaluators incorporating at least two rounds of discussion.

An empirical assessment of a systematic search process for systematic reviews

“…Hence, manual search would be a valuable complement to the automated search in these cases. This is also another aspect that supports the Kitchenham et al's finding regarding the relationship of the two search methods [7].…”

“…Kitchenham et al reported an participant-observer case study [7] that investigated several factors for the search in SLRs: search breadth, gray literature and performance of manual and automated search. The results support a restricted manual search targeting an appropriate set of venues may help to avoid the omission of good quality papers.…”

Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

“…Given the results of Study 3, it appears that the benefits of discussion may be more limited than we had expected. Study 2 and the study by Kitchenham et al [24] both favoured discussion among participants but they did not consider whether the additional costs of discussion would lead to significantly better reliability than simply adding another judge and aggregating scores arithmetically. Kitchenham et al [24] found relatively good results by simply taking the median value of three independent scores without any discussion.…”

“…However, in the related papers we found that, with the exception of Kitchenham [24], and apart from studies of randomised controlled trials, there was little empirical evaluation of the use of criteria to determine the quality of studies that are to be used in meta-analysis or systematic reviews, so any evidence-based recommendations are better than nothing.…”

“…In the context of software engineering, Kitchenham et al [24] report the outcome of several different strategies they used to assess the quality of systematic literature reviews in software engineering using four criteria. They suggest that a process that they refer to as "consensus and minority report" is more reliable than either the median of three independent assessments or basing an assessment on two independent assessments and a discussion.…”