Search. Review. Repeat? An empirical study of threats to replicating SLR searches

Krüger, Jacob; Lausberger, Christian; Nostitz-Wallwitz, Ivonne von; Saake, Gunter; Leich, Thomas

doi:10.1007/s10664-019-09763-0

Cited by 26 publications

(21 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since we replicated the searches at a different point in time, the returned publications are not identical, as we display in Table 2. We can clearly observe the differences in the original and replicated search results, which are mainly caused by changes and technical issues of digital libraries [22,24,42]. To ensure that all relevant papers are included, analysts can perform a separate snowballing and add the newly found papers manually in the BibTeX file.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Incorporating Altmetrics to Support Selection and Assessment of Publications During Literature Analyses

Shakeel

Alchokr

Krüger³

et al. 2022

The International Conference on Evaluation and Assessment in Software Engineering 2022

Self Cite

View full text Add to dashboard Cite

Background. The constantly increasing number of scientific publications poses challenges for researchers to monitor, select, and assess the publications relevant for their own research. Several guidelines for assessing publications manually during a literature analysis exist, with researchers proposing (semi-)automated techniques to facilitate such assessments. Aims. Still, research indicates that current techniques require further improvements to facilitate the analysis of large sets of publications. In this paper, we propose a semi-automatic technique with which we aim to improve in this direction by facilitating the selection and assessment of publications. Method. Our technique uses publicly available data of a publication, namely citation counts, article-level metrics, venue metrics, and altmetrics, to guide an analyst in assessing its relevance and impact. To evaluate the feasibility of our technique and the included metrics, we performed an experimental analysis to automatically assign ratings to the retrieved publications. Results. The results indicate that our technique can help an analyst in assessing publications, and reduce manual effort. Through our technique, we achieve an average accuracy of 53 % with a recall of 71 %. While precision (14 %) and F1-score (21 %) are-not surprisingly, due to the high number of irrelevant results returned by automatic searches in digital libraries-low, we see an improvement of these values for more recent reviews for which we could collect more complete data. However, some manual effort is still required for the final selection of papers. Conclusions. While it is not possible to achieve full automation for selecting and quality assessing publications, we can see that our metrics-based technique can be a helpful means to provide an initial rating for the analyst. Also, incorporating altmetrics seems to be a promising addition to rate comparably recent This work is licensed under a Creative Commons Attribution International 4.0 License.

show abstract

Section: Discussionmentioning

confidence: 99%

“…A systematic conduct is arguably a favorable property for any literature analysis to allow others to verify its quality, to replicate it, and to assess its completeness-essentially improving the trust in and usability of the results. So, a systematic, understandable conduct is important for any literature analysis [21,24,42,49].…”

Section: Background and Goalsmentioning

confidence: 99%

Incorporating Altmetrics to Support Selection and Assessment of Publications During Literature Analyses

Shakeel

Alchokr

Krüger³

et al. 2022

The International Conference on Evaluation and Assessment in Software Engineering 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Researchers have claimed the for gathering information through literature analyses, the SLR is the best qualitative research method (Karimi and Iordanova, 2021), (Psomas, 2021). Before conducting the systematic literature review, it is a must to set the review protocols for effective review of literature and body of knowledge as well as to extract accurate and as much as possible the factors of innovation (Krüger et al, 2020), (Mengist et al, 2020)…”

Section: Methodology and Propositionsmentioning

confidence: 99%

Do the organizational proficiencies dominate in innovation and can manage the major factors of innovation? A Systematic Literature Review of Last 5 Decades

Newaz¹,

Datta²,

Alam³

2022

JBRMR

View full text Add to dashboard Cite

Purpose of the research: Innovation is the key to organizational success in the present complex and competitive business environment. Numerous factors are continuously affecting the innovative performances of the organizations. researchers across the globe, have pointed out different factors and, often, several thematic factors that have been proven with strong positive effect on the innovation activities and innovation outcomes. The research attempted to find out the most dominating factors of innovations in the organizations through an extensive systematic literature review that ranges from 1973 to 2021. Methodology: Resulting on the 102 individual influencing factors of innovation through the literature review, the researchers classified those factors into 10 thematic groups, e.g., culture, HRM functions, top management orientation, external environment, organizational proficiencies, leadership, knowledge management, market pressure and competition, technology adaptation and research and development. Besides, the researchers endeavoured to figure out the interrelationships among the identified dominating factors. Findings: Finally, through couple of propositions, the research successfully identified 5 dominating factors of innovation, e.g., organizational proficiencies, external environment, culture, market pressure and competition, and HRM functions. Among 5, the factors of organizational proficiencies noticed the most dominating. The interrelationships between organizational proficiencies and external environment, culture, market pressure and competition, and HRM functions examined and delivered through an easy-to-understand diagram. Originality and implications: This robust research is very valuable, firstly, it has accumulated the factors of innovation from the studies of last 50 years, secondly, it has established 10 thematic factors of innovation, and thirdly, the research has figured out the most crucial thematic factor of innovation which accelerates innovation and can control the threats of other relevant factors of innovation.

show abstract

“…As mentioned above, the inconsistent search engine features and interfaces not only complicate the search process but also incur replicability problems. For example, search string adaptation has been identified as a major factor that threatens search replications [21]. Moreover, the engine features will have cascaded influences on the design and the usage of APIs.…”

Section: Recommendations For the Future Evolutionmentioning

confidence: 99%

“…Such evaluations are sparse on details, often being more like short experience reports. Although researchers individually may report the problems they encounter to the search-engine developers, as suggested by Kruger et al [21], there will still be a lack of holistic view for the search engine developers to understand the professional users' needs and to improve their products and services. We therefore decided to perform an extensive evaluation of academic search engines, in order to provide systematic feedback and evidence-based recommendations to both researchers and the relevant engine developers.…”

Section: Introductionmentioning

confidence: 99%

Academic Search Engines: Constraints, Bugs, and Recommendation

Li¹,

Rainer²

2022

Preprint

View full text Add to dashboard Cite

Background: Academic search engines (i.e., digital libraries and indexers) play an increasingly important role in systematic reviews however these engines do not seem to effectively support such reviews, e.g., researchers confront usability issues with the engines when conducting their searches. Aims: To investigate whether the usability issues are bugs (i.e., faults in the search engines) or constraints, and to provide recommendations to search-engine providers and researchers on how to tackle these issues. Method: Using snowballsampling from tertiary studies, we identify a set of 621 secondary studies in software engineering. By physically re-attempting the searches for all of these 621 studies, we effectively conduct regression testing for 42 search engines. Results: We identify 13 bugs for eight engines, and also identify other constraints. We provide recommendations for tackling these issues. Conclusions: There is still a considerable gap between the search-needs of researchers and the usability of academic search engines. It is not clear whether search-engine developers are aware of this gap. Also, the evaluation, by academics, of academic search engines has not kept pace with the development, by search-engine providers, of those search engines. Thus, the gap between evaluation and development makes it harder to properly understand the gap between the search-needs of researchers and search-features of the search engines.

show abstract

Search. Review. Repeat? An empirical study of threats to replicating SLR searches

Cited by 26 publications

References 52 publications

Incorporating Altmetrics to Support Selection and Assessment of Publications During Literature Analyses

Incorporating Altmetrics to Support Selection and Assessment of Publications During Literature Analyses

Do the organizational proficiencies dominate in innovation and can manage the major factors of innovation? A Systematic Literature Review of Last 5 Decades

Academic Search Engines: Constraints, Bugs, and Recommendation

Contact Info

Product

Resources

About