Irreproducibility in searches of scientific literature: A comparative analysis

Abstract

Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results. We present a comparative analysis of time‐synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms. We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions. To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine‐readable search metadata.

Psychosocial working conditions and mental health among medical assistants in Germany: a scoping review

“…This is because, unlike systematic reviews, scoping reviews are not intended to provide evidence to guide policies [ 21 ], but aim to provide an overview of available evidence regardless of its quality [ 79 ]. Further, the replicability of Google Scholar searches has been criticized [ 80 ]. We have addressed this limitation by uploading the results of our database retrieval on OSF ( https://osf.io/2837w/?view_only=37d260e1618e4556891819fd6e054f61 ), as recommended [ 80 ].…”

Standardising bee sampling: A systematic review of pan trapping and associated floral surveys

“…We identified post‐hoc that many studies using pan traps were not included in the search results, because the authors only mention the phrase “pan trap” in their methods section rather than in a WOS indexed field. In order to ensure full search reproducibility (Pozsgai et al., 2021 ), we proceeded with the systematic approach and additionally provide the complete list of references (Table S1 ).…”

Priority host plants for Ceratitis capitata, mediterranean fruit fly, based on the host reproduction number for surveillance, trade and eradication programs

“…We followed the methodology described by Dominiak (2022). In summary, Google Scholar was used as a search engine because it yields more results than other databases (Pozsgai et al 2021). The main research term used was "Ceratitis capitata".…”